Title: Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing
Venue: NeurIPS 2023
Accepted: yes
Keywords: computer vision; vision transformer; visual adapter; transfer learning
Abstract: The advent of high-capacity pre-trained models has revolutionized problem-solving in computer vision, shifting the focus from training task-specific models to adapting pre-trained models. Consequently, effectively adapting large pre-trained models to downstream tasks in an efficient manner has become a prominent research area. Existing solutions primarily concentrate on designing lightweight adapters and their interaction with pre-trained models, with the goal of minimizing the number of parameters requiring updates. In this study, we propose a novel Adapter Re-Composing (ARC) strategy that addresses efficient pre-trained model adaptation from a fresh perspective. Our approach considers the reusability of adaptation parameters and introduces a parameter-sharing scheme. Specifically, we leverage symmetric down-/up-projections to construct bottleneck operations, which are shared across layers. By learning low-dimensional re-scaling coefficients, we can effectively re-compose layer-adaptive adapters. This parameter-sharing strategy in adapter design allows us to further reduce the number of new parameters while maintaining satisfactory performance, thereby offering a promising approach to compress the adaptation cost. We conduct experiments on 24 downstream image classification tasks using various Vision Transformer variants to evaluate our method. The results demonstrate that our approach achieves compelling transfer learning performance with a reduced parameter count. Our code is available at https://github.com/DavidYanAnDe/ARC.

Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing

Abstract

The advent of high-capacity pre-trained models has revolutionized problem-solving in computer vision, shifting the focus from training task-specific models to adapting pre-trained models. Consequently, effectively adapting large pre-trained models to downstream tasks in an efficient manner has become a prominent research area. Existing solutions primarily concentrate on designing lightweight adapters and their interaction with pre-trained models, with the goal of minimizing the number of parameters requiring updates. In this study, we propose a novel Adapter ReComposing (ARC) strategy that addresses efficient pre-trained model adaptation from a fresh perspective. Our approach considers the reusability of adaptation parameters and introduces a parameter-sharing scheme. Specifically, we leverage symmetric down-/up-projections to construct bottleneck operations, which are shared across layers. By learning low-dimensional re-scaling coefficients, we can effectively re-compose layer-adaptive adapters. This parameter-sharing strategy in adapter design allows us to further reduce the number of new parameters while maintaining satisfactory performance, thereby offering a promising approach to compress the adaptation cost. We conduct experiments on 24 downstream image classification tasks using various Vision Transformer variants to evaluate our method. The results demonstrate that our approach achieves compelling transfer learning performance with a reduced parameter count. Our code is available at https://github.com/DavidYanAnDe/ARC.

1 Introduction

The utilization of large-scale pre-trained models for various downstream tasks has garnered significant interest in the computer vision community [1; 2; 3; 4]. These models continually push the boundaries of downstream task performance while eliminating the need for task-specific model design and training. In early attempts, a commonly adopted transfer learning strategy involved directly finetuning a pre-trained model on downstream tasks. However, the full fine-tuning strategy suffers from two major drawbacks: (1) Updating large-scale parameters is prohibitively expensive and typically requires a substantial amount of training data to prevent overfitting. (2) As the sizes of state-of-the-art pre-trained models continue to increase, it becomes impractical and unsustainable to store a distinct set of model weights for each downstream task.

In contrast to fine-tuning the entire pre-trained network, recent research has focused on parameterefficient model adaptation. The core idea behind this line of work is to keep the majority of pre-trained parameters frozen and only update or introduce a small fraction of task-specific parameters. Several methods fall under this umbrella, including prompt tuning [5; 6], visual adapter [7; 8], and linear feature modulation [9]. These methods have demonstrated competitive or even superior performance compared to full fine-tuning while significantly reducing the adaptation cost. They differ in the design of lightweight adapters and how these adapters interact with the pre-trained parameters. Prompt tuning methods [5; 6] adapt the features of pre-trained Vision Transformers by introducing trainable task-specific tokens into one or multiple attention layers. Visual adapter [7; 8] injects a non-linear lightweight adapter with a bottleneck architecture between layers of the pre-trained model to adjust the feature distribution. Such non-linear adapters are simplified using linear transformations such as shifting and scaling [9] to directly modulate the pre-trained features.

In this work, we not only focus on designing lightweight adapters but also emphasize the importance of adaptation parameter reusability in further compressing the adaptation cost. We adopt a low-rank design for the adapter using a bottleneck operation but propose a novel approach. Unlike other methods that place the adapter in different layers and directly learn different parameters for each adapter to cater to layer-wise variation, we propose sharing the down/up projections in the low-rank adapter across different layers and simply learning low-dimensional re-scaling coefficients to recompose the linear projections into layer-adaptive adapters. The idea of Adapter Re-Composing (ARC) is motivated by the observation that naturally derived adaptation matrices can exhibit extremely low-rank characteristics, even when not explicitly designed as such. This implies the possibility of using a shared 'basis' to re-compose the adapters. Furthermore, we design the low-rank adapters using symmetric down-projection and up-projection matrices, which further reduces the parameter size. Due to their linear nature and careful positioning design, our adapters can be seamlessly integrated into the pre-trained network, as in [7; 9; 10], without adding extra computation during the inference phase.

We evaluate our method on various large Vision Transformer models, including ViT-B [1] and its variants such as ViT-L [1], ViT-H [1], and Swin-B [11], using 24 downstream image classification benchmark datasets. The experimental results demonstrate that our method achieves compelling transfer learning performance while maintaining a smaller parameter size.

The key contributions of this paper are summarized as follows:

We approach efficient pre-trained model adaptation from a novel perspective by exploring the reusability of adaptation parameters, which goes beyond existing works that primarily focus on the lightweight design of adapter structures.

We introduce the Adapter Re-Composing (ARC) strategy, which shares bottleneck operation's down-/up-projections across layers and utilizes lower-dimensional re-composing coefficients to create layer-adaptive adapters. This approach enables fewer parameters than prior works.

Our parameter sharing scheme in the ARC method prevents a linear increase in parameter size with the number of layers, ensuring better scalability, particularly for larger-scale models.

Through extensive experiments on various Vision Transformer variations and numerous downstream tasks, we show that our method achieves highly competitive transfer learning performance while maintaining a relatively low level of additional parameter.

2 Related work

In this section, we present a concise review of the existing literature, focusing on two key areas: (1) Pre-training and fine-tuning, and (2) Parameter-efficient transfer learning.

Pre-training and fine-tuning. Pre-training and fine-tuning, also known as transfer learning [12; 13; 14], is a popular approach that utilizes large-scale datasets [15; 16; 17; 18; 19] to train models for adaptation to different downstream tasks. By extracting knowledge from these datasets and encoding it into parameters, models can be fine-tuned for specific tasks, resulting in improved performance compared to models without pre-training. The availability of large-scale datasets [17] has led to

Trainable Weights:

Trainable

Token

Different

Weight Tokens

Trainable

Token

Transformer Block

……

Token

Trainable

Token

……

Transformer Block

……

Token

Trainable

Token

Prompt Learning

ReLU

Up

Projection

Output

Weight

Matrix

Different

Weight Matrices

……

Different

Weight Matrices

……

Input

ReLU

Down

Projection

Output

Input

Output

Input

Weight

Matrix

Output

Input

Visual Adapter

LoRA

Token

……

Different

Weight Tokens

Transformer Block

Trainable

Token

Trainable

Token

Trainable Weights:

Token

Token

……

Transformer Block

Trainable

Token

Trainable

Token

Token

……

Prompt Learning

Shift

Scale

Output

……

Different

Weights

Input

SSF

·

β

Shift

Scale

Output

Input

·

γ

γ

β

Up

Projection

Up

Projection

Up

Projection

Down

Projection

Down

Projection

Down

Projection

(a) Prompt Learning [6].

ReLU

Up

Projection

Output

Weight

Matrix

Different

Weight Matrices

……

Different

Weight Matrices

……

Input

ReLU

Down

Projection

Output

Input

Output

Input

Weight

Matrix

Output

Input

Visual Adapter

LoRA

Token

……

Different

Weight Tokens

Transformer Block

Trainable

Token

Trainable

Token

Trainable Weights:

Token

Token

……

Transformer Block

Trainable

Token

Trainable

Token

Token

……

Prompt Learning

Shift

Scale

Output

……

Different

Weights

Input

SSF

·

β

Shift

Scale

Output

Input

·

γ

γ

β

Up

Projection

Up

Projection

Up

Projection

Down

Projection

Down

Projection

Down

Projection

(b) Visual Adapter [7].

ReLU

Up

Projection

Output

Weight

Matrix

Different

Weight Matrices

……

Different

Weight Matrices

……

Input

ReLU

Down

Projection

Output

Input

Output

Input

Weight

Matrix

Output

Input

Visual Adapter

LoRA

Token

……

Different

Weight Tokens

Transformer Block

Trainable

Token

Trainable

Token

Trainable Weights:

Token

Token

……

Transformer Block

Trainable

Token

Trainable

Token

Token

……

Prompt Learning

Shift

Scale

Output

……

Different

Weights

Input

SSF

·

β

Shift

Scale

Output

Input

·

γ

γ

β

Up

Projection

Up

Projection

Up

Projection

Down

Projection

Down

Projection

Down

Projection

(c) LoRA [24].

ReLU

Up

Projection

Output

Weight

Matrix

Different

Weight Matrices

……

Different

Weight Matrices

……

Input

ReLU

Down

Projection

Output

Input

Output

Input

Weight

Matrix

Output

Input

Visual Adapter

LoRA

Token

Token

Shift

Scale

Output

……

Different

Weights

Input

SSF

·

β

Shift

Scale

Output

Input

·

γ

γ

β

Up

Projection

Up

Projection

Up

Projection

Down

Projection

Down

Projection

Down

Projection

(d) SSF [9].

significant advancements in performance and convergence speed for downstream tasks. Scaling up models [20; 21] to handle the growing data volume enhances data utilization and improves the efficiency and robustness of pre-trained models across various data distributions and noise levels. Large models [1; 11; 22] also benefit from self-supervised pre-training [2; 23] using unlabeled data, reducing the cost, duration, and quality issues associated with human annotation. By leveraging this approach, models can effectively extract knowledge from readily available unlabeled data, further enhancing their generalization performance in downstream tasks.

Parameter-efficient transfer learning. The exponential increase in model parameters presents a computational challenge when fine-tuning the entire network on downstream tasks. In the field of NLP, researchers have explored parameter-efficient transfer learning approaches [7; 24; 25; 26; 27; 28] that train a subset of the model or add new modules with fewer parameters while achieving comparable or even superior performance. Inspired by the success of NLP, several notable works [8; 29; 30] have emerged in the computer vision domain. One approach, known as Prompt Tuning [5; 6], addresses the distribution mismatch between pre-training and downstream tasks by learning task-specific tokens. Adapter-like methods [8; 7; 29] insert trainable modules, such as MLPs with activation functions and residual structures, into the network to facilitate transfer learning. LoRA [24] exploits the low-rank update to a large-scale frozen model and introduces a bypass to the original parameter matrix to mimic the fine-tuning of the entire model parameters. SSF [9] introduces lightweight scaling and shift operations to modulate the pre-trained representations. The core concepts of these aforementioned works are visually summarized in Fig. 1.

As an advancement in visual adaptation, ARC addresses the limitations of Prompt Tuning, which requires different prompt designs for different downstream tasks. Moreover, ARC introduces the innovative concept of bottleneck matrix reuse, achieving state-of-the-art performance with minimal adaptation cost comparing to other rank-decomposition strategies. Additionally, ARC employs a linear design for the adapters and inherits the benefits of re-parameterization [9; 10; 24], ensuring that inference does not introduce any additional computational complexity.

3 Approach

In this section, we start by providing an introduction to the notations, symbols, and background related to Vision Transformers, which is followed by the presentation of our proposed Adapter Re-Composing (ARC) method. ARC focuses on efficient transfer learning for Vision Transformers by reusing rank-decomposition projections to adaptively compose layer-wise adapters, thereby reducing the size of learnable parameters. Additionally, we discuss the insights gained from our architecture design.

3.1 Preliminary

A plain Vision Transformer model contains a patch embedding layer and multiple encoder layers. Given an input image X ∈ R H × W × C , the patch embedding layer first splits the image into a sequence of flattened patches X patches ∈ R N × ( P 2 · C ) , where ( H,W ) is the resolution of the input image, ( P, P ) is the resolution of each patch, C denotes the number of input channels, and N = H · W/P 2 is the number of tokens. Subsequently, the image patches are mapped to a D -dimensional embedding space through a linear projection W ∈ R ( P 2 · C ) × D . A learnable [class] token vector ⃗ x cls ∈ R D is then prepended to the sequence of X patches , and the position embeddings X pos ∈ R ( N +1) × D are added to the sequence. The output of the patch embedding layer can be expressed as follows:

where [ · ; · ] denotes concatenation operation. This output is then fed into several consecutive encoder layers, each consisting of a Multi-Head Attention (MHA) block and a Feed-Forward Network (FFN) block. LayerNorm (LN) is applied before each block, and residual connections are applied thereafter. The process of l -th encoder layer is defined as:

where X ( l -1) denotes input tokens in l -th layer, X ( l ) ′ indicates intermediate representations produced by MHA, and the output of l -th layer is X ( l ) .

In MHA block, each Attention Head (AH) module utilizes the weight matrices W ( l ) q ∈ R D ( l -1) × D ( l ) h , W ( l ) k ∈ R D ( l -1) × D ( l ) h , and W ( l ) v ∈ R D ( l -1) × D ( l ) h for the query , key , and value operations, respectively. These operations enable an exclusive attention mechanism on the normalized feature representations X ( l -1) norm = LN( X ( l -1) ) :

where D ( l ) h = D ( l ) M is the feature dimensionality of the output representations X ( l ) ′ h for each attention head and M represents the number of attention heads. The MHA block concatenates multiple { X ( l ) ′ h } in sequence and generates the outputs through a linear projection W ( l ) o ∈ R ( M · D ( l ) h ) × D ( l ) :

The FFN block consists of two linear projections with the GELU activation function in between:

where W ( l ) 1 ∈ R D ( l ) × 4 · D ( l ) and W ( l ) 2 ∈ R 4 · D ( l ) × D ( l ) denote two linear projection matrices, and X ( l ) ′ norm = LN( X ( l ) ′ ) .

3.2 Adapter Re-Composing method

We observe that the majority of existing adaptation methods introduce adapters into various layers and learn separate parameters for adapting the pre-trained model to specific downstream tasks. Previous studies [8; 24] have shown the effectiveness of leveraging the low-rank properties of adapters to fine-tune frozen pre-trained models. Inspired by these findings, we propose a novel approach to create a unified linear space across different adapters to enhance parameter efficiency and adaptation performance.

Architecture. The ARC architecture incorporates a bottleneck operation for adapters, which consists of three key components: a linear down-projection, layer-specific re-scaling coefficients, and a linear up-projection. This architecture is illustrated in Fig. 2. To facilitate the reusability of adaptation matrices, we have developed a sharing and re-composing scheme for the ARC method.

Down

Projection

Up

Projection

……

Identical Symmetric

Weight Matrices for MHA

Re-Scaling

Coefficients

Trainable Weights:

Input

Multi-Head

Attention Block

(MHA)

Different Weight

Diagonal Matrices

Down

Projection

Up

Projection

Feed-Forward

Network Block

(FFN)

Representations

……

MHA

Identical Symmetric

Weight Matrices for FFN

FFN

Bottleneck Operation:

Bottleneck

for MHA

Bottleneck

for FFN

……

MHA

FFN

Bottleneck

for MHA

Bottleneck

for FFN

Weight Sharing

……

……

Layer

l

Layer

l

+ 1

Weight Sharing

Layer

l

Re-Scaling

Coefficients

This scheme involves sharing linear projections across layers and learning low-dimensional re-scaling coefficients to re-compose the layer-adaptive adapters. In addition, to enhance parameter compression, we leverage symmetric down-projection and up-projection in a single bottleneck operation:

where W down ∈ R D × D ′ and W up ∈ R D ′ × D with D ′ ≪ D denote shared down-projection and upprojection matrices across different layers; D ′ represents the hidden dimensionality of the projections. To accommodate the variations across different layers, we learn re-scaling coefficients to re-compose the layer-adaptive adaptation matrices. These coefficients are then diagonalized into a diagonal matrix C ( l ) ∈ R D ′ × D ′ specific to each layer l . This diagonal matrix allows for efficient and effective adjustment of the adaptation parameters at each layer. Formally, given an input X in ∈ R ( N +1) × D , the output of our ARC module is:

Unless otherwise specified, we adhere to the default configuration of inserting our ARC modules sequentially before both the MHA and FFN blocks in the Vision Transformer. The influence of adapter positions will be discussed in Section 4.3. Therefore, the Vision Transformer incorporating our ARC modules can be formulated as follows:

Note that ARC MHA and ARC FFN are two independent ARC modules, meaning that the projection matrices of the two modules are not shared between them. During the fine-tuning phase, we exclusively update the learnable parameters of our newly added ARC modules. This involves freezing all original parameters of the pre-trained model and solely focusing on updating the parameters of our ARC.

Inference. Our ARC employs a completely linear transformation so that we can re-parameterize it by fusing the module to the original framework of the pre-trained model. Take the ARC module of FNN as an example, the process can be defined by:

where ARC FFN ( X ( l ) ′ ) can be represented by X ( l ) ′ ( W down C ( l ) W up + I ) according to Eq. (7) with I ∈ R D × D being an identity matrix. Furthermore, we can construct W ( l ) ′ 1 by:

0.0

0.2

0.4

0.6

0

200

400

600



0.0

0.1

0.2

0.3

0.4

0

200

400



0.0

0.2

0.4

0.6

0

200

400



(a) The singular value distribution of MHA adapter.

0.0

0.2

0.4

0.6

0

200

400

600



0.0

0.2

0.4

0

200

400



0.0

0.2

0.4

0.6

0

200

400



(b) The singular value distribution of FFN adapter.

Therefore, we can replace the matrix W ( l ) 1 by W ( l ) ′ 1 in the inference stage, thereby avoiding extra computational overheads.

3.3 Insights of architecture design

In this work, we propose an approach that involves adopting a low-rank design for the adapters and sharing the bottleneck projections across layers. Our motivation for adopting this approach stems from the assumption that the layer-wise adapters can be effectively re-composed by re-scaling a small number of base linear projections. To validate this assumption, we conduct an analysis of the singular value distribution of adaptation matrices W full ∈ R D × D learned without the bottleneck operation, which are anticipated to be full-rank. In Fig. 3, we observe that the singular values exhibit extreme sparsity and follow a power-law distribution, with the majority of singular values being close to zero. This indicates that the learned adaptation matrices naturally exhibit low-rank characteristics. Furthermore, the pronounced sparsity of the singular values suggests that the adaptation matrices can be effectively reconstructed using a limited number of basis vectors. These findings provide strong evidence supporting the rationale behind our adaptation parameter-sharing strategy.

4 Experiments

In this section, we present the experimental settings, comparison to existing solutions, and ablation studies to unveil the key properties of the proposed method.

4.1 Experimental settings

Datasets. We evaluate the effectiveness of our ARC approach on two sets of visual task adaptation benchmarks, comprising a total of 24 datasets. The list of datasets used for evaluation is provided below:

FGVC. We conduct experiments with the default settings in the VPT [6] on a collection of five Fine-Grained Visual Classification (FGVC) datasets, known as FGVC. The FGVC dataset collection includes CUB-200-2011 [31], NABirds [32], Oxford Flowers [33], Stanford Dogs [34], and Stanford Cars [35].

VTAB-1k. We also evaluate our ARC method on the VTAB-1k benchmark [36], which consists of 19 diverse visual classification tasks. These tasks are divided into three groups: the Natural group, which contains images captured through standard cameras; the Specialized group, which includes images captured by specialist equipment such as remote sensing and medical imaging; and the Structured group, which comprises synthesized images from simulated environments, such as object counting and 3D depth prediction. Each downstream task in the VTAB-1k benchmark consists of 1000 training examples. Following VPT [6], we set aside 200 samples from the training set as the validation set to

select hyperparameters. Subsequently, we train the model on the full training data using the selected hyperparameters.

Pre-trained backbone. To evaluate the adaptation capacity of the proposed ARC method, we apply the ARC strategy to two typical types of Vision Transformers: ViT [1] and Swin Transformers [11]. For ViT, we conduct experiments using three different backbone variants with varying model sizes: ViTB ase/ L arge/ H uge. All the backbones are pre-trained on the ImageNet-21K [15] dataset.

Baselines and existing methods. In our comparative analysis, we evaluate the performance of the ARC method against two baselines and several state-of-the-art efficient pre-trained model adaptation methods. The two baselines we consider are: (1) Full Fine-tuning: This baseline involves updating all the parameters of the pre-trained model using the training data of the downstream task. (2) Linear Probing: This baseline focuses on learning a linear classification head on the downstream task while keeping the remaining parameters of the pre-trained model frozen. In addition to the baselines, we compare our ARC method with the following state-of-the-art solutions: (1) Adapter [7]: This method inserts lightweight adaptation operations, consisting of a down-projection, non-linear activation, and an up-projection, into the pre-trained model. (2) Bias [37]: The Bias method fine-tunes only the bias terms of the pre-trained models while keeping the remaining parameters frozen. (3) LoRA [24]: This approach introduces trainable low-rank adaptation matrices into each layer of the Transformer architecture. (4) VPT [6]: The VPT method incorporates extra learnable tokens into the input or all attention layers of the frozen Transformer. (5) SSF [9]: This method adds linear transformation parameters, including scaling and shifting, to modulate the pre-trained features. By comparing the performance of our ARC method with these baselines and state-of-the-art solutions, we aim to demonstrate its superiority in terms of efficiency and effectiveness in pre-trained model adaptation.

Implementation details. To ensure a fair evaluation of the effectiveness of our proposed ARC method, we have opted for a simple training setup without too many additional bells and whistles. Similar to VPT [6], we have used standard data augmentations during the training phase, which include image normalization using ImageNet means and standard deviation, random resize crop to 224 × 224 with random horizontal flip for FGVC datasets, and resize to 224 × 224 for VTAB-1k. We have used grid search to select hyper-parameters such as the learning rate, weight decay, and batch size, using the validation set of each task, as in VPT [6]. All experiments were conducted using the PyTorch [38] framework on an NVIDIA A40 GPU with 48GB of GPU memory. Further details can be found in the appendix.

4.2 Experimental comparisons

In this section, we provide a comprehensive comparison of our ARC method with baseline methods and other state-of-the-art solutions. We evaluate the performance in terms of classification accuracy on downstream tasks as well as the parameter size. The results are summarized in Table 1 and Table 2, with all results obtained using the ViT-B/16 backbone. Based on these comparative results, we make the following observations:

(1) The ARC method demonstrates highly competitive classification accuracy on both sets of visual adaptation datasets, while maintaining a low parameter size. As shown in Table 1 and Table 2, under a fair comparison, ARC achieves the best mean accuracy and outperforms the other methods on the majority of the 24 datasets. This confirms the effectiveness of our proposed pre-trained model adaptation strategy. Furthermore, thanks to the parameter-sharing strategy in ARC, we are able to noticeably reduce the parameter size compared to other rank-decomposition based adapters such as Adapter [7] and LoRA [24]. VPT-Shallow [6] also exhibits parameter efficiency as it only introduces learnable tokens in the input layer. However, this is achieved at the cost of a significant performance sacrifice, resulting in considerably inferior performance compared to VPT-deep [6] and ARC. Another parameter-efficient method, Bias [42], focuses on updating only the bias terms in the pre-trained network, but it also leads to a significant compromise in classification performance on downstream tasks. To further decrease the parameter size, we evaluate ARCatt, which omits the adapters applied to Feed-Forward Network (FFN) blocks and focuses solely on adapting the Multi-Head Attention (MHA) blocks. This approach achieves nearly a halving of the parameter size while experiencing only a minimal 1% drop in performance.

(2) In comparison to the efficient adaptation solutions presented in the tables, full fine-tuning yields comparable or even worse classification accuracy across various downstream tasks, despite updating a significantly larger number of parameters. This observation further emphasizes the importance and potential of lightweight adaptation designs. On the other end of the spectrum, linear probing requires minimal parameters but exhibits noticeable performance degradation.

Experiments on larger-scale ViT backbones. In addition to evaluating the ARC method on ViT-B/16 backbone, we also conducted experiments on larger-scale ViT backbones to assess its performance on more computationally demanding models. Specifically, we tested the ARC method on ViT-Large and ViT-Huge backbones. The results, presented in Table 3a and Table 3b, demonstrate that the ARC method maintains its competitive classification accuracy even with larger-scale backbones. It consistently outperforms the baseline methods and achieves comparable or superior performance compared to other state-of-the-art adaptation methods. Furthermore, the parameter size of the ARC method remains noticeably smaller than rank-decomposition based adapters like Adapter [7] and LoRA [24], as well as VPT-deep [6], showcasing its efficiency in terms of parameter utilization. These findings suggest that the ARC method is not only effective on smaller-scale ViT backbones

(a) ViT-Large

(b) ViT-Huge

but also scalable to larger models, making it a promising solution for efficient pre-trained model adaptation across a wide range of backbone sizes.

Experiments on hierarchical Vision Transformers. We extended the ARC method to Swin Transformer [11], a hierarchical Transformer architecture. To accommodate the varying feature dimensionalities in Swin Transformer, we introduced a stage-sharing strategy, enabling parameter sharing within each stage. The results on the VTAB-1k benchmark (Table 4) demonstrate the generalizability of ARC. It achieves competitive transfer learning accuracy and maintains favorable parameter scale, except for the Natural group where ARC performs relatively weaker. These findings highlight ARC's versatility and effectiveness in adapting different transformer architectures, showcasing its potential for practical applications in visual adaptation.

4.3 Ablation studies

In order to gain deeper insights into the ARC method, we performed ablation studies to explore its additional properties. The experiments were conducted on the VTAB-1k benchmark using the pre-trained ViT-B/16 model. The results of these ablation studies are presented in Table 5.

Bottleneck dimensionality. In our ARC method, we adopt the low-rank design for the adapters but with the added feature of sharing the down-/up-projection matrices across layers and learning lowdimensional re-scaling coefficients to re-compose adaptation matrices. In this section, we investigate the impact of the bottleneck dimensionality on adaptation performance. The results are presented in Table 5a. We find that a dimensionality of 50 achieves the best balance between transfer learning performance and parameter efficiency. Further reduction to a 10-dimensional space leads to fewer parameters but at the cost of performance degradation. Conversely, higher-dimensional hidden spaces result in inferior performance. These findings validate the effectiveness of our low-rank design, with 50 linear projections providing sufficient flexibility for composing layer-adaptive adapters.

Adapter positioning. By default, our adapters are positioned before the MHA and FFN modules, allowing the adaptation operations to be seamlessly integrated into the pre-trained network during inference without additional inference cost. In this section, we investigate the impact of adapter

position on pre-trained model adaptation performance using different positioning strategies. The results are presented in Table 5b. Interestingly, placing the adapters after the MHA and/or FFN modules leads to performance degradation, despite this strategy being commonly adopted in previous works such as[7; 29]. Moreover, using only one type of adapter for either MHA or FFN results in inferior performance compared to using both types of adapters. This suggests that employing both types of adapters allows for more comprehensive adaptation of the pre-trained model to the target task without significantly increasing the parameter count.

Sharing v.s non-sharing adaptation parameters. In ARC, we adopt a parameter-sharing strategy to effectively reduce the number of adaptation parameters. This strategy encompasses two aspects: intra-layer sharing and inter-layer sharing. Through symmetric down-projection and up-projection matrices, we achieve intra-layer sharing, while inter-layer sharing involves sharing projection matrices across different layers. In this section, we investigate the impact of adaptation parameter sharing by conducting experiments with non-sharing or partial sharing designs. 'intra + inter*' denotes sharing the bottleneck structure between MHA and FFN. The results presented in Table 5c demonstrate that using non-symmetric projection matrices or layer-independent adaptation parameters does not result in performance gains but leads to a noticeable increase in parameters. This validates the effectiveness of our parameter-sharing design.

Adapter insertion. We examine the performance of inserting the proposed adapters in a subset of layers using sequential or parallel insertion strategies. The results in Table 5d show that the performance of ARC improves as more layers are inserted. Furthermore, we observe that inserting adapters in the first six layers yields better results compared to inserting them in the last six layers. Additionally, we explore a parallel insertion setting inspired by [24], but the impact is not significantly pronounced. Another notable aspect is that our parameter sharing scheme in the ARC method prevents a linear increase in parameter size with the number of layers, ensuring better scalability, particularly for larger-scale models.

5 Limitation

The adaptation parameter sharing scheme in the ARC method is built on the assumption that layers have the same dimensionality. This assumption is crucial as it enables the sharing of down-/upprojection matrices involved in the bottleneck operation across layers, leading to parameter efficiency. However, it is worth exploring strategies to extend this scheme and accommodate dimensionality variation. This research direction holds promise for addressing scenarios where dimensionality varies and further enhancing the flexibility and applicability of the ARC method.

6 Conclusions

Our paper introduced the Adapter Re-Composing (ARC) method, which leverages the reusability of adaptation parameters to efficiently adapt pre-trained models. By sharing down-/up-projections in lowrank adapters across layers and learning layer-specific re-scaling coefficients to re-composing layeradaptive adapters, ARC balances transfer learning performance and adaptation overheads. Extensive experiments demonstrate the compelling performance of our approach with a reduced parameter size. ARC offers a promising solution for efficient pre-trained model adaptation, showcasing the potential of reusing adaptation parameters for competitive results.

References

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , 'An image is worth 16x16 words: Transformers for image recognition at scale,' in International Conference on Learning Representations , 2020.

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, 'Masked autoencoders are scalable vision learners,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 16 000-16 009.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , 'Learning transferable visual models from natural language supervision,' in International conference on machine learning . PMLR, 2021, pp. 8748-8763.

X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer, 'Scaling vision transformers,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 12 10412 113.

K. Zhou, J. Yang, C. C. Loy, and Z. Liu, 'Conditional prompt learning for vision-language models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 16 816-16 825.

M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, 'Visual prompt tuning,' in Proceedings of the European conference on computer vision (ECCV) . Springer, 2022, pp. 709-727.

N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, 'Parameter-efficient transfer learning for nlp,' in International Conference on Machine Learning . PMLR, 2019, pp. 2790-2799.

S. Chen, C. Ge, Z. Tong, J. Wang, Y. Song, J. Wang, and P. Luo, 'Adaptformer: Adapting vision transformers for scalable visual recognition,' in Advances in Neural Information Processing Systems (NeurIPS) , 2022.

D. Lian, D. Zhou, J. Feng, and X. Wang, 'Scaling & shifting your features: A new baseline for efficient model tuning,' in Advances in Neural Information Processing Systems (NeurIPS) , 2022.

G. Luo, M. Huang, Y. Zhou, X. Sun, G. Jiang, Z. Wang, and R. Ji, 'Towards efficient visual adaption via structural re-parameterization,' arXiv preprint arXiv:2302.08106 , 2023.

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, 'Swin transformer: Hierarchical vision transformer using shifted windows,' in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 10 012-10 022.

F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He, 'A comprehensive survey on transfer learning,' Proceedings of the IEEE , vol. 109, no. 1, pp. 43-76, 2020.

S. J. Pan and Q. Yang, 'A survey on transfer learning,' IEEE Transactions on knowledge and data engineering , vol. 22, no. 10, pp. 1345-1359, 2010.

M. Iman, H. R. Arabnia, and K. Rasheed, 'A review of deep transfer learning and recent advancements,' Technologies , vol. 11, no. 2, p. 40, 2023.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'Imagenet: A large-scale hierarchical image database,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Ieee, 2009, pp. 248-255.

T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor, 'Imagenet-21k pretraining for the masses,' in Advances in Neural Information Processing Systems (NeurIPS) , 2021.

C. Sun, A. Shrivastava, S. Singh, and A. Gupta, 'Revisiting unreasonable effectiveness of data in deep learning era,' in Proceedings of the IEEE international conference on computer vision , 2017, pp. 843-852.

D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. Van Der Maaten, 'Exploring the limits of weakly supervised pretraining,' in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 181-196.

W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev et al. , 'The kinetics human action video dataset,' arXiv preprint arXiv:1705.06950 , 2017.

K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang, 'Transformer in transformer,' Advances in Neural Information Processing Systems (NeurIPS) , vol. 34, pp. 15 908-15 919, 2021.

D. Zhou, B. Kang, X. Jin, L. Yang, X. Lian, Z. Jiang, Q. Hou, and J. Feng, 'Deepvit: Towards deeper vision transformer,' arXiv preprint arXiv:2103.11886 , 2021.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , 'Language models are few-shot learners,' Advances in Neural Information Processing Systems (NeurIPS) , vol. 33, pp. 1877-1901, 2020.

X. Chen, S. Xie, and K. He, 'An empirical study of training self-supervised vision transformers,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 9640-9649.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, 'Lora: Low-rank adaptation of large language models,' arXiv preprint arXiv:2106.09685 , 2021.

B. Lester, R. Al-Rfou, and N. Constant, 'The power of scale for parameter-efficient prompt tuning,' in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , 2021, pp. 3045-3059.

X. L. Li and P. Liang, 'Prefix-tuning: Optimizing continuous prompts for generation,' arXiv preprint arXiv:2101.00190 , 2021.

P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, 'Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,' ACM Computing Surveys , vol. 55, no. 9, pp. 1-35, 2023.

T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh, 'Autoprompt: Eliciting knowledge from language models with automatically generated prompts,' arXiv preprint arXiv:2010.15980 , 2020.

Y.-L. Sung, J. Cho, and M. Bansal, 'Vl-adapter: Parameter-efficient transfer learning for visionand-language tasks,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 5227-5237.

Y. Zhang, K. Zhou, and Z. Liu, 'Neural prompt search,' arXiv preprint arXiv:2206.04673 , 2022.

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, 'The caltech-ucsd birds-200-2011 dataset,' 2011.

G. Van Horn, S. Branson, R. Farrell, S. Haber, J. Barry, P. Ipeirotis, P. Perona, and S. Belongie, 'Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 595-604.

M.-E. Nilsback and A. Zisserman, 'Automated flower classification over a large number of classes,' in 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing . IEEE, 2008, pp. 722-729.

A. Khosla, N. Jayadevaprakash, B. Yao, and F.-F. Li, 'Novel dataset for fine-grained image categorization: Stanford dogs,' in Proc. CVPR workshop on fine-grained visual categorization (FGVC) , vol. 2, no. 1. Citeseer, 2011.

T. Gebru, J. Krause, Y. Wang, D. Chen, J. Deng, and L. Fei-Fei, 'Fine-grained car detection for visual census estimation,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 31, no. 1, 2017.

X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy et al. , 'A large-scale study of representation learning with the visual task adaptation benchmark,' arXiv preprint arXiv:1910.04867 , 2019.

E. B. Zaken, Y. Goldberg, and S. Ravfogel, 'Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,' in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , 2022, pp. 1-9.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al. , 'Pytorch: An imperative style, high-performance deep learning library,' Advances in Neural Information Processing Systems (NeurIPS) , vol. 32, pp. 8026-8037, 2019.

S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, 'Cutmix: Regularization strategy to train strong classifiers with localizable features,' in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 6023-6032.

H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, 'mixup: Beyond empirical risk minimization,' arXiv preprint arXiv:1710.09412 , 2017.

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, 'Rethinking the inception architecture for computer vision,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2016, pp. 2818-2826.

H. Cai, C. Gan, L. Zhu, and S. Han, 'Tinytl: Reduce memory, not parameters for efficient on-device learning,' Advances in Neural Information Processing Systems (NeurIPS) , vol. 33, pp. 11 285-11 297, 2020.

Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing

Supplementary Materials

In the supplementary materials involving our work, we demonstrate detailed dataset settings, supplemental insights and analysis, extra experimental details, supplemental experiments, and broader impacts, including:

A Detailed descriptions for datasets and implementation

B Insights of architecture design

C Parameter size analysis

D Experimental details on larger-scale and hierarchical ViT backbones

E Experimental details on ablation studies

F Expanded experiments with self-supervised pre-training

G Broader impacts

Due to the limitation that the file 'Supplementary Materials.zip' larger than 100MB cannot be uploaded on OpenReview, the supplementary materials only upload the code for the project. Please refer to the anonymous link https://drive.google.com/file/d/ 1ZblHbYF1Jr0u0GeTLI4uII6GHt3CV3I2/view to obtain the complete code, datasets, and models.

A Detailed descriptions for datasets and implementation

We describe the details of visual adaptation classification tasks in Table 6 (FGVC) and 7 (VTAB-1k), including the class number and the train/val/test sets.

Table 8 summarizes the detailed configurations we used for experiments. As mentioned in Section 4.1, we utilize grid search to select hyper-parameters such as learning rate, weight decay, batch size, and adapter dropout, using the validation set of each task. Note that we also apply dropout to the middle features produced by our ARC method, which we term as "adapter dropout". Specifically, during the ARC process, we randomly drop partial features before up-projection.

B Insights of architecture design

Similar to Fig. 3, we present more visualization results of singular value distribution of adaptation matrices W full ∈ R D × D learned without the bottleneck operation. As shown in Fig. 4, the singular value distribution of adaptation matrices learned on DTD downstream task exhibits a power-law



































































































































































































































































































































































































































































(a) Singular value distribution of MHA adapter.

(b) Singular value distribution of FFN adapter.

C Parameter size analysis

To showcase the parameter-efficiency of our ARC method, we compare its parameter size with other popular lightweight adaptation methods (Table 9), including Adapter [7], VPT [6], LoRA [24], and SSF [9]. Adapter [7] adds two linear projections to each encoder layer during fine-tuning, resulting in the introduction of 2 · D · D ′ · L learnable parameters, where D ′ denotes the hidden dimensionality of the linear projections. Furthermore, due to the presence of non-linear activations in Adapter, the additional parameters contribute to supernumerary overhead during the inference phase. VPT [6] incorporates m prompts into input space, leading to an increase of m · D parameters for VPT-Shallow and m · D · L parameters for VPT-Deep. In contrast to Adapter, both LoRA [24] and SSF [9] employ linear adaptation methods without incorporating non-linear functions. This design choice allows them to leverage re-parameterization benefits, thereby mitigating additional computations during inference. Specifically, the adaptation matrix of LoRA, which consists of a down-projection and an up-projection, introduces 2 · w · D · D ′ · L learnable parameters, where w denotes the number of attention matrices undergoing adaptation. SSF inserts linear scaling and shifting coefficients after

o operations, resulting in an addition of 2 · o · D · L extra parameters. The proposed ARC method offers additional parameter compression by sharing symmetric projection matrices across different layers. This approach introduces only D · D ′ parameters. Additionally, we learn low-dimensional re-scaling coefficients and bias terms for each layer, resulting in a total of ( D ′ + D ) · L additional parameters. Overall, the number of parameters in our default ARC is 2 · (( D · D ′ ) + ( D ′ + D ) · L ) .

We also compare the parameter size with lightweight adaptation methods on backbones of different scales, as shown in Fig. 5. Our ARCs demonstrate parameter efficiency across various model sizes, comparable to VPT-Shallow [6]. However, the unique advantage of our approach lies in its ability to effectively balance lower overheads and maintain competitive performance. Furthermore, the parameter count of our ARC remains stable even as the model scale increases, showcasing the scalability of our method with minimal additional resource consumption.

0

1

2

3

4

5

6

Adapter

VPT-Shallow

VPT-Deep

LoRA

SSF

ARCatt

ARC

ViT-Base

ViT-Large

Vit-Huge

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Adapter

VPT-Shallow

VPT-Deep

LoRA

SSF

ARCatt

ARC

0

1

2

3

4

5

6

7

8

9

10

11

12

Thanks to our adaptation parameter sharing strategy, the ARC method avoids a linear increase in the number of learnable parameters as the number of layers grows. We employ ViT-B as the backbone and integrate adapters into different layers. As shown in Fig.6, in contrast to other adaptation methods, both our ARCs and VPT-Shallow[6] effectively manage parameter growth as the number of inserted layers increases, but only our methods achieve promising performance without significant cost escalation. This highlights the scalability and effectiveness advantages of our ARCs.

D Experimental details on larger-scale and hierarchical ViT backbones

Table 10, 11 and 12 respectively display the comprehensive results of the comparison conducted in Section 4.2 among ViT-Large, ViT-Huge, and Swin-Base models.

E Experimental details on ablation studies

Table 13, 14, 15 and 16 display the complete results of the ablation studies in Section 4.3.

F Expanded experiments with self-supervised pre-training

In addition to the models pre-trained with supervised objectives in Section 4, we also conduct experiments with self-supervised pre-training approaches: MAE [2] and Moco V3 [23]. Specifically, We utilize MAE [2] and Moco V3 [23] self-supervised pre-trained ViT-B as the backbone and evaluate the performance of our ARC on VTAB-1k. The results of MAE and Moco V3 selfsupervised models are presented in Table 17 and Table 18, respectively. We observe that our ARC still exhibits competitive performance on two self-supervised ViTs. In addition, our ARC method outperforms other adaptation methods: Adapter[7] and LoRA [24] on the majority of downstream tasks. Surprisingly, the ARC att with smaller learnable parameters even surpasses the ARC across different self-supervised pre-trained models. A possible explanation could be that ARC att contains fewer parameters, which allows it to effectively prevent overfitting.

G Broader impacts

Efficient usability. Unlike previous approaches, our method incorporates a parameter sharing scheme across different layers of the model, resulting in a significant reduction in the number of parameters that need to be fine-tuned. This approach allows us to maintain competitive performance while achieving parameter efficiency. By maximizing the utilization of large-scale pre-trained models, our ARC methods offer enhanced usability and practicality in various applications.

Environmental-friendly consumption. In addition to the reduction in computational overheads, another significant benefit of our method is the positive impact on carbon emissions reduction and environmental protection. By optimizing the computational efficiency of the model, we minimize the energy consumption required during the training and deployment of the model. This reduction in energy consumption leads to a decrease in carbon emissions, contributing to environmental sustainability. Our method not only delivers improved performance and efficiency but also aligns with the larger goal of mitigating the environmental impact of AI technologies.

Ethical Considerations. Our model focuses on utilizing the representation and generalization capacity obtained from large-scale pre-trained datasets and models. However, it is crucial to acknowledge that if the pre-training datasets contain bias or illegal information, there is a risk of inheriting such issues into our model.

In order to address this concern, it becomes imperative to explore research directions that aim to identify and prevent privacy leakage and correct model bias. This involves developing robust mechanisms to detect and mitigate bias in training data, as well as implementing privacy-preserving techniques to safeguard sensitive information.

Peer reviews:
comments: SUMMARY:
This work presents an efficient adapter design for transfer learning of large retrained models. The key idea is to enable adapter to be shared across layers and imposing low-rank constraint, so that the overall trainable parameters can be reduced. Given the re-composed linear adapter, existing reparameterization methods can be further used to avoid extra computation cost.

STRENGTHS:
- Parameter efficient transfer learning is an important approach for making large pertained model useful in many applications.

- The paper is written to be easy to read.

WEAKNESSES:
First of all, the motivation of this work is weak -- Existing adapters are already lightweight, e.g., less than 0.5% of the pretrained model, and the scope for further reducing the size of adapter is marginal. That being said, this adapter efficiency problem is not significant. 

In introduction, it is hard to read out the novelty of this method since more related works such LoRA is not discussed and compared at all.

Limited novelty with the proposed method: Low-rank constraint is not new, as already used in LoRA [24] although not in the same way, and parameter sharing across layers is also not novel idea. This work combines the two in a single place, which could be considered as being not significant.

Limited performance gain: 
- In most cases, the proposed method cannot achieve a good margin over previous methods (e.g., SSF). 

- Another important metric, FLOPs, would be useful to report.

QUESTIONS:
See the weaknesses above.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a parameter-efficient transfer learning method named Adapter Re-Composing (ARC), which mainly focuses on investigating the reusability of adapted parameters. The authors propose to apply a shared adapter to all the layers (blocks) of the pre-trained model, and they use different Re-Scaling Coefficients (diagonal matrices) in different layers to ensure the diversity of parameters in different layers.\nThe motivation behind this design is the adapter module's low-rank property, shown in Fig.3 in the main text.\nMoreover, the authors conduct extensive experiments to demonstrate the effectiveness of their designs, where they train fewer parameters and achieve competitive or even better performance compared to prior arts.\n\nSTRENGTHS:\n1: This paper is well-written and well-organized.\n\n2: The ARC design in the paper is well-motivated.\n\n3: Although the technic is simple and easy to implement, the performance is impressive.\n\n4: The experiments are comprehensive.\n\nWEAKNESSES:\n1: The ARC leverages learnable re-scaling coefficients in different layers to maintain diversity. However, the authors didn't disscuss the numerical difference among the re-scaling coefficients across different layers. If they are also similar, a share-weight adapter can replace the ARC. \n\n2: In Tab. 1 and Tab. 2, the SSF* applies additional techniques (e.g., data augmentation) during training and gains non-trivial improvements. Why did the authors not use those techniques to improve the performance of ARC further? Will these techniques further benefit the ARC?\n\n3: I wonder about the computational overhead of the ARC, e.g., the GPU memory usage during training, because the number of learnable parameters may not necessarily be positively correlated with the GPU memory usage. Could authors provide the additional GPU memory usage comparison between ARC and other related works?\n\n4: How to use ARC in Hierarchical Vision Transformers such as Swin Transformer is unclear in Lines 277-279. Adding more details here may be helpful.\n\nOverall, my major concern is weakness 1. Happy to raise my rating if my concers are well adressed.\n\nQUESTIONS:\nSee the weakness.\n\nLIMITATIONS:\nThe authors disscuss the limitations and boarder impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores ARC, which is a novel parameter-efficient fine-tuning method which uses a similar architecture as adapters but introduces inter- and intra- layer weight sharing. Some down- and up- projection weights are shared but every adapter position uses an independent set of per-channel scaling factor on the channel-reduced intermediate features. The proposed method obtains competitive results on various vision transformer adaptation benchmarks in terms of recognition performance and number of parameters. Moreover it can be re-parameterized into adjacent fully-connected layers so no overhead is incurred during inference.\n\nSTRENGTHS:\n* The paper is generally well written. The figures are clear and the text is easy to follow.\n\n* The experiments are comprehensive and cover a wide range of visual recognition datasets and the results are competitive.\n\nWEAKNESSES:\n* The effectiveness of the proposed method is not well justified. Section 3.3 does not make much sense to me: Having long-tailed singular values only implies that the weight difference can be approximated well by a low-rank matrix and thus justifies the bottleneck design. Shared projections further requires the adapters to have largely overlapped kernel and image spaces, which, instead of the singular values themselves, are determined by the direction of the singular vectors corresponding to the top singular values. It also doesn't theoretically justify the intra-block sharing design (i.e., using the transpose of down-projection as up-projection).\n\n* In multiple places (e.g., caption of Table 1 and Line 229), the paper's claim of using simple augmentations on baselines for 'fair comparison' is questionable. Due to the vast differences in nature of different methods, it is expected that their optimal training configurations are different, and each method, including this paper's own, should have the right to choose its optimal training configuration as long as it does not violate some principal rules of machine learning (e.g., leak of test data): Conversely, it is also improper to request that the paper's method being run under SSF's data augmentation for 'fair comparison'.\n\nQUESTIONS:\n* The paper cited [7] in multiple tables for visual adapter results but [7] seems to not include any results on computer vision tasks (actually the paper title is *Parameter-efficient transfer learning for **NLP***). Are those results reproduced by the authors using the same architecture as [7]? If so, could the authors point to a place where more detailed settings (e.g., the bottleneck dimension, the activation function, the training hyper-parameters) of this baseline can be found?\n\n* In Table 3, it is strange to me why the proposed methods obtain lower performance on ViT-Huge than ViT-Large. Is it possible that the proposed method or some baselines are over-fitting in extremely large backbones?\n\n* Following weakness 1, an understanding of why the proposed method works fairly well can also be provided by experiments: For example, are features from adjacent layers similar due to the identity connections? What if the projections are progressively shared in groups of increasing sizes among consecutive layers before reaching the global sharing setting as reported in the paper?\n\nLIMITATIONS:\nThere are no unmentioned limitations in the paper to my mind.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposes to further reduce the parameters of the adapter by introducing a weight-sharing scheme between different layers. To accommodate the variations across different layers, re-scaling coefficients are learned to re-compose the layer-adaptive adaptation matrices. Experiments are conducted on 24 downstream image classification tasks using various Vision Transformer variants.

STRENGTHS:
+ The observation that “learned adaptation matrices naturally exhibit low-rank characteristics” as shown in Fig. 3 is quite interesting, making the motivation for the weight-sharing design compelling. 
+ This paper is well-written and easy to follow. The figures are well-prepared and illustrate the core idea clearly. 
+ The experiments are extensive.

WEAKNESSES:
- The comparisons to SSF shown in Tables 1 and 2 are re-implemented by the authors and data augmentations are removed. Their original performances of SSF are much higher than the proposed approach. It would be better if the authors reported the performance of the proposed approach with these advanced data augmentations.
- Fig. 3 shows the singular value distribution of the original MHA and FFN adapter. It would be better to show how the proposed approach alleviates such a problem by visualizing the value distribution after using the proposed approach.
- Though the observation of “learned adaptation matrices naturally exhibit low-rank characteristics” and the proposed weight-sharing scheme is very interesting, the motivation to further reduce the parameters of the adapter is not very compelling as the parameters of the adapter are already relatively small.

QUESTIONS:
Please see the weaknesses

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel parameter-efficient fine-tuning method called Adapter Re-Composing (ARC). ARC effectively reuses parameters across different layers, resulting in remarkable improvements in performance across 24 image classification datasets while utilizing fewer learnable parameters.\nThe experimental evaluation conducted on various downstream datasets provides compelling evidence of ARC's superiority. It outperforms existing methods and establishes a new benchmark for parameter-efficient fine-tuning techniques.\n\nSTRENGTHS:\n1. The proposed ARC method is simple yet effective, achieving superior performance on multiple downstream datasets while utilizing fewer trainable parameters.\n\n2. The experiments conducted provide compelling evidence, as they encompass various datasets, attention-based architectures, and ablations, ensuring the robustness and reliability of the findings.\n\nWEAKNESSES:\n1. In Table 3, it is observed that the performance of ViT-Huge is lower than ViT-Large. It would be beneficial if the authors could provide an explanation for this disparity.\n\n2. The utilization of symmetric matrices ($W_{up} = W_{down}^T$) in the bottleneck design helps reduce the number of learnable parameters. However, it would be interesting to explore whether further improvements in performance can be achieved by making the downsampling and upsampling matrices independent. It would be valuable if the authors could provide a comparison of performance and parameter statistics to address this potential enhancement.\n\n3. The experiments conducted have demonstrated the effectiveness of the proposed method. However, it would greatly enhance the strength of this paper if the authors could supplement these empirical results with theoretical analysis.\n\nQUESTIONS:\nPlease see Weaknesses.\n\nLIMITATIONS:\nPlease see Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-18"}