SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pre-trained backbone. To evaluate the adaptation capacity of the proposed ARC method, we apply the ARC strategy to two typical types of Vision Transformers: ViT [1] and Swin Transformers [11]. For ViT, we conduct experiments using three different backbone variants with varying model sizes: ViT-Base/Large/Huge. All the backbones are pre-trained on the ImageNet-21K [15] dataset.
Baselines and existing methods. In our comparative analysis, we evaluate the performance of the ARC method against two baselines and several state-of-the-art efficient pre-trained model adaptation methods. The two baselines we consider are: (1) Full Fine-tuning: This baseline involves updating all the parameters of the pre-trained model using the training data of the downstream task. (2) Linear Probing: This baseline focuses on learning a linear classification head on the downstream task while keeping the remaining parameters of the pre-trained model frozen. In addition to the baselines, we compare our ARC method with the following state-of-the-art solutions: (1) Adapter [7]: This method inserts lightweight adaptation operations, consisting of a down-projection, non-linear activation, and an up-projection, into the pre-trained model. (2) Bias [37]: The Bias method fine-tunes only the bias terms of the pre-trained models while keeping the remaining parameters frozen. (3) LoRA [24]: This approach introduces trainable low-rank adaptation matrices into each layer of the Transformer architecture. (4) VPT [6]: The VPT method incorporates extra learnable tokens into the input or all attention layers of the frozen Transformer. (5) SSF [9]: This method adds linear transformation parameters, including scaling and shifting, to modulate the pre-trained features. By comparing the performance of our ARC method with these baselines and state-of-the-art solutions, we aim to demonstrate its superiority in terms of efficiency and effectiveness in pre-trained model adaptation.
Implementation details. To ensure a fair evaluation of the effectiveness of our proposed ARC method, we have opted for a simple training setup without too many additional bells and whistles. Similar to VPT [6], we have used standard data augmentations during the training phase, which include image normalization using ImageNet means and standard deviation, random resize crop to  $224 \times 224$  with random horizontal flip for FGVC datasets, and resize to  $224 \times 224$  for VTAB-1k. We have used grid search to select hyper-parameters such as the learning rate, weight decay, and batch size, using the validation set of each task, as in VPT [6]. All experiments were conducted using the PyTorch [38] framework on an NVIDIA A40 GPU with 48GB of GPU memory. Further details can be found in the appendix.
Table 2: Comparison of ARC with baselines and state-of-the-art efficient adaptation methods on VTAB-1k benchmark. All methods utilize ViT-B/16 pre-trained on ImageNet-21k as the backbone. To ensure a fair comparison, we reproduced the SSF method using the code provided by [9], while employing the same basic data augmentations as our approach.
Dataset Method	Natural	Specialized	Structured
CIFAR-100	Caltech101	DTD	Flowers102	Pets	SVNH	Sim397	Mean	Camelyon	EuroSAT	Resics45	Retinopathy	Mean	Clev-Count	Clev-Dist	DMLab	KITTI-Dist	dSpr-Loc	dSpr-Ori	sNORB-Azim	sNORB-Ele	Mean	Mean Total	Params.(M)
Full fine-tuning	68.9	87.7	64.3	97.2	86.9	87.4	38.8	75.9	79.7	95.7	84.2	73.9	83.4	56.3	58.6	41.7	65.5	57.5	46.7	25.7	29.1	47.6	65.6	85.8
Linear probing	63.4	85.0	63.2	97.0	86.3	36.6	51.0	68.9	78.5	87.5	68.6	74.0	77.2	34.3	30.6	33.2	55.4	12.5	20.0	9.6	19.2	26.9	52.9	0.04
Adapter [7]	74.1	86.1	63.2	97.7	87.0	34.6	50.8	70.5	76.3	88.0	73.1	70.5	77.0	45.7	37.4	31.2	53.2	30.3	25.4	13.8	22.1	32.4	55.8	0.27
Bias [42]	72.8	87.0	59.2	97.5	85.3	59.9	51.4	73.3	78.7	91.6	72.9	69.8	78.3	61.5	55.6	32.4	55.9	66.6	40.0	15.7	25.1	44.1	62.1	0.14
VPT-Shallow [6]	77.7	86.9	62.6	97.5	87.3	74.5	51.2	76.8	78.2	92.0	75.6	72.9	79.7	50.5	58.6	40.5	67.1	68.7	36.1	20.2	34.1	47.0	64.9	0.11
VPT-Deep [6]	78.8	90.8	65.8	98.0	88.3	78.1	49.6	78.5	81.8	96.1	83.4	68.4	82.4	68.5	60.0	46.5	72.8	73.6	47.9	32.9	37.8	55.0	69.4	0.60
LoRA [24]	65.3	87.9	69.4	98.7	90.7	82.4	53.4	78.2	82.8	94.8	82.5	75.0	83.8	77.6	64.7	45.8	79.0	73.3	44.7	26.3	38.2	56.2	70.1	0.29
SSF [9]	58.0	89.8	70.5	98.9	90.2	90.5	52.9	78.7	86.7	95.2	86.4	75.4	85.9	68.2	61.0	52.8	80.7	77.3	48.5	27.6	31.1	55.9	70.6	0.24
SSF* [9]	69.0	92.6	75.1	99.4	91.8	90.2	52.9	81.6	87.4	95.9	87.4	75.5	86.6	75.9	62.3	53.3	80.6	77.3	54.9	29.5	37.9	59.0	73.1	0.24
\( \mathrm{ARC}_{\mathrm{att}} \)	70.1	90.5	70.5	98.8	90.8	88.6	53.6	80.4	84.6	95.5	86.6	75.5	85.6	79.0	65.6	48.6	81.3	75.1	48.7	29.1	39.6	58.4	72.2	0.08
ARC	72.2	90.1	72.7	99.0	91.0	91.9	54.4	81.6	84.9	95.7	86.7	75.8	85.8	80.7	67.1	48.7	81.6	79.2	51.0	31.4	39.9	60.0	73.4	0.13
ARC*	71.2	90.9	75.9	99.5	92.1	90.8	52.0	81.8	87.4	96.5	87.6	76.4	87.0	83.3	61.1	54.6	81.7	81.0	57.0	30.9	41.3	61.4	74.3	0.13
4.2 Experimental comparisons
In this section, we provide a comprehensive comparison of our ARC method with baseline methods and other state-of-the-art solutions. We evaluate the performance in terms of classification accuracy on downstream tasks as well as the parameter size. The results are summarized in Table 1 and Table 2, with all results obtained using the ViT-B/16 backbone. Based on these comparative results, we make the following observations:
(1) The ARC method demonstrates highly competitive classification accuracy on both sets of visual adaptation datasets, while maintaining a low parameter size. As shown in Table 1 and Table 2, under a fair comparison, ARC achieves the best mean accuracy and outperforms the other methods on the majority of the 24 datasets. This confirms the effectiveness of our proposed pre-trained model adaptation strategy. Furthermore, thanks to the parameter-sharing strategy in ARC, we are able to noticeably reduce the parameter size compared to other rank-decomposition based adapters such as Adapter [7] and LoRA [24]. VPT-Shallow [6] also exhibits parameter efficiency as it only introduces learnable tokens in the input layer. However, this is achieved at the cost of a significant performance sacrifice, resulting in considerably inferior performance compared to VPT-deep [6] and ARC. Another parameter-efficient method, Bias [42], focuses on updating only the bias terms in the pre-trained network, but it also leads to a significant compromise in classification performance on downstream tasks. To further decrease the parameter size, we evaluate  $\mathrm{ARC}_{\mathrm{att}}$  which omits the adapters applied to Feed-Forward Network (FFN) blocks and focuses solely on adapting the Multi-Head Attention (MHA) blocks. This approach achieves nearly a halving of the parameter size while experiencing only a minimal  $1\%$  drop in performance.
(2) In comparison to the efficient adaptation solutions presented in the tables, full fine-tuning yields comparable or even worse classification accuracy across various downstream tasks, despite updating a significantly larger number of parameters. This observation further emphasizes the importance and potential of lightweight adaptation designs. On the other end of the spectrum, linear probing requires minimal parameters but exhibits noticeable performance degradation.
Experiments on larger-scale ViT backbones. In addition to evaluating the ARC method on ViT-B/16 backbone, we also conducted experiments on larger-scale ViT backbones to assess its performance on more computationally demanding models. Specifically, we tested the ARC method on ViT-Large and ViT-Huge backbones. The results, presented in Table 3a and Table 3b, demonstrate that the ARC method maintains its competitive classification accuracy even with larger-scale backbones. It consistently outperforms the baseline methods and achieves comparable or superior performance compared to other state-of-the-art adaptation methods. Furthermore, the parameter size of the ARC method remains noticeably smaller than rank-decomposition based adapters like Adapter [7] and LoRA [24], as well as VPT-deep [6], showcasing its efficiency in terms of parameter utilization. These findings suggest that the ARC method is not only effective on smaller-scale ViT backbones
Table 3: Performance comparison on VTAB-1k using ViT-Large and ViT-Huge pre-trained on ImageNet-21k as backbone. “(\cdot)” denotes the number of tasks in the subgroup. Expanded results are presented in the appendix.
(a) ViT-Large
Natural (7)	Specialized (4)	Structured (8)	Mean Total	Params.
Full fine-tuning	74.7	83.8	48.1	65.4	303.4
Linear probing	70.9	69.1	25.8	51.5	0.05
Adapter [7]	68.6	73.5	29.0	52.9	2.38
Bias [37]	70.5	73.8	41.2	58.9	0.32
VPT-Shallow [6]	78.7	79.9	40.6	62.9	0.15
VPT-Deep [6]	82.5	83.9	54.1	70.8	0.49
LoRA [24]	81.4	85.0	57.3	72.0	0.74
ARC	82.3	85.6	57.3	72.5	0.18
(b) ViT-Huge
Natural (7)	Specialized (4)	Structured (8)	Mean Total	Params.
Full fine-tuning	70.9	83.6	46.0	63.1	630.9
Linear probing	67.9	79.0	26.1	52.7	0.06
Adapter [7]	68.1	76.4	24.5	51.5	5.78
Bias [42]	70.3	78.9	41.7	60.1	0.52
VPT-Shallow [6]	74.8	81.2	43.0	62.8	0.18
VPT-Deep [6]	77.9	83.3	52.2	68.2	0.96
LoRA [24]	77.1	83.5	55.4	69.3	1.21
ARC	79.1	84.8	53.7	69.6	0.22
Table 4: Performance comparison on VTAB-1k using Swin-Base pre-trained on ImageNet-21k as backbone. “(\cdot)” denotes the number of tasks in the subgroup. Expanded results are presented in the appendix.
Natural (7)	Specialized (4)	Structured (8)	Mean Total	Params.
Full fine-tuning	79.1	86.2	59.7	72.4	86.8
Linear probing	73.5	80.8	33.5	58.2	0.05
MLP-4 [6]	70.6	80.7	31.2	57.7	4.04
Partial [6]	73.1	81.7	35.0	58.9	12.65
Bias [42]	74.2	80.1	42.4	62.1	0.25
VPT-Shallow [6]	79.9	82.5	37.8	62.9	0.05
VPT-Deep [6]	76.8	84.5	53.4	67.7	0.22
ARC	79.0	86.6	59.9	72.6	0.27
but also scalable to larger models, making it a promising solution for efficient pre-trained model adaptation across a wide range of backbone sizes.
Experiments on hierarchical Vision Transformers. We extended the ARC method to Swin Transformer [11], a hierarchical Transformer architecture. To accommodate the varying feature dimensionalities in Swin Transformer, we introduced a stage-sharing strategy, enabling parameter sharing within each stage. The results on the VTAB-1k benchmark (Table 4) demonstrate the generalizability of ARC. It achieves competitive transfer learning accuracy and maintains favorable parameter scale, except for the Natural group where ARC performs relatively weaker. These findings highlight ARC's versatility and effectiveness in adapting different transformer architectures, showcasing its potential for practical applications in visual adaptation.
In order to gain deeper insights into the ARC method, we performed ablation studies to explore its additional properties. The experiments were conducted on the VTAB-1k benchmark using the pre-trained ViT-B/16 model. The results of these ablation studies are presented in Table 5.
Bottleneck dimensionality. In our ARC method, we adopt the low-rank design for the adapters but with the added feature of sharing the down-/up-projection matrices across layers and learning low-dimensional re-scaling coefficients to re-compose adaptation matrices. In this section, we investigate the impact of the bottleneck dimensionality on adaptation performance. The results are presented in Table 5a. We find that a dimensionality of 50 achieves the best balance between transfer learning performance and parameter efficiency. Further reduction to a 10-dimensional space leads to fewer parameters but at the cost of performance degradation. Conversely, higher-dimensional hidden spaces result in inferior performance. These findings validate the effectiveness of our low-rank design, with 50 linear projections providing sufficient flexibility for composing layer-adaptive adapters.
Adapter positioning. By default, our adapters are positioned before the MHA and FFN modules, allowing the adaptation operations to be seamlessly integrated into the pre-trained network during inference without additional inference cost. In this section, we investigate the impact of adapter
Table 5: Ablation experiments on VTAB-1k benchmark using ViT-B/16 backbone. The table shows average accuracy ("Acc.") and parameter count ("Params.") for all downstream datasets.
Bott. dim	Acc.	Params.
10	72.4	0.07
50	73.4	0.13
100	73.1	0.21
200	71.4	0.36
(a) Bottleneck dimension.
(b) ARC location.
(c) Para. sharing strategy.
Location	Acc.	Params.
Before MHA	72.2	0.08
After MHA	69.1	0.08
Before FFN	71.1	0.08
After FFN	69.0	0.08
Before MHA & FFN	73.4	0.13
After MHA & FFN	71.4	0.13
Strategy	Acc.	Params.
non-intra + non-inter	73.4	0.98
intra + inter*	72.9	0.10
intra + inter	73.4	0.13
non-intra + inter	73.4	0.21
(d) Insert selection.
Layer ind.	Form	Acc.	Params.
1 ~ 6	sequential	71.5	0.126
7 ~ 12	sequential	67.9	0.126
1 ~ 12	sequential	73.4	0.133
1 ~ 12	parallel	70.4	0.133
position on pre-trained model adaptation performance using different positioning strategies. The results are presented in Table 5b. Interestingly, placing the adapters after the MHA and/or FFN modules leads to performance degradation, despite this strategy being commonly adopted in previous works such as[7; 29]. Moreover, using only one type of adapter for either MHA or FFN results in inferior performance compared to using both types of adapters. This suggests that employing both types of adapters allows for more comprehensive adaptation of the pre-trained model to the target task without significantly increasing the parameter count.
Sharing vs. non-sharing adaptation parameters. In ARC, we adopt a parameter-sharing strategy to effectively reduce the number of adaptation parameters. This strategy encompasses two aspects: intra-layer sharing and inter-layer sharing. Through symmetric down-projection and up-projection matrices, we achieve intra-layer sharing, while inter-layer sharing involves sharing projection matrices across different layers. In this section, we investigate the impact of adaptation parameter sharing by conducting experiments with non-sharing or partial sharing designs. "intra + inter*" denotes sharing the bottleneck structure between MHA and FFN. The results presented in Table 5c demonstrate that using non-symmetric projection matrices or layer-independent adaptation parameters does not result in performance gains but leads to a noticeable increase in parameters. This validates the effectiveness of our parameter-sharing design.
Adapter insertion. We examine the performance of inserting the proposed adapters in a subset of layers using sequential or parallel insertion strategies. The results in Table 5d show that the performance of ARC improves as more layers are inserted. Furthermore, we observe that inserting adapters in the first six layers yields better results compared to inserting them in the last six layers. Additionally, we explore a parallel insertion setting inspired by [24], but the impact is not significantly pronounced. Another notable aspect is that our parameter sharing scheme in the ARC method prevents a linear increase in parameter size with the number of layers, ensuring better scalability, particularly for larger-scale models.
The adaptation parameter sharing scheme in the ARC method is built on the assumption that layers have the same dimensionality. This assumption is crucial as it enables the sharing of down-/up-projection matrices involved in the bottleneck operation across layers, leading to parameter efficiency. However, it is worth exploring strategies to extend this scheme and accommodate dimensionality variation. This research direction holds promise for addressing scenarios where dimensionality varies and further enhancing the flexibility and applicability of the ARC method.
Our paper introduced the Adapter Re-Composing (ARC) method, which leverages the reusability of adaptation parameters to efficiently adapt pre-trained models. By sharing down-/up-projections in low-rank adapters across layers and learning layer-specific re-scaling coefficients to re-composing layer-adaptive adapters, ARC balances transfer learning performance and adaptation overheads. Extensive experiments demonstrate the compelling performance of our approach with a reduced parameter size. ARC offers a promising solution for efficient pre-trained model adaptation, showcasing the potential of reusing adaptation parameters for competitive results.
[35] T. Gebru, J. Krause, Y. Wang, D. Chen, J. Deng, and L. Fei-Fei, “Fine-grained car detection for visual census estimation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1, 2017.
[36] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy et al., “A large-scale study of representation learning with the visual task adaptation benchmark,” arXiv preprint arXiv:1910.04867, 2019.
[37] E. B. Zaken, Y. Goldberg, and S. Ravfogel, "Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models," in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2022, pp. 1-9.
[38] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, high-performance deep learning library," Advances in Neural Information Processing Systems (NeurIPS), vol. 32, pp. 8026-8037, 2019.
[39] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, "Cutmix: Regularization strategy to train strong classifiers with localizable features," in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 6023-6032.
[40] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," arXiv preprint arXiv:1710.09412, 2017.
[41] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, "Rethinking the inception architecture for computer vision," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016, pp. 2818-2826.
[42] H. Cai, C. Gan, L. Zhu, and S. Han, "Tinytl: Reduce memory, not parameters for efficient on-device learning," Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 11-285-11-297, 2020.
In the supplementary materials involving our work, we demonstrate detailed dataset settings, supplemental insights and analysis, extra experimental details, supplemental experiments, and broader impacts, including:
- A Detailed descriptions for datasets and implementation
B Insights of architecture design
C Parameter size analysis
D Experimental details on larger-scale and hierarchical ViT backbones
- E Experimental details on ablation studies
F Expanded experiments with self-supervised pre-training
G Broader impacts
Due to the limitation that the file "Supplementary Materials.zip" larger than 100MB cannot be uploaded on OpenReview, the supplementary materials only upload the code for the project. Please refer to the anonymous link https://drive.google.com/file/d/1ZblHbYF1Jr0uOGeTLI4uII6GHt3CV3I2/view to obtain the complete code, datasets, and models.
A Detailed descriptions for datasets and implementation
We describe the details of visual adaptation classification tasks in Table 6 (FGVC) and 7 (VTAB-1k), including the class number and the train/val/test sets.
Table 6: Dataset statistics for FGVC. “*” denotes the train/val split of datasets following the dataset setting of VPT models [6].
Dataset	Description	Classes	Train size	Val size	Test size
CUB-200-2011 [31]	Fine-grained bird species recognition	200	5,394*	600*	5,794
NABirds [32]	Fine-grained bird species recognition	555	21,536*	2,393*	24,633
Oxford Flowers [33]	Fine-grained flower species recognition	102	1,020	1,020	6,149
Stanford Dogs [34]	Fine-grained dog species recognition	120	10,800*	1,200*	8,580
Stanford Cars [35]	Fine-grained car classificatio	196	7,329*	815*	8,041
Table 7: Dataset statistics for VTAB-1k [36].
Dataset	Description	Classes	Train size	Val size	Test size
CIFAR-100	100	10,000
Caltech101	102	6,084
DTD	47	1,880
Flowers102	Natural	102	800/1,000	200	6,149
Pets	37	3,669
SVHN	10	26,032
Sun397	397	21,750
Patch Camelyon	2	32,768
EuroSAT	10	800/1,000	200	5,400
Resisc45	Specialized	45	6,300
Retinopathy	5	42,670
Clevr/count	8	15,000
Clevr/distance	6	15,000
DMLab	6	22,735
KITTI/distance	4	800/1,000	200	711
dSprites/location	Structured	16	73,728
dSprites/orientation	16	73,728
SmallNORB/azimuth	18	12,150
SmallNORB/elevation	9	12,150
Table 8 summarizes the detailed configurations we used for experiments. As mentioned in Section 4.1, we utilize grid search to select hyper-parameters such as learning rate, weight decay, batch size, and adapter dropout, using the validation set of each task. Note that we also apply dropout to the middle features produced by our ARC method, which we term as "adapter dropout". Specifically, during the ARC process, we randomly drop partial features before up-projection.
B Insights of architecture design
Similar to Fig. 3, we present more visualization results of singular value distribution of adaptation matrices  $\mathbf{W}_{\mathrm{full}} \in \mathbb{R}^{D \times D}$  learned without the bottleneck operation. As shown in Fig. 4, the singular value distribution of adaptation matrices learned on  $DTD$  downstream task exhibits a power-law
Table 8: The implementation details of configurations such as optimizer and hyper-parameters. We select the best hyper-parameters for each download task via using grid search.
Optimizer	AdamW
Learning Rate	{0.2, 0.1, 0.05, 0.01, 0.005, 0.001, 0.0001}
Weight Decay	{0.05, 0.01, 0.005, 0.001, 0}
Batch Size	{256, 128, 32}
Adapter Dropout	{0.8, 0.5, 0.1, 0}
Learning Rate Schedule	Cosine Decay
Training Epochs	100
Warmup Epochs	10
distribution across various layers in the downstream tasks. This finding provides further support for our research motivation.
Figure 4: Singular value distribution of adaptation matrices without the bottleneck structure. Two adaptation matrices of both MHA and FFN blocks are fine-tuned on the DTD downstream task. The X-axis represents the singular values, while the Y-axis represents the count of singular values within specific ranges.
C Parameter size analysis
To showcase the parameter-efficiency of our ARC method, we compare its parameter size with other popular lightweight adaptation methods (Table 9), including Adapter [7], VPT [6], LoRA [24], and SSF [9]. Adapter [7] adds two linear projections to each encoder layer during fine-tuning, resulting in the introduction of  $2 \cdot D \cdot D' \cdot L$  learnable parameters, where  $D'$  denotes the hidden dimensionality of the linear projections. Furthermore, due to the presence of non-linear activations in Adapter, the additional parameters contribute to supernumerary overhead during the inference phase. VPT [6] incorporates  $m$  prompts into input space, leading to an increase of  $m \cdot D$  parameters for VPT-Shallow and  $m \cdot D \cdot L$  parameters for VPT-Deep. In contrast to Adapter, both LoRA [24] and SSF [9] employ linear adaptation methods without incorporating non-linear functions. This design choice allows them to leverage re-parameterization benefits, thereby mitigating additional computations during inference. Specifically, the adaptation matrix of LoRA, which consists of a down-projection and an up-projection, introduces  $2 \cdot w \cdot D \cdot D' \cdot L$  learnable parameters, where  $w$  denotes the number of attention matrices undergoing adaptation. SSF inserts linear scaling and shifting coefficients after
$o$  operations, resulting in an addition of  $2 \cdot o \cdot D \cdot L$  extra parameters. The proposed ARC method offers additional parameter compression by sharing symmetric projection matrices across different layers. This approach introduces only  $D \cdot D'$  parameters. Additionally, we learn low-dimensional re-scaling coefficients and bias terms for each layer, resulting in a total of  $(D' + D) \cdot L$  additional parameters. Overall, the number of parameters in our default ARC is  $2 \cdot ((D \cdot D') + (D' + D) \cdot L)$ .