SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wei Dong<sup>1,4</sup>
Dawei Yan
Zhijun Lin
Peng Wang $^{3}$  *
<sup>1</sup>College of Information and Control Engineering,
Xi'an University of Architecture and Technology.
$^{2}$ School of Computer Science, Northwestern Polytechnical University.
$^{3}$ School of Computer Science and Engineering,
University of Electronic Science and Technology of China.
$^{4}$  Xi'an Hypersonic Measurement Technology Co., Ltd.
The advent of high-capacity pre-trained models has revolutionized problem-solving in computer vision, shifting the focus from training task-specific models to adapting pre-trained models. Consequently, effectively adapting large pre-trained models to downstream tasks in an efficient manner has become a prominent research area. Existing solutions primarily concentrate on designing lightweight adapters and their interaction with pre-trained models, with the goal of minimizing the number of parameters requiring updates. In this study, we propose a novel Adapter ReComposing (ARC) strategy that addresses efficient pre-trained model adaptation from a fresh perspective. Our approach considers the reusability of adaptation parameters and introduces a parameter-sharing scheme. Specifically, we leverage symmetric down-/up-projections to construct bottleneck operations, which are shared across layers. By learning low-dimensional re-scaling coefficients, we can effectively re-compose layer-adaptive adapters. This parameter-sharing strategy in adapter design allows us to further reduce the number of new parameters while maintaining satisfactory performance, thereby offering a promising approach to compress the adaptation cost. We conduct experiments on 24 downstream image classification tasks using various Vision Transformer variants to evaluate our method. The results demonstrate that our approach achieves compelling transfer learning performance with a reduced parameter count. Our code is available at https://github.com/DavidYanAnDe/ARC.
The utilization of large-scale pre-trained models for various downstream tasks has garnered significant interest in the computer vision community [1; 2; 3; 4]. These models continually push the boundaries of downstream task performance while eliminating the need for task-specific model design and training. In early attempts, a commonly adopted transfer learning strategy involved directly fine-tuning a pre-trained model on downstream tasks. However, the full fine-tuning strategy suffers from two major drawbacks: (1) Updating large-scale parameters is prohibitively expensive and typically requires a substantial amount of training data to prevent overfitting. (2) As the sizes of state-of-the-art pre-trained models continue to increase, it becomes impractical and unsustainable to store a distinct set of model weights for each downstream task.
*Corresponding author. Email address: p.wang6@hotmail.com
In contrast to fine-tuning the entire pre-trained network, recent research has focused on parameter-efficient model adaptation. The core idea behind this line of work is to keep the majority of pre-trained parameters frozen and only update or introduce a small fraction of task-specific parameters. Several methods fall under this umbrella, including prompt tuning [5; 6], visual adapter [7; 8], and linear feature modulation [9]. These methods have demonstrated competitive or even superior performance compared to full fine-tuning while significantly reducing the adaptation cost. They differ in the design of lightweight adapters and how these adapters interact with the pre-trained parameters. Prompt tuning methods [5; 6] adapt the features of pre-trained Vision Transformers by introducing trainable task-specific tokens into one or multiple attention layers. Visual adapter [7; 8] injects a non-linear lightweight adapter with a bottleneck architecture between layers of the pre-trained model to adjust the feature distribution. Such non-linear adapters are simplified using linear transformations such as shifting and scaling [9] to directly modulate the pre-trained features.
In this work, we not only focus on designing lightweight adapters but also emphasize the importance of adaptation parameter reusability in further compressing the adaptation cost. We adopt a low-rank design for the adapter using a bottleneck operation but propose a novel approach. Unlike other methods that place the adapter in different layers and directly learn different parameters for each adapter to cater to layer-wise variation, we propose sharing the down/up projections in the low-rank adapter across different layers and simply learning low-dimensional re-scaling coefficients to recompose the linear projections into layer-adaptive adapters. The idea of Adapter Re-Composing (ARC) is motivated by the observation that naturally derived adaptation matrices can exhibit extremely low-rank characteristics, even when not explicitly designed as such. This implies the possibility of using a shared "basis" to re-compose the adapters. Furthermore, we design the low-rank adapters using symmetric down-projection and up-projection matrices, which further reduces the parameter size. Due to their linear nature and careful positioning design, our adapters can be seamlessly integrated into the pre-trained network, as in [7; 9; 10], without adding extra computation during the inference phase.
We evaluate our method on various large Vision Transformer models, including ViT-B [1] and its variants such as ViT-L [1], ViT-H [1], and Swin-B [11], using 24 downstream image classification benchmark datasets. The experimental results demonstrate that our method achieves compelling transfer learning performance while maintaining a smaller parameter size.
The key contributions of this paper are summarized as follows:
- We approach efficient pre-trained model adaptation from a novel perspective by exploring the reusability of adaptation parameters, which goes beyond existing works that primarily focus on the lightweight design of adapter structures.
- We introduce the Adapter Re-Composing (ARC) strategy, which shares bottleneck operation's down-/up-projections across layers and utilizes lower-dimensional re-composing coefficients to create layer-adaptive adapters. This approach enables fewer parameters than prior works.
- Our parameter sharing scheme in the ARC method prevents a linear increase in parameter size with the number of layers, ensuring better scalability, particularly for larger-scale models.
- Through extensive experiments on various Vision Transformer variations and numerous downstream tasks, we show that our method achieves highly competitive transfer learning performance while maintaining a relatively low level of additional parameter.
In this section, we present a concise review of the existing literature, focusing on two key areas: (1) Pre-training and fine-tuning, and (2) Parameter-efficient transfer learning.
Pre-training and fine-tuning. Pre-training and fine-tuning, also known as transfer learning [12; 13; 14], is a popular approach that utilizes large-scale datasets [15; 16; 17; 18; 19] to train models for adaptation to different downstream tasks. By extracting knowledge from these datasets and encoding it into parameters, models can be fine-tuned for specific tasks, resulting in improved performance compared to models without pre-training. The availability of large-scale datasets [17] has led to
(a) Prompt Learning [6].
(c) LoRA [24].
(d) SSF [9].
Figure 1: Visual summary of typical parameter-efficient pre-trained model adaptation methods.
significant advancements in performance and convergence speed for downstream tasks. Scaling up models [20; 21] to handle the growing data volume enhances data utilization and improves the efficiency and robustness of pre-trained models across various data distributions and noise levels. Large models [1; 11; 22] also benefit from self-supervised pre-training [2; 23] using unlabeled data, reducing the cost, duration, and quality issues associated with human annotation. By leveraging this approach, models can effectively extract knowledge from readily available unlabeled data, further enhancing their generalization performance in downstream tasks.
Parameter-efficient transfer learning. The exponential increase in model parameters presents a computational challenge when fine-tuning the entire network on downstream tasks. In the field of NLP, researchers have explored parameter-efficient transfer learning approaches [7; 24; 25; 26; 27; 28] that train a subset of the model or add new modules with fewer parameters while achieving comparable or even superior performance. Inspired by the success of NLP, several notable works [8; 29; 30] have emerged in the computer vision domain. One approach, known as Prompt Tuning [5; 6], addresses the distribution mismatch between pre-training and downstream tasks by learning task-specific tokens. Adapter-like methods [8; 7; 29] insert trainable modules, such as MLPs with activation functions and residual structures, into the network to facilitate transfer learning. LoRA [24] exploits the low-rank update to a large-scale frozen model and introduces a bypass to the original parameter matrix to mimic the fine-tuning of the entire model parameters. SSF [9] introduces lightweight scaling and shift operations to modulate the pre-trained representations. The core concepts of these aforementioned works are visually summarized in Fig. 1.
As an advancement in visual adaptation, ARC addresses the limitations of Prompt Tuning, which requires different prompt designs for different downstream tasks. Moreover, ARC introduces the innovative concept of bottleneck matrix reuse, achieving state-of-the-art performance with minimal adaptation cost comparing to other rank-decomposition strategies. Additionally, ARC employs a linear design for the adapters and inherits the benefits of re-parameterization [9; 10; 24], ensuring that inference does not introduce any additional computational complexity.
In this section, we start by providing an introduction to the notations, symbols, and background related to Vision Transformers, which is followed by the presentation of our proposed Adapter Re-Composing (ARC) method. ARC focuses on efficient transfer learning for Vision Transformers by reusing rank-decomposition projections to adaptively compose layer-wise adapters, thereby reducing the size of learnable parameters. Additionally, we discuss the insights gained from our architecture design.
A plain Vision Transformer model contains a patch embedding layer and multiple encoder layers. Given an input image  $\mathbf{X} \in \mathbb{R}^{H \times W \times C}$ , the patch embedding layer first splits the image into a sequence of flattened patches  $\mathbf{X}_{\mathrm{patches}} \in \mathbb{R}^{N \times (P^2 \cdot C)}$ , where  $(H, W)$  is the resolution of the input image,  $(P, P)$  is the resolution of each patch,  $C$  denotes the number of input channels, and  $N = H \cdot W / P^2$  is the number of tokens. Subsequently, the image patches are mapped to a  $D$ -dimensional embedding space through a linear projection  $\mathbf{W} \in \mathbb{R}^{(P^2 \cdot C) \times D}$ . A learnable [class] token vector  $\vec{x}_{\mathrm{cls}} \in \mathbb{R}^D$  is then prepended to the sequence of  $\mathbf{X}_{\mathrm{patches}}$ , and the position embeddings  $\mathbf{X}_{\mathrm{pos}} \in \mathbb{R}^{(N + 1) \times D}$  are added to the sequence. The output of the patch embedding layer can be expressed as follows:
$$
\mathbf {X} _ {\mathrm {e m b}} = \left[ \vec {\boldsymbol {x}} _ {\mathrm {c l s}} ^ {\mathrm {T}}; \mathbf {X} _ {\mathrm {p a t c h e s}} \mathbf {W} \right] + \mathbf {X} _ {\mathrm {p o s}}, \tag {1}
$$
where  $[\cdot ;\cdot ]$  denotes concatenation operation. This output is then fed into several consecutive encoder layers, each consisting of a Multi-Head Attention (MHA) block and a Feed-Forward Network (FFN) block. LayerNorm (LN) is applied before each block, and residual connections are applied thereafter. The process of  $l$  -th encoder layer is defined as:
$$
\mathbf {X} ^ {(l) \prime} = \operatorname {M H A} (\operatorname {L N} \left(\mathbf {X} ^ {(l - 1)}\right)) + \mathbf {X} ^ {(l - 1)}, \tag {2}
$$
$$
\mathbf {X} ^ {(l)} = \operatorname {F F N} (\operatorname {L N} (\mathbf {X} ^ {(l) \prime})) + \mathbf {X} ^ {(l) \prime},
$$
where  $\mathbf{X}^{(l-1)}$  denotes input tokens in  $l$ -th layer,  $\mathbf{X}^{(l)'}$  indicates intermediate representations produced by MHA, and the output of  $l$ -th layer is  $\mathbf{X}^{(l)}$ .
In MHA block, each Attention Head (AH) module utilizes the weight matrices  $\mathbf{W}_q^{(l)}\in \mathbb{R}^{D^{(l - 1)}\times D_h^{(l)}}$ $\mathbf{W}_k^{(l)}\in \mathbb{R}^{D^{(l - 1)}\times D_h^{(l)}}$  , and  $\mathbf{W}_v^{(l)}\in \mathbb{R}^{D^{(l - 1)}\times D_h^{(l)}}$  for the query, key, and value operations, respectively. These operations enable an exclusive attention mechanism on the normalized feature representations  $\mathbf{X}_{\mathrm{norm}}^{(l - 1)} = \mathrm{LN}(\mathbf{X}^{(l - 1)})$  ..
$$
\mathbf {X} _ {h} ^ {(l) \prime} = \mathrm {A H} _ {h} \left(\mathbf {X} _ {\text {n o r m}} ^ {(l - 1)}\right) = \operatorname {s o f t m a x} \left(\frac {\left(\mathbf {X} _ {\text {n o r m}} ^ {(l - 1)} \mathbf {W} _ {q} ^ {(l)}\right) \left(\mathbf {X} _ {\text {n o r m}} ^ {(l - 1)} \mathbf {W} _ {k} ^ {(l)}\right) ^ {\mathrm {T}}}{\sqrt {D _ {h} ^ {(l)}}}\right) \mathbf {X} _ {\text {n o r m}} ^ {(l - 1)} \mathbf {W} _ {v} ^ {(l)}, \tag {3}
$$
where  $D_h^{(l)} = \frac{D^{(l)}}{M}$  is the feature dimensionality of the output representations  $\mathbf{X}_h^{(l)'}$  for each attention head and  $M$  represents the number of attention heads. The MHA block concatenates multiple  $\{\mathbf{X}_h^{(l)'}\}$  in sequence and generates the outputs through a linear projection  $\mathbf{W}_o^{(l)} \in \mathbb{R}^{(M \cdot D_h^{(l)}) \times D^{(l)}}$ :
$$
\mathbf {X} ^ {(l) \prime} = \mathrm {M H A} (\mathbf {X} _ {\mathrm {n o r m}} ^ {(l - 1)}) = [ \mathrm {A H} _ {1} (\mathbf {X} _ {\mathrm {n o r m}} ^ {(l - 1)}), \dots , \mathrm {A H} _ {M} (\mathbf {X} _ {\mathrm {n o r m}} ^ {(l - 1)}) ] \mathbf {W} _ {o} ^ {(l)}. \qquad (4)
$$
The FFN block consists of two linear projections with the GELU activation function in between:
$$
\mathbf {X} ^ {(l)} = \operatorname {F F N} \left(\mathbf {X} _ {\text {n o r m}} ^ {(l) \prime}\right) = \operatorname {G E L U} \left(\mathbf {X} _ {\text {n o r m}} ^ {(l) \prime} \mathbf {W} _ {1} ^ {(l)}\right) \mathbf {W} _ {2} ^ {(l)}, \tag {5}
$$
where  $\mathbf{W}_1^{(l)}\in \mathbb{R}^{D^{(l)}\times 4\cdot D^{(l)}}$  and  $\mathbf{W}_2^{(l)}\in \mathbb{R}^{4\cdot D^{(l)}\times D^{(l)}}$  denote two linear projection matrices, and  $\mathbf{X}_{\mathrm{norm}}^{(l)'} = \mathrm{LN}(\mathbf{X}^{(l)'}).$
3.2 Adapter Re-Composing method
We observe that the majority of existing adaptation methods introduce adapters into various layers and learn separate parameters for adapting the pre-trained model to specific downstream tasks. Previous studies [8; 24] have shown the effectiveness of leveraging the low-rank properties of adapters to fine-tune frozen pre-trained models. Inspired by these findings, we propose a novel approach to create a unified linear space across different adapters to enhance parameter efficiency and adaptation performance.
Architecture. The ARC architecture incorporates a bottleneck operation for adapters, which consists of three key components: a linear down-projection, layer-specific re-scaling coefficients, and a linear up-projection. This architecture is illustrated in Fig. 2. To facilitate the reusability of adaptation matrices, we have developed a sharing and re-composing scheme for the ARC method.
Figure 2: Illustration of the proposed Adapter Re-Composing Method.
This scheme involves sharing linear projections across layers and learning low-dimensional re-scaling coefficients to re-compose the layer-adaptive adapters. In addition, to enhance parameter compression, we leverage symmetric down-projection and up-projection in a single bottleneck operation:
$$
\mathbf {W} _ {\mathrm {u p}} = \left(\mathbf {W} _ {\mathrm {d o w n}}\right) ^ {\mathrm {T}}, \tag {6}
$$
where  $\mathbf{W}_{\mathrm{down}} \in \mathbb{R}^{D \times D'}$  and  $\mathbf{W}_{\mathrm{up}} \in \mathbb{R}^{D' \times D}$  with  $D' \ll D$  denote shared down-projection and up-projection matrices across different layers;  $D'$  represents the hidden dimensionality of the projections. To accommodate the variations across different layers, we learn re-scaling coefficients to re-compose the layer-adaptive adaptation matrices. These coefficients are then diagonalized into a diagonal matrix  $\mathbf{C}^{(l)} \in \mathbb{R}^{D' \times D'}$  specific to each layer  $l$ . This diagonal matrix allows for efficient and effective adjustment of the adaptation parameters at each layer. Formally, given an input  $\mathbf{X}_{\mathrm{in}} \in \mathbb{R}^{(N + 1) \times D}$ , the output of our ARC module is:
$$
\mathbf {X} _ {\text {o u t}} = \operatorname {A R C} \left(\mathbf {X} _ {\text {i n}}\right) = \mathbf {X} _ {\text {i n}} \mathbf {W} _ {\text {d o w n}} \mathbf {C} ^ {(l)} \mathbf {W} _ {\text {u p}} + \mathbf {X} _ {\text {i n}}. \tag {7}
$$
Unless otherwise specified, we adhere to the default configuration of inserting our ARC modules sequentially before both the MHA and FFN blocks in the Vision Transformer. The influence of adapter positions will be discussed in Section 4.3. Therefore, the Vision Transformer incorporating our ARC modules can be formulated as follows:
$$
\mathbf {X} ^ {(l) \prime} = \operatorname {M H A} \left(\operatorname {A R C} _ {\text {M H A}} \left(\operatorname {L N} \left(\mathbf {X} ^ {(l - 1)}\right)\right)\right) + \mathbf {X} ^ {(l - 1)},
$$
$$
\mathbf {X} ^ {(l)} = \operatorname {F F N} \left(\operatorname {A R C} _ {\text {F F N}} \left(\operatorname {L N} \left(\mathbf {X} ^ {(l) ^ {\prime}}\right)\right)\right) + \mathbf {X} ^ {(l) ^ {\prime}}. \tag {8}
$$
Note that  $\mathrm{ARCMHA}$  and  $\mathrm{ARCFFN}$  are two independent ARC modules, meaning that the projection matrices of the two modules are not shared between them. During the fine-tuning phase, we exclusively update the learnable parameters of our newly added ARC modules. This involves freezing all original parameters of the pre-trained model and solely focusing on updating the parameters of our ARC.
Inference. Our ARC employs a completely linear transformation so that we can re-parameterize it by fusing the module to the original framework of the pre-trained model. Take the ARC module of FNN as an example, the process can be defined by:
$$
\mathbf {X} ^ {(l)} = \operatorname {G E L U} \left(\operatorname {A R C} _ {\mathrm {F F N}} \left(\mathbf {X} ^ {(l) \prime}\right) \mathbf {W} _ {1} ^ {(l)}\right) \mathbf {W} _ {2} ^ {(l)}, \tag {9}
$$
where  $\mathrm{ARC}_{\mathrm{FFN}}(\mathbf{X}^{(l)'})$  can be represented by  $\mathbf{X}^{(l)'}(\mathbf{W}_{\mathrm{down}}\mathbf{C}^{(l)}\mathbf{W}_{\mathrm{up}} + \mathbf{I})$  according to Eq. (7) with  $\mathbf{I}\in \mathbb{R}^{D\times D}$  being an identity matrix. Furthermore, we can construct  $\mathbf{W}_1^{(l)'}$  by:
$$
\mathbf {W} _ {1} ^ {(l) \prime} = \left(\mathbf {W} _ {\text {d o w n}} \mathbf {C} ^ {(l)} \mathbf {W} _ {\text {u p}} + \mathbf {I}\right) \mathbf {W} _ {1} ^ {(l)}. \tag {10}
$$
(a) The singular value distribution of MHA adapter.
(b) The singular value distribution of FFN adapter.
Figure 3: Singular value distribution of adaptation matrices without the bottleneck structure. Two adaptation matrices of both MHA and FFN blocks are fine-tuned on the DTD downstream task. The X-axis represents the singular values, while the Y-axis represents the count of singular values within specific ranges. Complete visualization is available in the appendix.
Therefore, we can replace the matrix  $\mathbf{W}_1^{(l)}$  by  $\mathbf{W}_1^{(l)'}$  in the inference stage, thereby avoiding extra computational overheads.
3.3 Insights of architecture design
In this work, we propose an approach that involves adopting a low-rank design for the adapters and sharing the bottleneck projections across layers. Our motivation for adopting this approach stems from the assumption that the layer-wise adapters can be effectively re-composed by re-scaling a small number of base linear projections. To validate this assumption, we conduct an analysis of the singular value distribution of adaptation matrices  $\mathbf{W}_{\mathrm{full}} \in \mathbb{R}^{D \times D}$  learned without the bottleneck operation, which are anticipated to be full-rank. In Fig. 3, we observe that the singular values exhibit extreme sparsity and follow a power-law distribution, with the majority of singular values being close to zero. This indicates that the learned adaptation matrices naturally exhibit low-rank characteristics. Furthermore, the pronounced sparsity of the singular values suggests that the adaptation matrices can be effectively reconstructed using a limited number of basis vectors. These findings provide strong evidence supporting the rationale behind our adaptation parameter-sharing strategy.
In this section, we present the experimental settings, comparison to existing solutions, and ablation studies to unveil the key properties of the proposed method.
Datasets. We evaluate the effectiveness of our ARC approach on two sets of visual task adaptation benchmarks, comprising a total of 24 datasets. The list of datasets used for evaluation is provided below:
FGVC. We conduct experiments with the default settings in the VPT [6] on a collection of five Fine-Grained Visual Classification (FGVC) datasets, known as FGVC. The FGVC dataset collection includes CUB-200-2011 [31], NABirds [32], Oxford Flowers [33], Stanford Dogs [34], and Stanford Cars [35].
VTAB-1k. We also evaluate our ARC method on the VTAB-1k benchmark [36], which consists of 19 diverse visual classification tasks. These tasks are divided into three groups: the Natural group, which contains images captured through standard cameras; the Specialized group, which includes images captured by specialist equipment such as remote sensing and medical imaging; and the Structured group, which comprises synthesized images from simulated environments, such as object counting and 3D depth prediction. Each downstream task in the VTAB-1k benchmark consists of 1000 training examples. Following VPT [6], we set aside 200 samples from the training set as the validation set to
Table 1: Comparison of ARC with baselines and state-of-the-art efficient adaptation methods on five FGVC datasets. All methods utilize ViT-B/16 pre-trained on ImageNet-21k as the backbone. "SSF*" denotes the performance reported in the original SSF paper [9], which incorporates advanced data augmentations like cutmix [39], mixup [40], and regularization techniques such as label smoothing [41]. To ensure a fair comparison, we reproduced the SSF method using the code provided by [9], while employing the same basic data augmentations as our approach, and we denote SSF's reported performance as "SSF*" and ARC's performance augmented with SSF's data augmentation as "ARC*". The bold font shows the best accuracy of all methods and the underline font shows the second best accuracy.
Dataset Method	CUB-200-2011	NABirds	Oxford Flowers	Stanford Dogs	Stanford Cars	Mean	Params.(M)
Full fine-tuning	87.3	82.7	98.8	89.4	84.5	88.5	85.98
Linear probing	85.3	75.9	97.9	86.2	51.3	79.3	0.18
Adapter [7]	87.1	84.3	98.5	89.8	68.6	85.7	0.41
Bias [42]	88.4	84.2	98.8	91.2	79.4	88.4	0.28
VPT-Shallow [6]	86.7	78.8	98.4	90.7	68.7	84.6	0.25
VPT-Deep [6]	88.5	84.2	99.0	90.2	83.6	89.1	0.85
LoRA [24]	88.3	85.6	99.2	91.0	83.2	89.5	0.44
SSF [9]	82.7	85.9	98.5	87.7	82.6	87.5	0.39
SSF* [9]	89.5	85.7	99.6	89.6	89.2	90.7	0.39
\( \text{ARC}_{\text{att}} \)	88.4	85.0	99.4	90.1	82.7	89.1	0.22
ARC	88.5	85.3	99.3	91.9	85.7	90.1	0.25
ARC*	89.3	85.7	99.7	89.1	89.5	90.7	0.25
select hyperparameters. Subsequently, we train the model on the full training data using the selected hyperparameters.