SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qiong  $\mathbf{W}\mathbf{u}^{12}$  Wei  $\mathbf{Y}\mathbf{u}^{12}$  Yiji Zhou $^{12}$ , Shubin Huang $^{1}$ , Xiaoshuai Sun $^{12}$ , Rongrong Ji $^{12*}$
<sup>1</sup> Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.
$^{2}$  Institute of Artificial Intelligence, Xiamen University, 361005, P.R. China. {qiong, weiyu}@stu.xmu.edu.cn, zhouyiyi@xmu.edu.cn, shubinhuang@stu.xmu.edu.cn, {xssun, rrji}@xmu.edu.cn
With ever increasing parameters and computation, vision-language pre-trained (VLP) models exhibit prohibitive expenditure in downstream task adaption. Recent endeavors mainly focus on parameter efficient transfer learning (PETL) for VLP models by only updating a small number of parameters. However, excessive computational overhead still plagues the application of VLPs. In this paper, we aim at parameter and computation efficient transfer learning (PCETL) for VLP models. In particular, PCETL not only needs to limit the number of trainable parameters in VLP models, but also to reduce the computational redundancy during inference, thus enabling a more efficient transfer. To approach this target, we propose a novel dynamic architecture skipping (DAS) approach towards PCETL. Instead of directly optimizing the intrinsic architectures of VLP models, DAS first observes the significances of their modules to downstream tasks via a reinforcement learning (RL) based process, and then skips the redundant ones with lightweight networks, i.e., adapters, according to the obtained rewards. In this case, the VLP model can well maintain the scale of trainable parameters while speeding up its inference on downstream tasks. To validate DAS, we apply it to a bunch of representative VLP models, and conduct extensive experiments on a set of VL tasks. The experimental results not only show the great advantages of DAS in reducing computational complexity, e.g.  $-11.97\%$  FLOPs of METER on VQA2.0, but also confirm its competitiveness against existing PETL methods in terms of parameter scale and performance. Our source code is given in https://github.com/DoubtedSteam/DAS.
Inspired by the great success in natural language processing (NLP) [8, 19, 32, 36], large-scale pre-training on massive image-text pairs also becomes the de-facto standard in vision-language research [4, 24, 35, 65]. To accommodate the large-scale pre-training corpora, vision-language pre-trained (VLP) models [4, 70, 10, 24, 28, 35, 56, 74] often adopt Transformer-based networks with sheer sizes of parameters and computations. In this case, directly transferring these VLP models to downstream tasks is excessively expensive in terms of memory footprint and computation overhead.
To reduce the costs of pre-training models, recent advances resort to parameter efficient transfer learning (PETL) for affordable downstream task adaptations [16, 13, 26, 37, 59, 72, 74, 75]. In particular, the PETL methods aim to save the memory usage for downstream tasks by only updating
(a) Layer-Performance Curve
(b) Flops-Accuracy Scatter
Figure 1: (a) The performance of METER [10] is barely affected when skipping a certain number of its Transformer layers. (b) The comparison on VQA2.0 between the conventional PETL methods [17, 21, 26, 36, 59] and the proposed Dynamic Architecture Skipping (DAS) for METER. The circle size represents the memory footprint. DAS is the only method faster than the original VLP model.
or inserting a small number of trainable parameters rather than fully tuning the entire model. For instance, prompt-tuning methods [1, 5, 26, 36, 37, 50, 52, 74, 75] expand the input sequence with hand-craft or learnable tokens to bridge the gap between pre-training and downstream tasks. Practitioners also insert lightweight neural networks called Adapter [13, 20, 46, 47, 59, 72, 41] into the pretrained models, thereby projecting the hidden features onto the semantic space of downstream tasks. More recently, these PETL methods have been successfully introduced to VLP models [13, 33] for either prompt-based image classification [26, 52, 74, 75] or conventional VL tasks like visual question answering [58, 59, 43]. Despite the great successes, PETL methods still cannot reduce the computation complexity of VLP models, which is of more significance for applications.
In this paper, we study a novel problem called parameter and computation efficient transfer learning (PCETL). To achieve more efficient downstream task adoptions, PCETL is not only expected to maintain the scale of trainable parameters similar to PETL, but more importantly, also needs to reduce the computation complexity of pre-training models, thereby speeding up their inference on downstream tasks. In existing works, the efficiency of the network itself is largely attributed to its manually [6, 22, 44, 54] or automatically structural designs [62, 66, 78, 45]. Although the computation complexity can be further reduced by the compression methods, such as pruning [67, 49, 3, 31, 7, 12, 30, 9, 64, 69, 55], quantification [11, 29, 73] or distillation [2, 25], these approaches usually require retraining after optimizing the network architecture, which is not applicable to the VLP models that are well pre-trained on massive data. On one hand, the large-scale pre-training data still needs a certain model capacity to learn these prior knowledge, thus it is hard to obtain a good trade-off between performance and computation overhead for the pre-training objectives. On the other hand, devising a small and effective model for each downstream task is still laborious and expensive, which also contradicts the target of PETL [20, 36], since fully fine-tune is often required.
In this case, we argue that the key to PCETL is to explore the parameter and computation redundancy in existing VLP models. It is generally assumed that the model scale is proportional to the complexity of the task [77, 1, 18, 37, 76]. To robustly serve a variety of downstream tasks, VLP models are pretrained by multiple pre-training objectives based on tens of millions of image-text pairs [14, 51, 52, 57]. In this case, the excessive parameters are suitable for pre-training, but prone to redundant for a downstream task. As shown in Fig. 1-(a), the performance of METER [10] on VQA is barely affected when skipping a certain number of its Transformer layers. This empirical result also suggests that exploring a short-cut pathway in existing VLP models is a feasible way for PCETL.
To this end, we propose a novel Dynamic Architecture Skipping (DAS) approach towards efficient transfer learning for VLP models. By observing the module redundancy of VLP models, DAS can realize the optimal subnetwork routing of VLP models for a downstream task, thereby reducing the computation during inference. In practice, DAS regards this process as a  $k$ -armed bandit problem, and evaluates the importance of each VL layer/block to the downstream task via numerous subnetwork samplings and quick validations. Thus, the obtained rewards can be used to reflect the redundancy of VL modules and determine which layers to be skipped. Meanwhile, to achieve parameter efficiency,
we also adopt lightweight networks, i.e. Adapter [20, 59], to serve the hidden feature adoptions and the short-cut connections of DAS for VLP models.
To validate DAS, we apply it to a set of VLP models, namely including [10], ViLT [28] and LaVIN [42]  $^{2}$ , on three VL benchmarks, namely VQA2.0 [14], NLVR $^{2}$  [57] and Flickr30K [51]. The experimental results not only show the competitive performance of DAS against the fully finetune and PETL methods [17, 21, 26, 59], but also witness its great advantage in reducing the computation complexity of VLP models. For instance, DAS can help METER achieve  $96.60\%$  performance of full tuning on the VQA2.0 benchmark with only  $1.65\%$  trainable parameters, while decreasing  $11.97\%$  FLOPs. For the practical deployment of a specific VL task, DAS can reduce up to  $93.75\%$  parameters of the VLP models  $^{3}$ . These results well confirm our assumption about the redundancy of VLP models on downstream tasks, and also validated the design of the proposed DAS.
Overall, our contributions can be summarized as three-fold:
- We raise a new problem called parameter and computation efficient transfer learning (PCETL) for VLP models, which not only requires to keep the scale of training parameters but also needs to reduce the computation complexity of VLP models on downstream tasks.
- We propose a novel Dynamic Architecture Skipping (DAS) for PCETL, which can explore the optimal short-cut pathway in VLP models with the combination of parallel adapters.
- On two VLP models and three benchmark datasets, the proposed DAS not only reduces the computation overhead by a large extent, e.g.,  $-11.97\%$  FLOPs of METER on VQA2.0, but also is on par with existing PETL methods in terms of parameter and performance.
2.1 Vision-Language Pre-training
In recent years, the advancement in natural language processing (NLP) [32, 36] also sparks the prevalence of large-scale pre-training in vision-language (VL) research [4, 10, 24, 28, 35, 56, 74]. In particular, VL pre-training also accomplishes self-supervised learning on massive image-text pairs based on the generative prediction tasks, e.g. masked language modeling (MLM) and masked image modeling (MIM). Furthermore, Image Text Matching (ITM) [10, 28] is also applied to align two modalities. In terms of network architecture, most VLP models are equipped with two modality encoders to extract the visual and text features, e.g. BERT [8] and Faster-RCNN [53], respectively, based on which a stack of Transformer-based layers are deployed for cross-modal fusions [4, 10, 23, 24, 28, 34, 38, 39, 56, 61, 63]. For instance, ViL-BERT [39] and LXMERT [61] contain two independent Transformer-based branches [63] for region and text feature extractions, and another Transformer block is used for cross-modal interaction. To simplify the framework, Visual-BERT [34], VL-BERT [56] and UNITER [4] abandon additional branches and directly feed features into a single Transformer network. Then Pixel-BERT [24], CLIP-ViL [38], and METER [10] break the limitation of object detection backbones by directly applying grid features for multi-modal pre-training. To further simplify the model complexity, ViLT [28] directly feeds the word embeddings and image patches into the Transformer blocks. Additionally, CLIP [52] applies cross-modal contrastive learning for vision-language alignment with a shallow fusion layer for prediction. Overall, these VLP models often require more network branches due to the increase of modalities, resulting in more parameter and computation overheads. In this paper, we present the first attempt to evaluate their network redundancy on downstream tasks.
2.2 Parameter Efficient Transfer Learning
Parameter Efficient Transfer Learning (PETL) [13, 15, 17, 20, 21, 46-48, 58-60, 71, 72] aims to approach the fully-tuned performance on downstream by updating a small number of parameters. One of the main methodology in PETL is prompt-tuning [1, 5, 26, 36, 37, 50, 52, 74, 75], which is originally designed for large pre-trained language models such as GPT-3 [1]. Concretely, the hand-craft prompts [50, 52] expand the original input sequence with natural language and regard all
Due to the page limit, the results of LaVIN are given in our Github project.
3When the model is only deployed for a task, the skipped layers can be also removed during deployment.
Figure 2: Illustration of Dynamic Architecture Skipping (DAS). DAS regards the network skipping as a  $k$ -armed bandit problem, and evaluates the redundancy of each VL layer/block via numerous subnetwork samplings. The accumulated rewards are used to determine which layers can be skipped, and adapters are also used for feature adoptions and short-cut connections.
problems as a generation task. To better fit downstream tasks, soft prompt tuning methods [26, 36, 75] replace the handcraft prompts with a sequence of trainable embeddings. In addition to prompt-tuning, adapter-based [13, 20, 46, 47, 59, 72] methods insert lightweight feed-forward networks into VLP models, and these methods transfer VLP models by projecting hidden features onto the downstream distributions [20, 59]. Furthermore, LoRA [21] is proposed to transfer VLP models without additional calculation overhead in the inference stage by updating the low-rank parts of the original parameters. Besides, Side-tuning [71] runs in parallel with the pre-trained models to adapt downstream tasks while overcoming the constraint from the concrete structure. In addition, LST [58] stacks the outputs of the pre-trained modules in a parallel path. Without feedback to the VLP model, LST alleviates the memory requirement in the transfer while increasing the computation overhead. Compared to fine-tuning the entire model, PETL methods significantly improve the efficiency in transferring VLP models to downstream tasks. However, all of the above methods take the original VLP model as the upper bound of inference efficiency. In this paper, the proposed DAS method is the first to reduce the computation of VLP models while maintaining competitive performance. In terms of computation efficiency, network compression methods can also reduce the computation overhead, but they often require to fully tune the model on the downstream tasks, such as LayerDrop [12], EfficientVLM [64] and J2C [9]. This setting makes them against the target of PCETL about parameter efficiency.
We first revisit the principle of PETL methods for VLP models. Given a vision-language pre-trained (VLP) model, denoted as  $G(\cdot)$ , the target of PETL is to achieve the parameter-efficient adaption on the downstream task, which can be summarized as
$$
\underset {\sigma} {\operatorname {a r g m i n}} \mathcal {L} \left(G (I, T | [ \boldsymbol {\theta}, \boldsymbol {\sigma} ])\right), \tag {1}
$$
where  $\theta = \{\theta_{1},\theta_{2},\dots,\theta_{n}\}$  represent the parameters of  $n$  layers in the VLP model, and  $\theta$  is usually frozen in PETL.  $(I,T)$  denotes the image-text pair, and  $\sigma$  is a small number of updated parameters. Since all VLP layers are reserved on downstream tasks, PETL methods can only reduce the parameter expenditure but not the computation of VLP models. Moreover, most PETL methods often incur non-negligible latency during inference [21, 40].
According to the observation in Fig 1-(a), there exists obvious redundancy in the VLP models. To this end, the objective of the proposed task of parameter and computation efficient transfer learning (PCETL) can be defined by
$$
\underset {\boldsymbol {\sigma}, \mathbf {K}} {\operatorname {a r g m i n}} \mathcal {L} \left(G (I, T | [ \boldsymbol {\theta} _ {\mathbf {K}}, \boldsymbol {\sigma} ])\right), \tag {2}
$$
Algorithm 1 Dynamic Architecture Skipping
Require: The training and validation sets,  $D_{t}$ ,  $D_{v}$ . VLP modules  $\{\theta_i\}$ . Adapters  $\{\sigma_i\}$ .
Ensure: The optimal network structure  $G_{s}$ .
Initialize the degree of redundancy of all modules,  $\mathbf{r}^{(0)} = [\mathbf{r}_1^{(0)},\mathbf{r}_2^{(0)},\dots,\mathbf{r}_n^{(0)}]$
for  $t$  in  $T$  do
Calculate the score according to Eq. 3 and sample a subnetwork.
Update the parameter of adapters for adaptation and substitute  $\sigma$  by  $\mathcal{L}_{train}$ .
if  $t$  mod interval == 0 then
Obtain a validation batch  $d_v \gets D_v$ .
Evaluate the performance of  $c$  subnetworks and get their reward  $r_i = e^{-\mathcal{L}_{val}}$
Update the degree of redundancy  $\mathbf{r}^{(t)}$  according to Eq. 5.
Obtain the optimal structure  $\theta_{\mathbf{K}}$  based on  $\mathbf{r}$
where  $\pmb{\theta}_{\mathbf{K}} = \{\theta_{k_1},\theta_{k_2},\dots,\theta_{k_m}\} \in \pmb{\theta}$  is the parameters of VLP modules except the skipped ones. Via skipping the redundant layers in VLP models and the combination of PETL methods, VLP models can accelerate the inference speed and maintain the scale of updated parameters.
4 Dynamic Architecture Skipping
4.1 Redundancy Estimation
In this paper, we propose a novel transfer learning approach called Dynamic Architecture Skipping (DAS) towards the parameter and computation efficient adaption of VLP models.
DAS first observes the model redundancy to downstream tasks before skipping the layers of VLP models. In practice, we regard this process as a  $k$ -armed bandit problem, as illustrated in Fig. 2. Firstly, we define the degree of redundancy as  $\mathbf{r} \in \mathbb{R}^n$ , where  $n$  is the number of VLP modules. To correctly estimate the redundancy, we equally initialize  $\mathbf{r}_i = 0$  and update it momentarily.
In each training step, we skip  $m$  modules according to uniform distribution based on  $\mathbf{r}$ , and train the sampled architectures on the downstream data. For  $t$ -th step, the action policy  $\pi_i^{(t)}$  for the  $i$ -th VLP module follows the distribution:
$$
\pi_ {i} ^ {(t)} \sim U (0, \rho (\mathbf {r} _ {i} ^ {(t)})), \tag {3}
$$
where  $U(a, b)$  is the uniform distribution between  $a$  and  $b$ . And  $\rho$  represents the Sigmoid function. We randomly pick a probability from the  $\pi_i^{(t)}$  of each module as the score  $s_i^{(t)}$ . According to the score  $s_i^{(t)}$ , the sampled subnetwork can be defined by
$$
G _ {s} = g _ {0} \circ g _ {1} \circ \dots \circ g _ {n},
$$
$$
w h e r e \quad g _ {i} = \left\{ \begin{array}{l} \theta_ {i}, i \in \left\{j \mid s _ {j} ^ {(t)} <   s _ {m} ^ {(t)} \right\}, \\ \sigma_ {i}, i \in \left\{j \mid s _ {j} ^ {(t)} \geq s _ {m} ^ {(t)} \right\}. \end{array} \right. \tag {4}
$$
Here,  $g_{i} \circ g_{i+1}$  represents the compositional function  $g_{i}(g_{i+1}(\cdot))$ .  $\theta_{i}$  denotes the original VL module, and  $\sigma_{i}$  is the lightweight module like adapter for short-cut connection. And  $s_{m}^{(t)}$  are the  $m$  largest values in the picked scores. Here, the module with a larger  $\mathbf{r}_{i}^{(t)}$  is more likely to be skipped during training. Meanwhile, Eq. 4 also help  $\sigma_{i}$  learn pre-trained knowledge from  $\theta_{i}$  in a distillation way [68].
Then, DAS observes the redundancy of VLP modules in a reinforcement learning manner, as shown in 2. DAS samples  $c$  candidate network structures and calculates their rewards according to their loss values during validation, i.e. reward  $v = e^{-loss}$ . Based on the rewards, the degree of redundancy  $r$  can be updated by
$$
\mathbf {r} _ {i} ^ {(t + 1)} = \mathbf {r} _ {i} ^ {(t)} + \left(v _ {h} - \frac {1}{c} \sum_ {j = 1} ^ {c} v _ {j}\right). \tag {5}
$$
Here,  $v_{h}$  denotes the reward of the sampled subnetwork, where the  $i$ -th module is skipped. When its validation loss is larger than the mean value, it suggests that this skipped module is more redundant. Eq. 5 is conducted at short training intervals to make sure that most subnetworks can be sufficiently validated via numerous samplings, and the theorem of large numbers can guarantee the optimality of search results. The detailed search procedure of DAS is illustrated in Algorithm 1. Finally, according to the degree of redundancy  $\mathbf{r}$ , we can select top-  $m$  layers to be skipped, thereby reducing the computation complexity of VLP models.
4.2 Model Adapting
To reduce the updated parameter scale during adaptation, DAS also introduces lightweight adapters [20, 59] to serve the hidden feature transformations as well as the short-cut connections in VLP models. Typically, an adapter is constructed by two linear projection layers and an activation function in between:
$$
a d a p t e r (\mathbf {x}) = R e L U \left(\mathbf {x} \mathbf {W} _ {i n}\right) \mathbf {W} _ {o u t}. \tag {6}
$$
Here,  $\mathbf{W}_{in} \in \mathbb{R}^{d \times h}$  and  $\mathbf{W}_{out} \in \mathbb{R}^{h \times d}$  are two trainable matrices, where  $h \ll d$ . For the  $i$ -th VLP module, the adaptation can be defined by
$$
\mathbf {x} _ {i} = \mathbf {x} _ {i - 1} + V L P \left(\mathbf {x} _ {i - 1}\right) + a d a p t e r \left(\mathbf {x} _ {i - 1}\right), \tag {7}
$$
where  $\mathbf{x}_i$  is the output of the  $i$ -th component. In this manner, DAS can freeze most parameters in the VLP models during adaption, similar to existing PETL methods [59].
Notably, directly removing the redundant modules will make the subsequent layers to receive the hidden features with drastic changes. Meanwhile, we do not expect the fully tuning of the whole model. In this case, we apply the adapter to serve the short-cut connection of the skipped layers:
$$
\mathbf {x} _ {i} = \mathbf {x} _ {i - 1} + a d a p t e r _ {r} \left(\mathbf {x} _ {i - 1}\right). \tag {8}
$$
In this way, DAS can not only bridge the gap between feature transformations, but also retain parameter efficiency. Based on the estimated redundancy, DAS skips the redundant modules and finds out the optimal pathway for the downstream task with the helps of adapters, as shown in Fig. 2.
5.1 Datasets and Experimental Setup
Visual Question Answering. We conduct experiments on VQA2.0 [14]. Instead of answering the question in open-ended natural language, it is converted into a classification task with 3, 129 classes. Following the previous setting [10, 28], the PETL methods and DAS are trained on the train and validation sets of VQA2.0, and we report the test-dev results from the online evaluation<sup>4</sup>.
Natural Language for Visual Reasoning. The NLVR $^2$  [57] is a dataset for classifying triplets of two images and a question into two classes. Because its form is different from the setup of VLP models, which has two images in one VL example, we feed these triplet examples to the model following the default setting of ViLT [28] and METER [10]. Under this setting, the paired images and the question are input to the network, respectively. And the classifier predicts the results according to the concatenation of two representations.
Retrieval Task. For cross-modal retrieval, we measure the performance on Flickr30K [51] re-splited by Karpathy et al. [27]. We initialize the predictor for similarity measurement from the pre-trained ITM head. During the training, we randomly sample 15 instances as negative samples.
We validate DAS on two deep-fusion based VLP models, which are ViLT [28] and METER [10]. In terms of ViLT, we update the parameters of the additional components, classifier, class token and modal-type embeddings, while the rest are frozen. Following the most conventional setting [17, 59], the width of hidden states in adapters is set to 96. And the hidden dimension of the adapter used for
4https://eval.ai/web/challenges/challenge-page/830/overview
Table 1: Comparison between DAS and the PETL methods for METER and ViLT on VQA, NLVR² and Flickr30K. The best performance is bold and the second best is underlined.
METER
Method	Updated Parameter	VQA	NLVR2	Flickr30K	Avg.
test-dev	Additional FLOPs	test-P	Additional FLOPs	IR/TR R@1	Additional FLOPs	Per.	Additional FLOPs
Full Tuning	323.31M	77.43	0.00	83.05	0.00	82.22/94.30	0.00	84.25	0.00
Classifier Only	-	69.93	0.00	73.23	0.00	78.80/89.00	0.00	77.74	0.00
Shallow Prompt [36]	0.30M	68.51	+28.71G	65.69	26.84G	74.20/88.60	+28.71G	74.25	+28.71G
Deep Prompt [26]	1.84M	70.78	+6.53G	72.64	+5.59G	78.84/89.40	+6.53G	77.92	+6.53G
LoRA [21]	0.29M	74.00	0.00	78.82	0.00	79.86/92.60	0.00	81.32	0.00
Adapter [59]	5.34M	74.70	+1.64G	79.93	+1.38G	80.38/91.90	+1.64G	81.73	+1.64G
Scaled PA [17]	3.59M	75.11	+1.12G	80.38	+0.66G	80.40/93.20	+1.12G	82.27	+1.12G
DAS4-Fusion	5.34M	74.80	-11.16G	80.11	-5.13G	80.12/91.80	-11.16G	81.71	-9.15G
DAS4-Global	6.23M	75.09	-4.51G	80.69	-3.67G	80.42/91.40	-6.06G	81.90	-4.74G