Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models
Authors: Qiong Wu
Wei Yu
Yiyi Zhou
Shubin Huang
Xiaoshuai Sun
Rongrong Ji
Venue: NeurIPS 2023 poster (2023)
Keywords: vision and language
parameter and computation efficient transfer learning

Abstract: With ever increasing parameters and computation, vision-language pre-trained (VLP) models exhibit prohibitive expenditure in downstream task adaption. Recent endeavors mainly focus on parameter efficient transfer learning (PETL) for VLP models by only updating a small number of parameters. However, excessive computational overhead still plagues the application of VLPs. In this paper, we aim at parameter and computation efficient transfer learning (PCETL) for VLP models. In particular, PCETL not only needs to limit the number of trainable parameters in VLP models, but also to reduce the computational redundancy during inference, thus enabling a more efficient transfer. To approach this target, we propose a novel dynamic architecture skipping (DAS) approach towards effective PCETL. Instead of directly optimizing the intrinsic architectures of VLP models, DAS first observes the significances of their modules to downstream tasks via a reinforcement learning (RL) based process, and then skips the redundant ones with lightweight networks, i.e. adapters, according to the obtained rewards. In this case, the VLP model can well maintain the scale of trainable parameters while speeding up its inference on downstream tasks. To validate DAS, we apply it to two representative VLP models, namely ViLT and METER, and conduct extensive experiments on a bunch of VL tasks. The experimental results not only show the great advantages of DAS in reducing computational complexity, e.g. -11.97%  FLOPs of METER on VQA2.0, but also confirm its competitiveness against existing PETL methods in terms of parameter scale and performance. Our source code is given in our appendix.

Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models
Qiong Wu Wei Yu Yiyi Zhou 12 Shubin Huang', Xiaoshuai Sun 12 Rongrong J Key Laboratory of Multimedia Trusted Perception and Efficient Computing.
Ministry of Education of China, Xiamen University, 361005, P.R. China. Institute of Artificial Intelligence, Xiamen University, 361005, P.R. China.
{qiong, weiyu)@stu. xmu, edu. en, zhouyiyi@xmu. edu. en, shubinhuang@stu. xmu, edu.cn, (xssun, rrji)exmu. edu.cc.eddcc
Abstract
With ever increasing parameters and computation, lision-language pre-trained (VLP) models exhibit prohibitive expenditure in downstream task adaption. Recent endeavors mainly focus on parameter efficient transfer learning (PETL) for VL.P models by only updating small number of parameters. However, excessive computational overhead still plagues the application of VLPs. In this paper, we aim at parameter and computation efficient transfer learning (PCETL) for VLP models. In particular, PCETL not only needs to limit the number of trainable parameters in VLP models, but also to reduce the computational redundancy during inference, thus enabling a more efficient transfer. To approach this target, we propose novel dynamic architecture skipping (DAS) approach towards PCETL. Instead of directly optimizing the intrinsic architectures of VLP models, DAS first observes the significances of their modules downstream tasks via reinforcement learning (RI.) based process, and then skips the redundant ones with lightweight networks, i.e., adapters, according the obtained rewards. In this case, the VLP model can well maintain the scale of trainable parameters while speeding up its inference on downstream tasks. To validate DAS, we apply representative VLP models, and conduct extensive experiments on set of VL tasks. The experimental results not only show the great advantages of DAS in reducing computational complexity, e.g. -11.97% FLOPs of METER on VQA2.0, but also confirm its competitiveness against existing PETL methods in terms parameter scale and performance. Our source code is given in https::///// :/pgit///itt thub com/DoubtedSteam/DDS
bunch of
Introduction
Inspired by the great success in natural language processing (NLP) 19 32 36 large-scale pre-training massive image-text pairs also becomes the de-facto standard in vision-language research 4 24 35 65
To accommodate the large-scale pre-training corpora, vision-language
[0 24 28, 35| 56 74] often adopt Transformer-based networks with sheer sizes of parameters and computations. In this case, directly transferring these VLP models to downstream tasks is excessively expensive in terms of memory footprint and computation overhead.
pre-trained (VLP) models 70
To reduce the costs of pre-training models, recent advances resort to parameter efficient transfer learning (PETL) for affordable downstream task adaptions [16 13 26 37
59
72
74 75). In
partieular, the PETL methods aim to save the memory usage for downstream tasks by only updating
"Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

80
77
76,
12
76,10
74,25
71,28
78-
Fine-tuning
70
67.07
Accuracy
60
Accuracy%%
76
DAS
Scaled PA
74
Adapter
50
72
Deep Prompt
70
29
Shallow Prompt
68
20
0
2
3
5
85
90
95
100
105 Flops(G)
110
115 120 125
130
Skipped Fusion Layers
(a) Layer-Performane Curve
(b) Flogs-Aceuracy Scatter
Figure 1: (a) The performance METER 10 is barely affected when skipping certain number of its Transformer layers. (b) The comparison on VQA2.0 between the conventional PETL methods [17]
21 26 36 [9] and the proposed Dynamic Architecture Skipping (DAS) for METER. The circle size
represents the memory footprint. DAS is the only method faster than the original VLP model.
or inserting a small number of trainable parameters rather than fully tuning the entire model. For instance. prompt-tuning methods [| 36 37 37 50 52 (5) expand the input sequence with hand- craft learnable tokens bridge the gap between pre- pre-training and downstream tasks. Practitioners
also insert lightweight neural networks called Adapter 13 20
46
47
59
72
41 into the pre-
trained models, thereby projecting the hidden features onto the semantic space of downstream tasks. More recently. these PETL methods have been successfully introduced to VLP models [I3 33
for either prompt-based image classification 226 52 74 75
or conventional VL tasks like visual 59 43 Despite the great successes, PETL methods still cannot reduce the
question answering (58
computation complexity of VLP models, which of more significance for applications.
In this paper, we study novel problem called parameter and computation efficient transfer learning (PCETL). To achieve more efficient downstream task adaptions. PCETL is not only expected to maintain the scale of trainable parameters similar to PETL, but more importantly, also needs to reduce the computation complexity of pre-training models, thereby speeding up their inference on downstream tasks. In existing works, the efficiency of the network itself is largely attributed
its manually 6 22 54 or automatically structural designs [62 66
78 45). Although the
computation complexity can be further reduced by the compression methods, such as pruming [67] 44
31 12 30,
69 55), quantification [T 73 or distiiliation 22 25), these approaches usually require retraining after optimizing the network architecture, which is not applicable to the VLP models that are well pre-trained on massive data. On one hand, the large-scale pre-training data still needs certain model capacity to learn these prior knowledge. thus it is hard to obtain good trade-off between performance and computation overhead for the pre-training objectives. On the other hand, devising a small and effective model for each downstream task is still laborious and expensive, which also contradicts the target of PETL 20 36), since fully finettune is often required.
In this case, we argue that the key to PCETL is to explore the parameter and computation redundancy in existing VLP models. It generally assumed that the model scale proportional to the complexity of the task [77 [ 37 766) To robustly serve variety of downstream tasks, VLP models are pre- trained by multiple pre-training objectives based on tens of millions of image-text pairs [4 52 (7).
In this case, the excessive parameters are suitable for pre-training, but prone to redundant for downstream task. As shown in Fig. (a), the performance of METER (IO) on VQA is barely affected when skipping certain number of its Transformer layers. This empirical result also suggests that exploring a short-cut pathway in existing VLP models is a feasible way for PCETL.
To this end, we propose a novel Dynamic Architecture Skipping (DAS) approach towards efficient transfer learning for VLP models. By observing the module redundancy of VLP models, DAS can realize the optimal subnetwork routing of VLP models for a downstream task, thereby reducing the computation during inference. practice, DAS regards this process as k-armed bandit problem, and evaluates the importance of each VL layer/block to the downstream task via numerous subnetwork samplings and quick validations. Thus, the obtained rewards can used to reflect the redundancy of VL modules and determine which layers be skipped. Mcanwhile, to achieve parameter efficiency,

---

we also adopt lightweight networks, i.e. Adapter [2, to serve the hidden feature adaptions and the short-cut connections of DAS for VLP models.
To validate DAS, we apply it to a set of VLP models, namely including 10 ViLT 28 and
LaVIN 42
three VI. benchmarks, namely VQA2.0 [14, NLVR (57) and Flickr30K [51] The
experimental results not only show the competitive performance of DAS against the fully finetune and PETL methods [17 21 26 59), but also witness its great advantage in reducing the computation complexity of VLP models. For instance, DAS can help METER achieve 96.60% performance of full tuning on the VQA2.0 benchmark with only 1.65% trainable parameters, while decreasing 11.97% FLOPs. For the practical deployment of a specific VL task. DAS can reduce up to 93. 75% parameters of the VLP models These results well confirm our assumption about the redundancy of VLP models on downstream tasks, and also validated the design of the proposed DAS.
Overall, our contributions can be summarized three-fold:
We raise new problem called parameter and computation efficient transfer learning (PCETL) for VLP models, which not only requires to keep the scale of training parameters but also needs to reduce the computation complexity of VLP models on downstream tasks.
We propose a novel Dynamic Architecture Skipping (DAS) for PCETL, which can explore the optimal short-cut pathway in VLP models with the combination of parallel adapters.
On two VLP models and three benchmark datasets, the proposed DAS not only reduces the computation overhead by large extent, e.g., 11.997% FLOPs of METER VQA2.0, but also is on par with existing PETL methods in terms of parameter and performance.
2 Related Work
2.1 Vision-Language Pre-training
In recent years, the advancement in natural language processing (NLP) 32
36
also sparks the
74). In particular, VI pre-training also accomplishes self-supervised learning on massive image-text pairs based on the generative prediction tasks, e.g. masked language modeling (MI.) and masked image modeling (MIM). Furthermore, Image Text Matching (ITM) [10] 28) is also applied to align two modalities. In terms of network architecture, most VLP models are equipped with two modality encoders to extract the visual and text features, e.g. BERT K and FaNter-RCNN (53) respectively, based on which a stack of Transformer-
prevalence of large-scale pre-training in vision-language (VL.) research 4
10
24
28
35
56
layers are deployed for modal fusions 44 10
61 63 For instance, ViL-BERT 39 and LXMERT 61 contain two independent Transformer- based branches [63 for region and text feature extractions, and another Transformer block used for cross- modal interaction. To simplify the framework, Visual-BERT (4), VL--BERT' (56) and UNITER [4] abandon additional branches and directly feed features into single Transformer network. Then Pixel-BERT (24, CLIP-ViL (38, and METER [10 break the limitation of object detection backbones by directly applying grid features for multi-modal pre-training. To further simplify the model complexity, ViLT (28) directly feeds the word embeddings and image patches into the Transformer blocks. Additionally, CLIP (52) applies cross-modal contrastive learning for lisionelanguage alignment with a shallow fusion layer for prediction. Overall, these VLP models often require more network branches due to the increase of modalities, resulting in more parameter and computation overheads. In this paper, we present the first attempt to evaluate their network redundancy on downstream tasks.
23
24
28
34
38
39
56
2.2 Parameter Efficient Transfer Learning
Parameter Efficient Transfer Learning (PETL) B3
15
17
20
21
48
46 48
58
60
71 72
aims to
approach the fully-tuned performance on downstream by updating small number parameters.
One of the main methodology in PETL is prompt-tuning [ 5 20 Bol 37 50
52
74 75), which
originally designed for large pre-trained language models such as GPT-3 1 Concretely, the
hand-craft prompts [50] 52 expand the original input sequence with natural language and regard all
2 Due to the page limit, the results LaVIN are given our Github project.
When the model is only deployed for a task, the skipped layers can be also removed during deployment.
3

---

Prediction
Prediction
Comparison and Obtain Rewards
VLP
Update R Redundancy
VL.P
VLP
Adapter
Belg
VLP
Subnetworks
U Jpdate
VLP
t
20
VLP
Validation
Sample
[++1]
VLP
Adapter
t Inputs
Inputs
Estimated Redundancy
Inputs
Original VLP model
Dynamic Architecture Skipping
Accelerated VLP model
Figure 2: Illustration of Dynamic Architecture Skipping (DAS). DAS regards the network skipping
as a k-armed bandit problem, and evaluates the redundancy of each VL layer/block via numerous subnetwork samplings. The accumulated rewards used determine which layers can be skipped, and adapters are also used for feature adaptions and short-cut connections.
problems as a generation task. To better fit downstream tasks, soft prompt tuning methods 26 36 75
replace the handcraft prompts with sequence of trainable embeddings. In addition prompt-tuning.
59 (22 methods insert lightweight feed-forward networks into VI.P the downstream distributions [20] 55]] Furthermore, LoRA [2I] is proposed to transfer VLP models without additional calculation overhead in the inference stage by updating the low-rank parts of the original parameters. Besides, Side-tune Side-tuning [7]] runs in parallel with the pre-trained models to adapt downstream tasks while overcoming the constraint from the concrete structure. In addition, LST [88 stacks the outputs of the pre-trained modules in parallel path. Without feedback to the VLP model, LST alleviates the memory requirement in the transfer while increasing the computation overhead. Compared to fine-tuning the entire model, PETL methods significantly improve the efficiency in transferring VL. models to downstream tasks. However, all of the above methods take the original VLP model as the upper bound of inference efficiency. In this paper, the proposed DAS method is the first to reduce the computation of VLP models while maintaining competitive performance. efficiency, network compression methods can also reduce the computation overhead ,but they often require fully tune the model on the downstream tasks, such as LayerDrop [2], EfficientVLMM 64 and J2C [9] This setting make them against the target of PCETL about parameter efficiency.
adapter-based 13 20 46 47
models, and these methods transfer VLP models by projecting hidden features
terms of computation
Preliminary
We first revisit the principle of PETL methods for VLP models. Given vision-language pre-trained (VLP) model, denoted as G'(-,, the target of PETI. is to achieve the parameter-efficien adaption on the downstream task, which can be summarized as
argmin L(G
(1)
where 0 (0,, 02, 0,) represent the parameters of n layers in the VLP model, and is usually frozen in PETL. (I, denotes the image-text pair, and or is small number of updated parameters. Since all VLP layers are reserved on downstream tasks, PETL methods can only reduce the parameter expenditure but not the computation of VLP models. Moreover, most PETL methods often incur non-negligible latency during inference 21
According to the observation in Fig [ there exists obvious redundancy in the VLP models. To this end, the objective of the proposed task of parameter and computation efficient transfer learning (PCETL) can be defined by
argmin [[[[,,](OO.... a))), o.K
(2)

---

Algorithm Dynamic Architecture Skipping
Require: The training and validation sets, Dr, D,.. VLP modules {0:). Adapters {oi). The optimal network structure Gs-
Ensure:
Initialize the degree of redundancy of all modules, r())) for t T do
[r]
I' (0)
Calculate the score according to Eq. and sample subnetwork. Update the parameter of adapters for adaptation and substitute by Ltrain- if L mod interval == 0 then Obtain validation batch d, <- D..
Evaluate the performance of subnetworks and get their reward r; Cual Update the degree of redundancy r(t) according to Eq.
end if
end for
Obtain the optimal structure 0x based on r.
where 0K (0k; , Okg..... O...) is the parameters of VI.P modules except the skipped ones. Via skipping the redundant layers in VLP models and the combination of PETL methods, VL. models can accelerate the inference speed and maintain the scale of updated parameters.
Dynamic Architecture Skipping
4.1 Redundancy Estimation
In this paper, we propose novel transfer learning approach called Dynamic Architecture Skipping (DAS) towards the parameter and computation efficient adaption of VLP models.
DAS first observes the model redundancy to downstream tasks before skipping the layers of VLP models. In practice, Firstly, we define the degree of redundancy as E R", where is the number of VLP modules. To correctly estimate the redundancy, we equally initialize r; 0 and update it momentously.
regard this process k-armed bandit problem, illustrated in Fig. 2
In each training step, we skip m modules according to uniform distribution based on r, and train the sampled architectures on the downstream data. For t-th step, the action policy n. (t) for the i-th VLP
module follows the distribution:
p(r,((r)))
where U (a, b) is the uniform distribution between and b. And p represent the Sigmoid function. We randomly pick a probability from the n of each module as the score sf According to the score the sampled subnetwork can be defined by
G = 90 o 91 o o In.
0,, i (ils)) (2)
< 8m }.
where gi
o, E (ils))
2 sm }.
Here, Ii 9++1 represents the compositional function gi (9)+1(())) Og denotes the original VL module, and a; is the lightweight module like adapter for short-cut connection. And 8m are the m largest values in the picked scores. Here, the module with larger r: is more likely to be skipped during training. Meanwhile, Eq. also help d, learn pre- trainer knowledge from 0, a distillation way (68).
Then, DAS observes the redundancy of VLP modules in a reinforcement learning manner, as shown in DAS samples e candidate network structures and calculates their rewards according to their loss values during validation, i.e. reward e eloss, Based on the rewards, the degree of redundancy can be updated by
(t+1) r. + (un
(5)
5

---

Here, 2/ denotes the reward of the sampled subnetwork, where the i-th module is skipped. When its validation loss is larger than the mean value, it suggests that this skipped module is more redundant. Eq. is is conducted at short training intervals to makes sure that most subnetworks can be sufficiently validated via numerous samplings, and the theorem of large numbers can guarantee the optimality of search results. The detailed search procedure of DAS is illustrated in Algorithm. [ Finally, according to the degree of redundancy r, we can select top-m layers to be skipped, thereby reducing the computation complexity of VLP models.
4.2 Model Adapting
To reduce the updated parameter scale during adaptation, DAS also introduces lightweight adapters [2 59] to serve the hidden feature transformations well the short-cut connections in VLP models. Typically, an adapter is constructed by two linear projection layers and an activation function in between:
adapter(x) = Re W in) ) Wout.
(6)
Here, Win E Rd and W out E R are two trainable matrices, where h < d. For the i-th VLP module, the adaptation can be defined by
x = x;- + VLP(x;-1) + adapter(x:-1).
(7)
where x; is the output of the i-th component. In this manner, DAS can freeze most parameters in the VLP models during adaption, similar to existing PETL methods [59].
Notably, directly removing the redundant modules will make the subsequent layers to receive the hidden features with drastic changes. Meanwhile, we do not expect the fully tuning of the whole model. this case, we apply the adapter to serve the short-cut connection of the skipped layers:
x x + adapter, (x;-1):
In this way, DAS can not only bridge the gap between feature transformations, but also retain parameter efficiency. Based on the estimated redundancy, DAS skips the redundant modules and finds out the optimal pathway for the downstream task with the helps of adapters, as shown Fig.
Experiment
5.1
Datasets and Experimental Setup
Visual Question Answering. We conduct experiments on VQA2.0 14, Instead of answering the
question in open-ended natural language, it is converted into classification task with 3. 129 classes. Following the previous setting [10 [28], the PETL methods and DAS are trained on the train and validation sets of VQA2.0, and we report the test-dev results from the online evaluation [
Natural Language for Visual Reasoning. The NLVR2 57 is a dataset for classifying triplets of two images and question into two classes. Because its form is different from the setup of VLP models, which has two images in one VL example, we feed these triplet examples to the model following the default setting of ViLT (28) and METER [10]. Under this setting, the paired images and the question are input to the network, respectively. And the classifier predicts the results according to the concatenation of two representations.
Retrieval Task. For cross-modal retrieval, we measure the performance on Flickr30K [51 re-splited by Karpathy al. (27). We initialize the predictor for similarity measurement from the pre-trained ITM head. During the training, we randomly sample 15 instances as negative samples.
5.2 Implementation details
We validate DAS on two deep-fusion based VLP models, which are ViLT (28 and METER [10] In terms of ViLT, we update the parameters of the additional components, classifier, class token and modal-type embeddings, while the rest are frozen. Following the most conventional setting [17 59) the width of hidden states in adapters is set to 96. And the hidden dimension of the adapter used for

---

Table 1: Comparison between DAS and the PETL methods for METER and ViLT on VQA, NLVR and Flickr30K. The best performance is bold and the second best is underlined.
METER
VQA
NLVR
Flickr30K
Avg.
Updated Parameter
Method
Additional FLOPs
Additional FLOPs
Additional FLOPs
Additional
test-dev
test-P
IR/TR R@!
Per.
FLOPs
Full Tuning
323.31M
77.43
0.00
83.05
0.00
82.22/94.30
0.00
84.25
0.00
Classifier Only
69.93
0.00
73,23
0.00
78.80/89.00
0.00
77.74
0.00
Shallow Prompt 36
0.30M
68.51
+28.7IG
65.69
26.84G
74.20/88.60
+28.71G
74.25
+28.71G
Deep Prompt 126
1.84M
70.78
+6.53G
72.64
+5,59G
78.84/89.44 40
+6,53G
77.92
+6,53G
LoRA 21
0.29M
74.00
0.00
78.82
0,00
79.86/92.60
0.00
81.32
0.00
Adapter 59]
5.34M
74.70
+1.64G
79.93
+1.38G
80.38/91.99 .00
+1.64G
81.73
+1.64G
Scaled PA [17]
3.59M
75.11
+1.12G
80.38
+0.66G
80,40/93.20
+1.12G
82.27
+1.12G
DAS
5.34M
74.80
11.16G
80.11
-5.13G
80.12/91.88
-11.66
81.71
9915G
GGlobll
6.23M
75.09
4.51G
80.69
.3.67G
80.42/91.40
6.06G
81.90
4.74G
ViLT
VQA
NLVR
Flickr30K
Avg. Additional
Updated Parameter
Method
Additional FLOPs
Additional FLOPs
Additional
test-dev
test-P
IR/TR R@1
Per.
FLOPPs
FLOPs
Full Tuning
115.43M
71.26
0.00
76.13
0.00
64.40/83.50
0.00
73.82
0.00
Classifier Only
65.75
0.00
66.08
0.00
57.42/78.00
0.00
66.81
0.00
Shallow Prompt 36
0.15M
66.47
+19,53G
66.47
+19.53G
55,92/74.80
+19,53G
65.92
+19,53G
Deep Prompt 26
1.84M
69.30
+5.14G
73.34
+5,14G
58.64/79.50
+5.14G
70.20
+5.14G
LoRA 21
0.15M
68.44
0.00
72.77
0.00
57.44/77.70
0.00
69.09
0.00
Scaled PA 17
1.80M
70.40
+0.44G
75.13
+0.44G
61.88/79000
+0.44G
71.60
+0.44G
Adapter 599
3.56M
70.85
+0.86G
75,51
+0.86G
62.68/81.44
+0,86G
72.61
+0.86G
DAS
3.56M
69.28
-L,03G
74.89
-1,03G
60.66/80.80
-1.03G
71.41
-1.03G
the skip connection set to 192 to retain a certain capacity. The VL. model first warmed up for one epoch. In this epoch, the subnetwork is randomly sampled according to the skipped number m. Then the search runs epochs and the redundancy observation is executed at 10-th step per interval. Finally, the optimal architecture will be trained for another 10 epochs. Notably, the validation set is used during training for all methods. In terms of METER, we split its fusion layer into two modules, i.e. the vision and language ones, which are skipped separately. The hidden dimension of the adapter used as the skip connection is set to 192 for encoders and 288 for fusion layers. The rest settings the same as ViLT. We conduct all experiments with single NVIDIA Tesla A A100 GPU and the settings not mentioned are the same as ViLT (28) and METER 100
5.3
Experimental results
5.3.1
Quantitative analysis
Comparion with PETL methods. We first compare DAS with a bunch of PETL methods (17 21 26, 36 59] on the VLP models, of which results are given in Tab. [ Here, the suffix of DAS denotes the number of skipped layers, and "fusion" and "global" refer to the range of network skipping, i.e., only the fusion branch or the complete model. From Tab. [ the first observation is that existing PETL methods can largely reduce the amount of updated parameters for VLP models. For instance, the prompt-based methods only require about 1M parameters for two VLP models, nearly 300 times less than full tuning. Meanwhile, their performance gap to fully tuning is also marginal, e.g., Scaled PA [17]. However, we can also see that all these PETI methods cannot reduce the computation of VLP models, and some of them incurs obvious increases in FLOPs, e.g. +28. 71G by Shallow Prompt (36 on METER. The most computation efficient one is LoRA (21). which applies the re-parameterization technique to merge the additional modules into the VLP model, taking no extra computations. However, its performance obviously lags behind other PETL methods and our DAS. In stark contrast, DAS is the only method that can reduce the computation overhead on the downstream VL tasks, e.g., 1.16G FLOPs by - on VQA. More importantly, its updated parameter scale is only slightly larger than Adapter and Scaled PA, while the overall performance is still competitive. These results well confirm the effectiveness of DAS towards PCETL.
Ablation of the number of skipped layers. Tab. 2, we report the results of skipping different numbers of layers by DAS. In terms of METER,
first observe that skipping a few layers
7

---

Table 2: Ablation study of different numbers of skipped layers by DAS. "Fusion" denotes the skipping is only for the fusion branch, while "Global" refers to the entire METER.
DAS
75.12
76.10
Random
MRTKK
Method
Skipped Number
test-dev
VQA
Additional
FL.OPs
test-P
NEVK
Additional
FLOPS
Per.
Avg. Additional
FLOPS
13:45
Baseline
75.28
1,80
8122
0,980
78.25
1,3GG
74.92
99,00G
50.07
2,060
77.30
-5.60
DAS-Fusion
74.80
-17.58G
-11.16G
80.11
-9.70
-1,14G
77.46
-13.78G
-7.65G
70
73.70
-24.00G
79.30
-11.4GG
76.50
-17.2GG
2
75.24
-3.96
81.37
-2.19G
78.31
-3.08G
DAS-Global
75.13
455GG
81.34
-3.7GG
78.24
4.09G
75.02
-5000
80.04
.4.22G
77:53
644
Skipped Modićer
74.95
-3.1GG
79.61
-8. .8344G
77.28
-699G
Figure 3: The comparison between
VILT
DAS and random skipping for ME-
0.730
0.73G
DAS
2
67.64
09:28
1.00G
-2.79
73.00
74.89
-2.79G
-103G
70.32
72.09
-2.79G
-10G
TER the VQA2.0.
Table 3: Computation overhead of different methods for METER on the VQA2.0.
METER
VQA
Additional FLOPs
Training
Testing
Method
test-dev
Memory(G)
Time(h)
Memory(G)
SpeddSampless)
Full Tuning
77,43
0
>40
NIA
6.8
4,16
LoRA
74.00
0
21.5
27
6.8
4.16 (+0.00%)
Adapter
74.70
+L.4G
22.9
7.2
4.09 (-1.68%)
Scaled PAA
75.11
+1.12G
23.1
30
7.1
3.95 (-5.04%)
Global
75.09
-,31G
21.7 (scurch) & 20.6 (train)
10 (search) 20 (train)
6.5
4.37 (+9.83%)
DAS,--ussio
74,80
-11.66
21.7 (search) & 18.1 (Irain)
10 (search) 18 (Irain)
6.5
4.96 (+19.23%)
has limited impact on its performance, e.g., skipping up to 8 fusion layers only has about 2.2% performance drops, strongly suggesting the redundancy of this VLP model. However, DAS can only reduce about one layer for ViLT without notable performance degradations. To explain, METER is a deep VLP model with two independent encoders to extract the features of image and text, while ViLT processes the multi-modal information from image pixels and word embeddings. In this case, ViLT is more compact than METER for downstream tasks, and this is also reflected in their parameter scales and performance. In terms of METER, we can also see the difference in optimizing the fusion branch and the entire model, i.e. DAS-Fusion and DAS-Global. With the same number of skipped layers, DAS-Fusion can reduce more FLOPs since the length of multi-modaa inputs is often larger than the single-modal one. Mcanwhile, when evaluating the entire model, DAS often tends reduce the layers in the language encoders [ which also suggests that natural language understanding is often less difficult in the VL tasks. Overall, these results well confirm our motivation about the redundancy of VLP models in downstream VL tasks, especially the ones with independent encoders like METER.
Reliability of DAS. Considering that DAS is an RL-based search approach, we also examine its stability and reliability via comparing to random skipping, of which results are given in Fig. 3 It can be seen that DAS consistently better than random skipping without regard to the number of skipping layers, well confirming its effectiveness. In particular, when the number of skipped layers increases. the performance deviation of random skipping will become much more obvious, e.g. =1.33 for skipping 8 layers. Instead, DAS is still more stable and superior than the random solution, which also suggests the importance of correct redundancy estimations. Overall, these results further confirm the effectiveness and reliability of the proposed DAS.
Inference Efficiency. We further compare the actual inference efficiencies of DAS and other methods. The computation overhead during both the training and testing stages are reported in Tab. 3 We can first observe that the PETL methods, i.e. LoRA, Adapter and Scaled PA. significantly reduce the computational burden during the training stage. However, during test, these methods lose their efficiency advantage. For instance, Scaled PA is 5.04% slower compared to the full tuning method. In contrast, the proposed DAS enhances the efficiency in both phases. Specifically, DAS only takes similar computation overhead to the PETL methods in the training stage and improves inference efficiency by +19.23%. Overall, these results well confirm the effectiveness of the proposed DAS in the PCETL task.
'The detailed results are given in our Appendix.
8

---

Lang
1.0
1.0
Lang
Dogree Rodundancy
Degree Redundancy
Degrue of Redundancy
P_Lang_2
0.8
0.8
0.8
3_Lang_3
Lang
Lang_
0.4
0.4
0.4
Visual
Visual
0.2
0.2
0.2
Visual 2
Visual
0.0
0.2 0.4 0.6 0.8
1.0 0.0
0.2
0.4
0.6 0.8
1.0
0.0
0.2 0.4 0.6 0.8 1.0
Flitering Progress
Flitering Progress
Flitering Progress
F_ 5
(a) Redundancy on VQA when skipping Modules
(b) Redundancy on VQA
(c) Redunduncy on NLVR when skipping Modules
when skipping 6 Modules
Figure 4: The change of the redundancies of METER's fusion layers. The horizontal axis shows the progress of training and the vertical axis represents the degree of redundancy based on Eq. 5 Lang and F Visual denote the language and visual modules in the fusion branch.
Language
Langaage
Vision
Language Encoder
Vision Encoder
lang
lang
lang
lang
lang
P lang 2
vis
F lang
vis
lang 2
vis
lang
vis
F lang 3
vis
F long
F vis
lang
vis 4
long 4
F.
lang
vis
lang
F vis
lang 5
vis
Multimodal Fusion Modules (12 modules)
Multimedal Fusion Modules (12 modules)
Multimodal Fusion Modules (12 modules)
Skipping modules for VQA
Skipping modules for VQA
(e) Skipping 6 medules for NLVR
The optimal subnetworks METER searched by DAS. Here, the green modules are trainable adapters while the blue ones frozen during adaption.
Figure 5:
5.3.2 Qualitative analysis
To obtain deep insight into DAS, we also visualize its changes of redundancy estimations during search, as depicted in Fig. 4 From these plots, we can observe several patterns of DAS across tasks. The first is that the most redundant layers can quickly emerge during DAS, especially when number skipped layers smaller, e.g. Fig. 4a. Meanwhile, for some uncertain layers, their redundancies can be gradually determined after short period of oscillation. We can also see that the numbers of the skipped language and visual layers are similar on VQA and NLVR. However, the preference about the skipping layers is still different the tasks, see Fig. 4 b and Fig. 4..
In Fig. we visualize the subnetworks METER searched by DAS, which are also the final results of Fig. As discussed above, the preferences of the hidden layers of METER are still different for two tasks. In terms of VQA, DAS tends to discard the language modules at the middle level while skipping the visual ones on the top and bottom of the network. In contrast, the language layers skipped on NLVR are all the top ones. These search results can somewhat reflect the properties of these two tasks. For instance, VQA is reasoning task, thus it focuses more the high-level semantics the input text, while NLVR needs detailed comparison between two images and one sentence, the last few language layers may be more redundant to this task.
Overall, Fig. and Fig. not only confirm the feasibility of DAS towards PCETL, but also yields some interesting findings about the downstream adaption of VLP models.
6 Limitation
Currently, DAS has two main limitations. First, it still needs to set the number of layers to skip. In future work, we will introduce the computation hardware constraints to DAS for the more

---

automatic network skipping. Second, DAS only regards the complete Transformer layers in VLP models as the skipping candidates, limiting its potential in pathway routing. In the future, we will extend the search space with more detailed components, such as MHA and FFN.
7 Conclusion
In this paper, we propose a new problem for vision-language pre-trained (VLP) models termed parameter and computation efficient transfer learning (PCETL). Existing transfer learning solutions for VLP models can only save the parameter expenditure during downstream task adaption, e-g- the PETL ones, while the excessive computation is still unconquered problem. To this end, propose a novel approach called Dynamic Architecture Skipping (DAS) towards effective PCETL. DAS can observe the redundancies of VLP modules to downstream tasks via a reinforcement learning based process, and then skip the redundant ones to speed up inference. Meanwhile, DAS also adopts lightweight adapters serve the hidden feature adaptions and the short-cut connections, thereby reducing the scale of trainable parameters. On two VLP models and three VL tasks, DAS not only shows a great superiority in reducing computation, but is also on par with the PETL methods in terms of parameter overhead and performance.
8 Acknowledge
This work was supported by National Key R&D Program of China NN.20022D01182011) the National Science Fund for Distinguished Young Scholars (No.62025603), the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, , No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), the Natural Science Foundation of Fujian Province of China 200.002 No.2022J06001) and the
China Fundamental Research Funds for the Central Universities (Grant No. 20720220068).
References
[1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[2] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient. object detection models with knowledge distillation. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[3] Jianda Chen, Shangyu Chen, and Sinno Jialin Pan. Storage efficient and dynamic flexible runtime channel pruning via deep reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), pages 14747-14758, 2020.
[4] Ycn-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In European Conference on Computer Vision (ECCV), pages 104-120, 2020.
[5] Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. Template-based named entity recognition using BART. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1835-1845, 2021.
[6] Zihang Dai, Hanxiao Liu, Quoe V. Le, and Mingxing Tan. Coatnet: for all data sizes. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Marrying convolution and attention
[7] Chaitanya Devaguptapu. Samarth Sinha, Joseph. Vineeth N. Balasubramanian, and Animesh Garg. A A-networks for efficient model patching. CORR, abs/2303. 147772, 2023.
[8] Jacob Devlin, Ming Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: pre-training deep bidirectional transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language (eclnologie((AAAALLHLL pages 4171 4186, 2019.
10

---

[9] Alexander Yom Din. Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Short-cutting transformers with linear transformations. CoRR, abs/2303.09435, 2023.
[10] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang. Shuohang Wang. Lijuan Wang. Chenguang Zhu, Pengehuan Zhang, Lu Yuan, Nanyun Peng, al. An empirical study of training end-to-end vision-and- language transformers. Proceedings of the IEER/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18166-18176, 2022.
[11] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra Modha. Learned step size quantization. arXiv preprint 1902:01002.0811 2019.
[12] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In ICLR. OpenReview.net, 2020.
[13] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. Computing Research Repository (CoRR), 2021.
[14] Yash Goyal, Tejas Khot, Douglas Summers-She Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role image understanding in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 6325-6334, 2017.
[15] Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efffcient transfer learning with diff pruning. In Annual Meeting of the Association for Computational Linguistics (ACL.), pages 4884-4896, 2021.
[16] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogério Schmidt Feris. Spottune: Transfer learning through adaptive fine-tuning, In CVPR, pages 4805-4814 14. Computer Vision Foundation / IEEE, 2019.
(17] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efffcient transfer learning. In International Conference on Learning Representations (ICLR), 2022.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual networks. In European Conference on Computer Vision (ECCV), 2016.
[19] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhancee bert with disentangled attention. Computing Research Repository (CoRR), 2020.
[20] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan. and Sylvain Gelly. efrfciete-efffccient transfer learning for NLP, In International Conference Machine Learning (ICML), pages 2790 2799, 2019.
[21] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Reprexentations (ICLR), 2022.
[22] Gao Huang. Shichen Liu, Laurens van der Maaten, and Kilian Q Weinberger. Condensenet: efficient densenet using learned group convolutions. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018
[23] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision-language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11976-12983 2021.
[24] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. Computing Research Repository (CoRR), 2020.
[25] Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, and Richard J Radke. Dynamic distillation network for cross-domann few-shot recognition with unlabeled data, Advances in Neural Information Processing Systems (NeurIPS), pages 3584 3595, 2021.
[26] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongic, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision (ECCV), pages 709-727, 2022.
[27] Andrej Karpathy and Li Fei-Fei. Deep visual- semantic alignments for generating image descriptions. JEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI/ PAMI), pages 664-676, 2017.
11

---

[28] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and aan-languageeeeeeea transformer without convolution or region supervision. In International Conference Machine Learning (ICML), pages 5583-5594, 2021.
[29] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.0m 08342, 2018.
[30] Woosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers. In NeurIPS, 2022.
[31] François Lagunas. Ella Charlaix, Victor Sanh, and Alexander M. Rush. Block pruning for faster transform- ers. In EMNLP, pages 10619-10629, 2021.
[32]
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9045-3033 2021.
[33] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu- Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Systems (NeurlPS), pages 9694 9704499755220 2021.
[34] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. Computing Research Repository (CoRR), 2019.
[35] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. UNTMO: towards unified-modal understanding and generation via cross-modal contrastive learning. In Annual Meeting of the Association for Computational Linguistics (ACL.), pages 2592-2607, 2021.
[36] Xiang Lisa Li and Percy Liang. Prefix-tuning Optimizing continuous prompts for generation. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4582-4597, 2021.
[37] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding. Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. Computing Research Repository (CoRR), 2021.
[38] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. Computing Research Repository (CoRR), 2019.
[39] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining lask-agnostic visiolinguistic representations for vision-and- (NeurIPS), pages 13 23, 2019.
vision-and-languuae language tasks. In Advances in Neural Information Processing Systems
[40] Gen Luo, Minglang Huang. Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, and Rongrong Ji. Towards efficient visual adaption via structural re re-parameterization. Computing Research Repository (CoRR), 2023.
[41] Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision- Language instruction tuning for large language models. CoRR, abs/2305. 150033, 2023.
[42] Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision- lisiun-language instruction tuning for large language models. arXiv preprint arXiv:2305. 15003, 2023.
[43] Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Yan Wang, Liujuan Cao, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Towards lightweight transformer via group-wise transformation for sssion-ndd-anguuaa tasks. IEEE Transactions on Image Processing, 31:3386-3398 2022.
[44] Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Yan Wang, Liujuan Cao, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Towards lightweight transformer via group-wise transformation for vision-and- aniio-agd-language tasks. IEEE Transactions on Image Processing (TIP), 2022.
[45] Gen Luo. Yiyi Zhou, Xiaoshuai Sun, Yongjian Wu, Yue Gao, and Rongrong Ji. Towards language-guided visual recognition via dynamic convolutions. International Journal of Computer Vision, pages 1-19, 2023.
[46] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems (NeurIPS), pages 1022-1035, 2021.
[47] Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efftiient task fine-tuning for transformers via shared hypernetworks. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 565 566-576,220 2021.
12

---

[48] Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma. Jiawei Han, Scott Yih, and Madian Khabsa. Unipelt: A unified framework for parameter-efliiient language model tuning. In Annual Meeting of the Association for Computational Linguisticx (ACL), pages 6253-6264, 2022.
[49] Fanxu Meng, Hao Cheng, Ke Li, Huixiang Luo, Xiaowei Guo, Guangming Lu, and Xing Sun. Pruning filter in filter. Advances in Neural Information Processing Systems (NeurIPS), pages 17629-17640, 2020.
[50] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2463-2244400 2273,227,,2220 2019.
[51] Bryan A. Plummer. Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier. and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-senne -eeeeeenc models. International Journal of Computer Vision (IJCV), pages 74-93, 2017.
[52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748- 8763, 2021.
[53] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 91-99, 2015.
[54] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[55] Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang. Chun Yuan, and Jiaqi Wang. Upop: Unified and progressive pruning for compressing vision-language transformers. In JCML, volume 202, pages 31292 31311. PMLR, 2023.
[56] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: pre-training of generic visual-linguistic representations. In International Conference on Learning Representations (ICLR), 2020.
[57] Alane Suhr, Stephanie Zhou, Ally Zhang, Tris Zhang, Huajun Bai, and Yoav Artzi, A corpus for reasoning about natural language grounded in photographs. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 6418-6412 6428, 2019.
[58] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. LST: ladder side-tuning for parameter and memory efficient transfer learning. Computing Research Repository (CoRR), 2022.
[59] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VL-ADAPTER: parameter-efficient transfer learning for lainn-and tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5217-5227, 2022.
[60] Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks, In Advances in Neural Information Processing Systems (NeurIPS), pages 24193-24205, 2021.
[61] Hao Tan and Mohit Bansal, LXMERT: leamnigg mross-modality encoder representations from transformers. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5099-51 51 10, 2019.
[62] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quae V. Le. Mnasnet: Platform- aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), pages 5998-6008, 2017.
[64] Tiannan Wang, Wangchunshu Zhou, Yan Zeng, and Xinsong Zhang. Efficientvim: Fast and accurate vision-language models via knowledge distillation and modal-adaptive pruning. In ACL (Findings), pages 13899-13913, 2023.
[65] Zirui Wang. Jiahui Yu, Adams Wei Yu. Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In International Conference on Learning Representations (ICLR), 2022.
13

---

[66] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-awar efficient convnet design via differentiable neural architecture search. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[67] Zuxuan Wu, Tushar Nagarajan. Abhishek Kumar, Steven Rennie, Larry S. Davis, Kristen Grauman, and Rogério Schmidt Feris. Blockdrop: Dynamic inference paths in residual networks. In CVPR, pages 8817-8826. Computer Vision Foundation/ / JEEE Computer Society, 2018.
[68] Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. ofrt-of-theseur theseus: Compressing BERT by progressive module replacing. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7859-7866 2020.
[69]
Hao Yu and Jianxin Wu. A unified pruning framework for vision transformers. Sci. China Inf. Sci., 66(7), 2023.
[70] Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron C. Courville. Can subnetwork structure be the key out-of-distribution generalization? In Marina Mcila and Tong Zhang, editors, ICMI, volume 139, pages 12356-12367. PMLR, 2021.
[71] Jeffiey O. Zhang, Alexander Sax, Amir Zamir, Leonidas J. Guibas, and Jitendra Malik. Side-tunings A baseline for network adaptation via additive side networks. European Conference on Computer Vision (ECCV), pages 698-714, 2020.
[72] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-fres free clip-adapter for better -aiinnl modeling. Computing Research Repository (CoRR), 2021.
[73] Yunshan Zhong, Mingbao Lin, Mengzhao Chen, Ke Li, Yunhang Shen, Fei Chao, Yongjian Wu, and Rongrong Ji. Fine-grained data distribution aligmment for post-training quantization. In European Conference on Computer Vision (ECCV), pages 70-86, 2022.
[74] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16795-16804, 2022.
[75] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV), pages 2337-2348, 2022.
[76] Yiyi Zhou, Rongrong Ji, Gen Luo, Xiaoshuai Sun, Jinsong Su, Xinghao Ding, Chia-Wen Lin, and Qi Tian. A real-time global inference network for one-stage referring expression comprehension. IEFE Transactions on Neural Networks and Learning Systems, 2021.
[77] Yiyi Zhou, Rongrong Ji, Xiaoshuai Sun, Jinsong Su, Deyu Meng, Yue Gao, and Chunhua Shen. Plenty plague: Fine-grained learning for visual question answering. IEEE transactions on pattern analysis and machine intelligence, 44(2):697-709, 2019.
[78] Yiyi Zhou, Tianhe Ren, Chaoyang Zhu, Xiaoshuai Sun, Jianzhuang Liu, Xinghao Ding, Mingliang Xu. and Rongrong Ji. Trar: Routing the attention spans in transformer for visual question answering. In P Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2074-2084, 2021.
14

---

Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models
Qiong Wu Wei Yu Yiyi Zhou 12 Shubin Huang', Xiaoshuai Sun 12 Rongrong J Key Laboratory of Multimedia Trusted Perception and Efficient Computing. Ministry of Education of China, Xiamen University, 361005, P.R. China. Institute of Artificial Intelligence, Xiamen University, 361005, P.R. China.
{qiong, weiyu)@stu. xmu, edu. en, zhouyiyi@xmu. edu. en, shubinhuang@stu. xmu, edu. cn, (xssun, rrji)]xmu.eduu
A
The detailed skipping results
Visiom Embedding
Embedding
LEAU
LEni
En6
Enl
LPn7
LEn8 LEa10
FI.4
En8
Ená
En9
Fn4
VFa0 VEal
Vin2 VEa3
Viir VPaS
Ea10
+H+ VEA0 VEntoVEal1
Enil
VBa6 VEa7
VEn8
Transformer Enceder Layers
CLIP-VIT
(4) VILT
(b) METER
Figure 1: Architectures of the baseline models (a) ViLT and (b) METER. The blue modules are the default Transformer layers that are frozen during the adaptation, while the green ones are the trainable adapters. "En" denotes the encoding layers. "LEn" and "FEn" represent the encoding layers of METER for texts and images, and "FL" and "FV" are the fusion layers for language and vision, respectively.
The architectures of two based models are given in Fig. We also report their detailed skipping results by DAS in Tab, A Here, "LEn" represents Language Encoder, and "VEn" represents Vision Encoder. We can first see that ViLT is a relatively compact model to METER, which only has 12 Transformer layers without any sedaalltt--pecifie encoder. In this case, it can only be skipped one or two layers without obviously degrading the performance. In stark contrast, METER is deep and huge VLP model, which redundancy much more obvious. By skipping up 8 layers, its performance drops are still marginal on all tasks. Meanwhile, we also observe that discarding its visual encoder layers will greatly disturb its training and performance during experiments, thus these layers are not considered the skipping candidates. From Tab, A we also have some interesting observations. For instance, the language encoding layers are less important to VQA. This may suggest that most questions in VQA2.0 are shorter and less complex, and the model needs to focus more the visual understanding and cross-modal interactions. This case is less significant on NLVR², which requires detailed comparison between images and texts. Overall, these results confirm that the large VLP models exhibit obvious redundancy to downstream VL tasks. More importantly, the importance of their modules is different to different tasks, requiring proper estimations.
Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table A: The kipped layers and performance for different base models and tasks. For VQA, report the test-Dev as the performance. For NLVR², we report the test-P as the performance. For Flickr30k, report IR/TR @@) the performance. "Fusion" refers to only skipping the layers in the multimodal fusion modules of METER, while "Global" denotes the skipping scope of the fusion modules and the language encoder.
METER
Datasets
Candidates
Number
Skipped
Pet:
Additional
FLOPs
Skipped Layers
75.28
1.6G
FV4
Fusion
74.80
-11.160
FL2. FL3, IVO,FVS
74.67
-17.58G
FLI. FL2, FL2, FL3, FVO. FV4, PVS
VQA
73.70
-24.00G
FL,2, FL., FIA, FVO, FVI, PVS
75.24
33996
FVG, LEmb
LFnll
Global
75.02
-5.06G
FVO, LEe4, LEn6, LEn8, LEat0, LBnll
74.05
3.61G
FV4, LER4, LEa5,, LEi6,, LEn8, LEa9, LEn10, LEall
80.07
-2,66G
FV3 FL2, FL3, FVS
Fusion
80.11
4.14G
78.16
-9.7G
FL3, FL4, FLS, FVI, FV3.
N
79.30
-11.45G
FLL FL2, PL3, FL4, EVO, FV3, FV4
81.34
-3.67G
FL3, PVS, LEn4 FL2, FLS, FL6, LEnS, LEn6, LEall
Global
80.04
4422G
79.61
8.34G
FL3, FL4, FVI, PVI, PVS, LEs4, LEall
81 20/92.40
1688
Flickr30k
Fusion Global
12/91890
-11166
FL3, EVO,
80.42/91.41
4,066
FLS, FVO, LEF8
ViLT
Number Skipped
Additional FLOPs
Datasets
Candidates
Pet.
Skipped Layers
70.13
0.736
VQA
69.28
-1.03G
Global
67.64
-2.79G
Eni. En3
76.26
0.73G
N
Globul
74.89
-100G
73.00
2,79G
En5, Enl1
62.4 02.44/22.10
0.73G
Plickr3kk
Global
00.66/80.80
-1,03G
B
The results of random sampling
Table B: The detailed experiment results of random sampled subnetworks for Fig.3 in the main paper.
METER
Datasets
Candidates
Number of Skipped
VQA test-Dev
Additional
Skipped Layers
FLOPs
74.24
-11.16G
FL2, FL4, FV2, FV3
4
74.67
-11.6G
FLI. FLS, FVO,
74.08
-11.6G
FLI. FVI.
-17,58G
FL3, FV2, FVS
Fusion
73.26
-17,58G
FLO. FLI, FL4, FVO, FV2, FV5
73.03
-17,58G
FL2, FLA, FLS, FVI, FV2, PV5
71.81
24.00G
FLAL FL,T HL3, FLS, FV2, FV3, FV4, FVS FL3, FL4, FLS, FVI, FV2. FV4 FLO, FL2, FLS, FVI, FV2, FV3, FV4, PV5
8
68.56
-2400GG
FV4
69.88
-24.00G
Tab. B gives the detailed results of random sampling mentioned in Fig.3 of the main paper. We can see that random sampling not only consistently worse than our DAS, but also varies greatly in terms of skipped layers and performance, especially when the number of skipped layers is large. On contrary, these results just confirm the effectiveness of the proposed DAS.
C Generalization on Pre-trained Language Model
To validate the generalization ability of DAS, we also apply it to pre-trained language model called RoBERTa [5], shown in Tab. C Due to the time limit, we do not conduct careful tunings for

---

Table C: Comparison between DAS and PETL methods for RoBERTa MNLI and SST2. "En" denotes the encoding layers. "Acc." denotes the accuracy.
RoBERlia
Updated
Additional FLOPs
MNLI
SST2
Methods
Parameter
Acc.
Skipped Layers
Acc.
Skipped Layers
Full Tuning
124.65M
0.0
87.6
94.6
Bit-Fit [6]
0.10M
0.0
84.7
93.7
Pre-fix
0.14M
1.20G
86.3
94.0
LoRA
0.59M
0.0
87.2
94.2
Adapter
0.63M
0.33G
87.2
94.2
MAM
0.61M
0.79G
87.4
94.2
DAS1
0.63M
-3.71G
86.8
Enl()
94.1
En10
DAS2
0,63M
-7.74G
86.7
En10, Enll
93.9
En8, En10
DAS3
0.63M
-11.77G
86.2
En9, En10, Enll
93.8
En7, En8, En10
RoBERTa. The settings of DAS follow the main paper, while the From this table, we can first see that DAS is also applicable to pre-trained language models. can also achieve the target of PCETL in terms of computation and update parameter scales, while obtaining limited performance drops However, we can also see that the competitiveness of DAS to other PETL methods is slightly worse on MNLI, of which objective is close the pre-training ones. We think that the task gap may be a potential factor affecting PCETL., Overall, these results well validate the generalization ability of DAS on LLMMs towards PCETL..
the same with MAM (
References
[1] Junxian Hc, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatriiee and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations (ICLR), 2022.
[2] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruma Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference Machine Learning (ICML). pages 2790-2799, 2019.
[3] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
[4] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4582-4597, 2021.
[5] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. Roberta: robustly optimized BERT pretraining approach. Computing Research Repository (CoRR). 2019.
[6] Elad Ben Zaken, Shauli Ravfogel, Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer- based masked language-monl models. arXiv preprint urXiv:2106.10133 2021.
3


Reviews:
[{"review_id": "oRszhOtGXH", "reviewer": "Reviewer_LDaL", "summary": "This work focuses on the problem of transfer learning in the context of vision-language pre-trained (VLP) models. Existing works that adapt VLP models primarily address the issue of parameter efficient transfer learning (PETL). However, these methods do not effectively reduce the computation complexity of VLP models. Therefore, this work takes PETL a step further and introduces a new setting called parameter and computation efficient transfer learning (PCETL). The goal of PCETL is to reduce the computation complexity of pre-training models, enabling faster inference, while only tuning a fraction of parameters.\n\nTo achieve PCETL, a method called dynamic architecture skipping (DAS) is proposed. DAS assists in finding the optimal subnetwork routing of VLP models for downstream tasks. Although the concept of dynamic architecture skipping has been extensively explored in transfer learning for image spaces, it has not yet been investigated in the era of large-scale pre-trained models. The proposed approach demonstrates promising results on two VLP models and three vision-language benchmarks.\"", "questions": "1. Looks like random skipping is a very strong baseline for this work (Line 256-257). The performance difference between random skipping and the proposed approach appears marginal, yet considering that random skipping is less costly, are these gains still substantial?\n2. Could you please explain the rationale for modeling this as a k-armed bandit problem? Additionally, what would be the primary challenges in solving this problem in an end-to-end differentiable manner?", "limitations": "Limitations Section discusses the limitations of this work clearly", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The concept of dynamic skipping, as yet unexplored in the era of large-scale VLP models, holds considerable promise. Its introduction to VLP models could significantly reduce computational and time complexity.\n3. As the daily release of pre-trained large-scale models continues to increase, methods for efficiently transferring learned features become increasingly significant.\n4. The experimental results compellingly illustrate the potential of the proposed methodology.", "weaknesses": "1. The concept of dynamic architecture skipping is not new and is already well-established in the case of transfer learning for standard imagery (see references below). It would be beneficial if the authors could provide reasons why existing approaches are unsuitable for the current problem and setting.\n2. The proposed approach is not end-to-end differentiable and models the problem as a k-armed bandit. A more comprehensive end-to-end approach would be ideal (refer to references 2 and 4).\n3.  The paper lacks a comparison with existing dynamic architecture skipping methods, which is a crucial element for an encompassing evaluation.\n4. The authors acknowledge that the number of layers to be skipped must be manually determined. Some of the transfer learning methodologies listed below are capable of handling this automatically, borrowing ideas from there can benefit this work. \n\n\n**References**\n1. [BlockDrop: Dynamic Inference Paths in Residual Networks](https://arxiv.org/abs/1711.08393)\n2. [SpotTune: Transfer Learning through Adaptive Fine-tuning](https://arxiv.org/abs/1811.08737)\n3. [Can Subnetwork Structure be the Key to Out-of-Distribution  Generalization?](https://arxiv.org/abs/2106.02890)\n4. [$\\Delta$-Networks for Efficient Model Patching](https://arxiv.org/abs/2303.14772)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer #LdaL\nWe highly appreciate your time and effort in reviewing this paper, as well as your encouraging and constructive comments on our work. Below, we response to your key concerns point by point.\n\n**Comment 1:** The concept of dynamic architecture skipping is not new and is already well-established in the case of transfer learning for standard imagery (see references below).\n\n**Response:** Thanks for this suggestion. In this paper, we propose a new task called Parameter and Computation Efficient Transfer Learning (PCETL) for large-scale pre-trained models, which should not only reduces the computation redundancy but also needs to avoid the expensive full fine-tune on various downstream tasks. \n\nThis is a new field yet unexplored, and we find it difficult for existing solutions to directly achieve the above goals simultaneously. Thus, we propose a novel approach called Dynamic Architecture Skipping based on the k-armed bandit theory. \n\nIn terms of your mentioned methods, most of them have obvious shortcomings towards PCETL. BlockDrop is an example-dependent method that uses a policy network to predict the routing path for each input. Its combination with PETL methods like Adapter is intractable, since the problem definition as well as the training scheme should be largely changed. SpotTune is an approach for choosing which layers to finetune rather than to skip. Δ-networks is unable to reduce the computation since its weighted connections only serve to adaption. \n\nA potential solution is MRM, which learns gating functions to decide layering options. In fact, its computation reduction is very non-deterministic, and in some cases, it does not skip redundant layers, see Table A.\n\n**Comment 2:**  A more comprehensive end-to-end approach would be ideal (refer to references 2 and 4).\n\n**Response:** Thanks for this comment. Following your suggestion, we compare our DAS with MRM [3] and Δ-Networks [4] in the following table.\n\nTable. A: The comparison between DAS and alternative methods.\n\n| Method|Updated Parameter|Training Memory (GB)|Training Time|VQA  test-dev|VQA  Additional FLOPs|NLVR2 test-P|NLVR2 Additional FLOPs |\n| -|-|-|-|-|-|-|- |\n| Full Tuning|323.31M|\\>40|N/A|77.43|0.0|83.05|0.00 |\n| Classifier Only|\\-|21.4|27h|69.93|0.0|73.23|0.00 |\n| MRM(1e-5)|6.23M|26.3 (search) /24.0 (training)|8h (search) +22h (training)|75.28|+1.80%|81.39|-4.16% |\n| Δ-Networks|36|21.5|27h|67.34|+0.00%|71.49|+0.00% |\n| DAS2-Global|6.23M|21.6 (search) /20.5 (training)|11h (search) +22h (training)|75.24|-4.25%|81.37|-4.16% |\n| DAS4-Global|6.23M|21.7 (search) /20.6 (training)|10h (search) +20h (training)|75.09|-4.84%|80.69|-6.95% |\n\nWe can first observe that the adaption of Δ-Networks is much inferior to our DAS, and it also cannot reduce the computation as we mentioned above. \nMRM can be extended to a PETL method when combined with Adapters. \nHowever, its computational efficiency is unstable and difficult to directly constrain on different VL tasks. Simply put, MRM is likely not to choose to skip layers during training. More importantly, its GPU memory overhead is much more expensive than our method. \n\nOverall, our DAS is still the best choice considering efficiency and effectiveness.\n\n**Comment 3:** The paper lacks a comparison with existing dynamic architecture skipping methods.\n\n**Response:**  Following your suggestion, we supplement the comparison with MRM in Tab. A, from which we can see that our DAS is the best trade-off between efficiency and performance.\n\n**Comment 4:** The authors acknowledge that the number of layers to be skipped must be manually determined. Some of the transfer learning methodologies listed below are capable of handling this automatically, borrowing ideas from there can benefit this work.\n\n**Response:** Thanks for this insightful comment. In fact, the manual setup of DAS is an advantage over the automatic solutions. \n\nTo explain, different from Network Architecture Search (NAS), the pre-trained VL models have a fixed structure, which greatly limits the choice of network skipping. In this case, gradient-based methods can only rely on the training loss to automatically select which layer should be skipped, and this process is unstable and uncontrollable. As shown in Tab. A, MRM keeps all Transformer layers of METER on VQA2.0, but our method can reduce up to four layers with similar performance.\n\nOverall, our DAS can better meet the PCETL requirement in practice.\n\n**Comment 5:** Looks like random skipping is a very strong baseline for this work (Line 256-257)?\n\n**Response:**  In fact, the number described in Line 256-257 refers to the performance deviation of random skipping. The actual performance gains of our method are obvious, e.g. +5.6\\% when skipping 8 layers, as shown in Fig. 3. \n\n**Comment 6:** Could you please explain the rationale for modeling this as a K-armed bandit problem? Additionally, what would be the primary challenges in solving this problem in an end-to-end differentiable manner?\n\n**Response:** Thanks for this question. As discussed above, in PCETL, the structures of VLP models are fixed, so we cannot change the network depth to reduce the computation just like NAS. In this case, we model network skipping as a k-armed bandit problem, *i.e.*, which k layers can be skipped, and evaluate the policy via numerous single-shot samplings. This setting can directly set the computation cost for PCETL. \n\nIn contrast, in PCETL, the gradient-based or differentiable methods like MRM are uncontrollable for layer skipping, as shown in Tab. A, since its search results are attributed to the training loss rather than the pre-defined target. \n\n**References**\n\n[1] BlockDrop: Dynamic Inference Paths in Residual Networks.\n\n[2] SpotTune: Transfer Learning through Adaptive Fine-tuning.\n\n[3] Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?\n\n[4] Δ-Networks for Efficient Model Patching."}, {"review_id": "rS9klvCnYi", "reviewer": "Reviewer_pBnL", "summary": "The paper presents dynamic architecture skipping (DAS), which can drop some transformer layers during inference. The routing of DAS is learned by reinforcement learning. To make the training parameter efficient, every layer has an adapter for training. After training, DAS drops several redundant layers and replaces them with adapters to reduce the inference FLOPs.", "questions": "1. The training pipeline is more complex than the other approaches since the training involves search, redundancy observation, and final fine-tuning. I wonder how much cost (e.g. training time and memory) is needed to train the method compared to other approaches?\n2. Are the results on one run or multiple runs? If it is on one run, I would suggest using the average of multiple runs to justify the robustness of the approach.\n3. Given the GFLOPs saving, how much the approach can improve the inference speed and inference memory?\n\n---\n**Post-rebuttal**\nThank you for the authors' response. I have read it and it addressed my questions.", "limitations": "The limitation in the paper is sufficient.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Most parameter-efficient training methods add extra cost in inference, and it is interesting to explore how to reduce the cost.\n2. The design of the proposed approach, which replaces some transformer layers of the backbone model with adapters, is reasonable.", "weaknesses": "1. Several baselines [2] and related works [1, 2] are missing. I think the paper would be stronger if it can compare to [2].\n2. I am not sure what is the uniqueness of the approach only applies to the VL domain. If the approach is general, I would expect applying the approach to some other models and tasks too (LLM or ViT).\n\n[1] Fan, Angela, Edouard Grave and Armand Joulin. “Reducing Transformer Depth on Demand with Structured Dropout.”\n\n[2] Din, Alexander Yom, Taelin Karidi, Leshem Choshen and Mor Geva. “Jump to Conclusions: Short-Cutting Transformers With Linear Transformations.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer #pBnL\nWe highly appreciate your time and effort in reviewing this paper, as well as your encouraging and constructive comments on our work. Below, we response to your key concerns point by point. \n\n**Comment 1:** Several baselines [2] and related works [1, 2] are missing. I think the paper would be stronger if it can compare to [2].\n\n**Response:**  Thanks for this suggestion. We will cite and discuss your mentioned excellent works. Following your suggestion, we supplement the comparison with J2C [2] in the following table, which is also combined with adapters like our DAS. \n\nTable A: The comparison between DAS and the suggested baseline for METER.\n\n| Method|Updated Parameter|VQA test-dev|VQA Additional FLOPs|NLVR2 test-P|NLVR2  Additional FLOPs |\n| -|-|-|-|-|- |\n| Full Tuning|323.31M|77.43|0%|83.05|0.00|\n| Classifier Only|-|69.93|0%|73.23|0.00 |\n| J2C-2|4.18M|67.34|-11.52%|68.89|-8.83% |\n| J2C-4|3.58M|69.26|-18.60%|69.08|-15.95% |\n| DAS4-Fusion|5.34M|74.80|-11.97%|80.11|-9.70% |\n| DAS6-Fusion|5.34M|75.67|-18.86%|79.30|-17.72% |\n\nIt can be seen that in the absence of layer redundancy evaluation, the effect of layer skipping by J2C is not satisfactory, and its performance greatly lags behind our DAS.\n\n**Comment 2:** I am not sure what is the uniqueness of the approach only applies to the VL domain. If the approach is general, I would expect applying the approach to some other models and tasks too (LLM or ViT).\n\n**Response:** Thanks for this constructive suggestion. We have applied our DAS to LLaMA-7B [a] on ScienceQA [b] following the settings of LaVIN [c]. The results are given in Table. B. \n\nTable B: Comparison of DAS and PETL methods on ScienceQA for LLaMA.\n\n| Method|Update Params|FLOPs(G)|Modality Natural|Modality Social|Modality Language|Context Text|Context Image|Context No|Grade G1-6|Grade G7-12|Avg |\n|-|-|-|-|-|-|-|-|-|-|-|-|\n| LLaVA-13B|13B|-|90.36|95.95|88.00|89.49|88.00|90.66|90.93|90.90|90.92 |\n| LaVIN-7B|3.8M|833|89.25| 94.94| 85.24| 88.51| 87.46| 88.08| 90.16| 88.07| 89.41 |\n| DAS4-7B| 44.26M|729 (-18.61%)|90.54|94.26|86.82|89.74|87.65|89.76|90.97|89.26|90.36 |\n| DAS6-7B|44.26M|678 (-24.85%)|89.96|94.71|87.18|89.00|87.7|89.97|90.75|89.32|90.24 |\n\nBy saving up to 25% FLOPs, our DAS can even achieve better results than LaVIN on LLaMA-7B. Notably, its best performance is very close to LLaVA[d] with LLaMA-13B while saving much more expenditure. These results well confirm our DAS towards the target of PCETL. \n\n**Comment 3:** The training pipeline is more complex than the other approaches since the training involves search, redundancy observation, and final fine-tuning. I wonder how much cost (e.g. training time and memory) is needed to train the method compared to other approaches?\n\n**Response:** Thanks for this comment. Following your suggestion, we report the training costs of DAS and the compared methods in the following table. \n\nTable C: Comparison of DAS and PETL methods on efficiency for METER.\n\n| Method|VQA test-dev|Training Memory (GB)|Training Time |\n|-|-|-|-|\n| Full Tuning|77.43|\\>40|N/A |\n| LoRA|74.00|21.5|27h |\n| Adapter|74.70|22.9|28h |\n| Scaled PA|75.11|23.1|30h |\n| DAS4-Global|75.09|21.7 (search) / 20.6 (training)|10h (search) + 20h (training) |\n\nIt can be seen that our memory overhead is similar to LoRA during layer searching, and it can be further slightly reduced during training. Meanwhile, the search process is quick, and the training hours are also fewer than the PETL methods since DAS has fewer adapters to train. In this case, the overall training expenditure is not significantly more expensive than the PETL methods.\n\n**Comment 4:** Are the results on one run or multiple runs? If it is on one run, I would suggest using the average of multiple runs to justify the robustness of the approach.\n\n**Response:** Thanks for this suggestion. In fact, the search result of our DAS is very stable. On METER, we test DAS with three random seeds, and the skipped layers are the same. In this case, we only report the training result on one run, similar to the PETL methods. Following your suggestion, we will provide the multiple-run ones in our new version.\n\n**Comment 5:** Given the GFLOPs saving, how much the approach can improve the inference speed and inference memory?\n\n**Response:**  Thanks for this question. The detailed improvements are given in the following tables. In terms of METER, the memory saving is about 4.4%, while the inference speeds up to 19.14\\%.  Similar improvements can be also seen on LLaMA-7B, while the performance is even better.\n\nTable D: Comparison of DAS and PETL methods on inference efficiency for METER.\n\n| Method|VQA Test-Dev|Inference Memory (GB)|Inference Speed(Sample/s)|FLOPs |\n|-|-|-|-|-|\n| Full Tuning|77.43|6.8|133.27|93.2 |\n| LoRA|74.00|6.8|133.27 (+0.00%)|93.2 (-0.0%) |\n| Adapter|74.70| 7.2|130.93 (-1.75%)|94.9 (+1.82%) |\n| Scaled PA|75.11|7.1|126.50 (-5.08%)|94.3 (+1.18%) |\n| DAS4-Global|75.09|6.5|146.34 (+9.81%)|88.7 (-4.82%) |\n| DAS4-Fusion|74.80|6.4|158.79 (+19.14%)|82.1 (-11.9%) |\n\nTable E: Comparison of DAS and PETL methods on inference efficiency for LLaMA-7B.\n\n| Method|Avg|Inference Memory (GB)|Inference Speed (sample/s)|FLOPs (G) |\n|-|-|-|-|-|\n| LaVIN-7B|89.41|40.1|3.51|833 |\n| DAS4-7B|90.36|36.5|3.88 (+10.54%)|729 (-18.61%) |\n| DAS6-7B|90.24|34.7|4.09 (+16.52%)|678 (-24.85%) |\n\n**Reference**\n\n[1] Reducing Transformer Depth on Demand with Structured Dropout.\n\n[2] Jump to Conclusions: Short-Cutting Transformers With Linear Transformations.\n\n[a] LLaMA: Open and Efficient Foundation Language Models.\n\n[b] Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering.\n\n[c] Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models.\n\n[d] Visual Instruction Tuning."}, {"review_id": "bomiOxH7ru", "reviewer": "Reviewer_5vAx", "summary": "The paper proposes a dynamic architecture skipping (DAS) method for the parameter and computation efficient transfer learning (PCETL) problem. DAS explores the optimal short-cut pathway in VLP models. Extensive experiments show the effectiveness of DAS both in reducing computation and parameters.", "questions": "N/A. Already written in the weakness section.", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper introduces a novel and intriguing approach by considering network jumps as k-armed bandit sampling. It highlights the significance of reducing computational complexity in Visual Language Pretraining (VLP) models, providing strong motivation for the proposed methodology. The paper is well-written and effectively communicates its ideas. Experiments are solid, with necessary analysis and ablations for different parts of the method.", "weaknesses": "Compared to the number of parameters and FLOPS, it would be better to add the training time and GPU memory cost as metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 5vAx\nWe highly appreciate your time and effort in reviewing this paper. Your comments and feedback are instrument to the improvement of our work.\n\n**Comment 1:** Compared to the number of parameters and FLOPS, it would be better to add the training time and GPU memory cost as metrics.\n\n**Response:** Thanks for this comment. Following your suggestion, we report the training time and GPU memory cost in the following table. \n\n| Method | VQA Test-Dev | Training Memory (GB) | Training Time |\n|-|-|-|-|\n| Full Tuning | 77.43 | \\>40G | N/A |\n| LoRA | 74.00 | 21.5 | 27h |\n| Adapter | 74.70 | 22.9 | 28h |\n| Scaled PA | 75.11 | 23.1 | 30h |\n| DAS4-Global | 75.09 | 21.7 (search) / 18.1 (training) | 10h (search) + 20h (training) |\n| DAS4-Fusion | 74.80 | 21.7 (search) / 20.6 (training) | 10h (search) + 18h (training) |\n\nIt can be seen that our memory overhead is similar with LoRA during layer searching, and it can be further slightly reduced during training. Meanwhile, the search process is quick, and the training hours are also fewer than the PETL methods since DAS has fewer adapters to train. In this case, the overall training expenditure is not significantly more expensive than the PETL methods."}, {"review_id": "ZTPtBZv6LM", "reviewer": "Reviewer_fJhT", "summary": "This paper aims to achieve both parameters and computation efficiency for the transfer learning of pre-trained VLMs. They combine the previous ladder-side tuning adapter with the proposed dynamic architecture skipping technique to achieve such a goal.", "questions": "1. Why the method is called \"Dynamic Architecture Skipping\"? Where the dynamic comes from? From my point of view, the skipped modules are fixed after training and it will not change with the input during inference.", "limitations": "As I discuss in the Weaknesses, the authors do not fully address the limitations of the proposed mehtods.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "It is a good idea to achieve both parameters and computation efficiency for the transfer learning of VLM.", "weaknesses": "Although it is a good idea to achieve both parameters and computation efficiency for the transfer learning of VLM, the author does not show the pros and cons of the proposed methods clearly.\n\n1. The training cost of the DAS and the benefit of the DAS is not clear. In Table 1, the authors do not show the running time in practice. FLOPs are not equal to the inference speed, especially the reduction of FLIOPs is marginal. Plus, please provide the training cost of DAS.\n\n2. Why only take experiments on classification tasks? The authors claim they propose a new problem, PCETL, but lack experiments on import VL tasks - image captioning. Is the parameter pruning of VLM not suitable for the image captioning tasks? Does the layer dropping break the text generation ability of pre-trained VLM?\n\n3. Why only provide retrieval results on Flickr30K, it is small and easy. A large model may be unnecessary on Flickr30K.  Please provide retrieval results on MSCOCO.\n\n4. How does the proposed DAS compare to the pruning method in EfficientVLM[1] or other layer pruning methods in the pruning area? Will DAS be necessary? A random baseline in Figure 3 is not enough.\n\n\n[1] https://arxiv.org/pdf/2210.07795.pdf\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer #fJhT\nWe highly appreciate your time and effort in reviewing this paper, and your valuable feedback has been instrumental in improving the paper. Below, we response to your key concerns point by point. \n\n**Comment 1:** Although it is a good idea to achieve both parameters and computation efficiency for the transfer learning of VLM, the author does not show the pros and cons of the proposed methods clearly. The training cost of the DAS and the benefit of the DAS is not clear. In Table 1, the authors do not show the running time in practice. FLOPs are not equal to the inference speed, especially the reduction of FLIOPs is marginal. Plus, please provide the training cost of DAS.\n\n**Response:** Thanks for this constructive comment. Following your suggestion, we report these results in the following table.\n\nTable A: Comparison of DAS and PETL methods on efficiency for METER.\n\n|Method|VQA Test-Dev|Training Memory (GB)|Training Time|Inference Memory (GB)|Inference Speed (Sample/s)|\n|-|-|-|-|-|-|\n|Full Tuning|77.43|\\>40G|N/A|6.8|133.27 |\n|LoRA|74.00|21.5|27h|6.8|133.27 (+0.00%)|\n|Adapter | 74.70 | 22.9 | 28h | 7.2 | 130.93 (-1.75%) |\n|Scaled PA | 75.11 | 23.1 | 30h | 7.1 | 126.50 (-5.08%) |\n|DAS4-Global | 75.09 | 21.7 (search) / 18.1 (training) | 10h (search) + 20h (training) | 6.5 | 146.34 (+9.81%) |\n|DAS4-Fusion | 74.80 | 21.7 (search) / 20.6 (training) | 10h (search) + 18h (training) | 6.5 | 158.79 (+19.14%) |\n\nIt can be first seen that our training expenditure is comparable to most PETL methods. Our memory overhead is similar with LoRA during layer searching, and it is also slightly reduced during training. Meanwhile, the search process is quick, and the training hours are also fewer than the PETL methods since DAS has fewer adapters to train. In this case, the overall training expenditure is indeed not expensive. \n\nThe inference efficiency is also notable. The memory saving is about 4.4%, while the inference speeds up to 19.14\\%.  Similar improvements can be also seen on LLaMA-7B, while the performance is even better.\n\n**Comment 2:** Why only take experiments on classification tasks? The authors claim they propose a new problem, PCETL, but lack experiments on import VL tasks - image captioning. Is the parameter pruning of VLM not suitable for the image captioning tasks? Does the layer dropping break the text generation ability of pre-trained VLM?\n\n**Response:** Thanks for this constructive comment. Following your suggestion, we apply our DAS to LLaMA and report its results on ScienceQA [a], a generative QA benchmark. Here, we follow the settings of LaVIN [c], which can be regarded as our baseline. \n\nTable B: Comparison of DAS and PETL methods on ScienceQA for LLaMA.\n\n| Method | Updated Parameters | FLOPs | Modality Natural | Modality Social | Modality Language | Context  Text | Context Image | Context No | Grade G1-6 | Grade G7-12 |  Avg  |\n|-|-|-|-|-|-|-|-|-|-|-|-|\n| LLaVA-13B  | 13B | - | 90.36 | 95.95 | 88.00 | 89.49 | 88.00 | 90.66 | 90.93 | 90.90 | 90.92 |\n| LaVIN-7B | 3.8M | 833 | 89.25 | 94.94 | 85.24 | 88.51 | 87.46 | 88.08 | 90.16 | 88.07 | 89.41 |\n| DAS4-7B | 44.26M | 729 (-18.61%) | 90.54 | 94.26 | 86.82 | 89.74 | 87.65 | 89.76 | 90.97 | 89.26 | 90.36 |\n| DAS6-7B | 44.26M | 678 (-24.85%) | 89.96 | 94.71 | 87.18 | 89.00 | 87.7 | 89.97 | 90.75 | 89.32 | 90.24 |\n\nIt can be seen that our DAS can not only greatly reduces the FLOPs and even improve the performance by skipping 6 layers of LLaMA, which is indeed significant. These results also validate the generalization of DAS on text generation tasks.\n\n**Comment 3:** Why only provide retrieval results on Flickr30K, it is small and easy. A large model may be unnecessary on Flickr30K. Please provide retrieval results on MSCOCO.\n\n**Response:** Thanks for this suggestion. Following your suggestion, we report the retrieval results on MSCOCO in the following table, where the target of PCETL can be still met on this benchmark. \n\nTable C: Comparison of DAS and Finetuning methods on COCO retrieval for METER.\n\n|Method|Update Parameter | Additional FLOPs | COCO IR@1 | COCO IR@5 | COCO IR@10 | COCO TR@1 | COCO TR@5 | COCO TR@10 |\n|-|-|-|-|-|-|-|-|-|\n| Full Tuning | 323.31M | 0.0 | 54.85 | 81.41 | 89.31 | 72.96 | 92.02 | 96.26 |\n| DAS4-Fusion | 5.34M | -9.54% | 54.22| 79.36 | 87.67 | 71.56 | 91.17 | 94.79 |\n| DAS4-Global | 6.23M | -8.68% | 54.60| 80.36 | 88.42 | 72.06 | 91.42 | 95.42 |\n\n**Comment 4**: How does the proposed DAS compare to the pruning method in EfficientVLM[1] or other layer pruning methods in the pruning area? Will DAS be necessary? A random baseline in Figure 3 is not enough.\n\n**Response:** Thanks for recommending this excellent work. However, pruning methods like EfficientVLM are not applicable to PCETL. On one hand, most pruning methods require another full tuning on the downstream tasks, against the target of PCETL. On the other hand, existing PETL methods like Adapter are unable to be combined with the pruning methods, since pruning methods often skip/prune parameter-wise components. We will cite and discuss EfficientVLM in our new version.\n\n**Comment 5:** Why the method is called \"Dynamic Architecture Skipping\"? Where the dynamic comes from? From my point of view, the skipped modules are fixed after training and it will not change with the input during inference.\n\n**Response:** Thanks for this question. Large pre-trained models like LLaMA are often transferred to various down-stream tasks for practical use. In this case, we think that our DAS can provide optimal inference paths for each task during its real-world applications. \n\n**Reference**\n\n[1] EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning.\n\n[a] Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering.\n\n[b] LLaMA: Open and Efficient Foundation Language Models.\n\n[c] Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models."}]

Author rebuttal:
We highly appreciate AC for pushing forward NeurIPs 2023, and also thank all reviewers for their valuable and encouraging comments on this paper, such as \`\`*Its introduction to VLP models could significantly reduce computational and time complexity.*\'\' by Reviewer LDaL, \`\`*The experimental results compellingly illustrate the potential of the proposed methodology.*\'\' by Reviewer LDaL, \`\`*The design of the proposed approach is reasonable.*\'\' by Reviewer pBnL, \`\`*introduces a novel and intriguing approach*\'\' by Reviewer 5vAx , \`\`*It highlights the significance of reducing computational complexity in Visual Language Pretraining (VLP) models*\'\' by Reviewer 5vAx, *et al*.

During the rebuttal phrase, our main responses include:

1. The details of training costs, including GPU memory and training time, are similar to most PETL methods.

2. The actual inference speed-up, which can be +19% in practice.

3. The comparison with alternative skipping methods, where our merits can be still witnessed.

4. The application to LLaMA on ScienceQA, where the target of PCETL is still achieved by our methods.

Meanwhile, the key concerns of all reviews are also point-by-point responded in each rebuttal.

Here, we would like to emphasize our key contributions again:

1. We raise a new problem called Parameter and Computation Efficient Transfer Learning (PCETL) for VLP models.

2. We propose a novel Dynamic Architecture Skipping (DAS) for PCETL, which can greatly reduce the computation redundancy on downstream tasks.

Lastly, the new results in rebuttal will be added to our final version, and our source codes will be publicly released after acceptance.

Best,

The authors.

Meta-review: This paper proposes a PETL (parameter-efficient transfer learning) method by utilizing dynamic architecture skipping. It receives 3 positive reviews and 1 slightly negative reviews. The raised issues include similarity upon existing DAS methods, application range, and experimental  validation. In the rebuttal phase, the authors provide sufficient explanation and results to convince 3 reviewers remain positive. While the 4th reviewer is slightly negative,  the remaining issues are minor from the PETL perspective. Overall, the AC has checked all the files, and welcomes this designing reporting in this venue. The authors shall incorporate the additional results and analysis in the camera-ready version.
Decision: Accept (poster)