SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ViLT
Method	Updated Parameter	VQA	NLVR2	Flickr30K	Avg.
test-dev	Additional FLOPs	test-P	Additional FLOPs	IR/TR R@1	Additional FLOPs	Per.	Additional FLOPs
Full Tuning	115.43M	71.26	0.00	76.13	0.00	64.40/83.50	0.00	73.82	0.00
Classifier Only	-	65.75	0.00	66.08	0.00	57.42/78.00	0.00	66.81	0.00
Shallow Prompt [36]	0.15M	66.47	+19.53G	66.47	+19.53G	55.92/74.80	+19.53G	65.92	+19.53G
Deep Prompt [26]	1.84M	69.30	+5.14G	73.34	+5.14G	58.64/79.50	+5.14G	70.20	+5.14G
LoRA [21]	0.15M	68.44	0.00	72.77	0.00	57.44/77.70	0.00	69.09	0.00
Scaled PA [17]	1.80M	70.40	+0.44G	75.13	+0.44G	61.88/79.00	+0.44G	71.60	+0.44G
Adapter [59]	3.56M	70.85	+0.86G	75.51	+0.86G	62.68/81.40	+0.86G	72.61	+0.86G
DAS1	3.56M	69.28	-1.03G	74.89	-1.03G	60.66/80.80	-1.03G	71.41	-1.03G
the skip connection is set to 192 to retain a certain capacity. The VLP model is first warmed up for one epoch. In this epoch, the subnetwork is randomly sampled according to the skipped number  $m$ . Then the search runs 2 epochs and the redundancy observation is executed at 10-th step per interval. Finally, the optimal architecture will be trained for another 10 epochs. Notably, the validation set is used during training for all methods. In terms of METER, we split its fusion layer into two modules, i.e. the vision and language ones, which are skipped separately. The hidden dimension of the adapter used as the skip connection is set to 192 for encoders and 288 for fusion layers. The rest settings are the same as ViLT. We conduct all experiments with a single NVIDIA Tesla A100 GPU and the settings not mentioned are the same as ViLT [28] and METER [10].
5.3 Experimental results
5.3.1 Quantitative analysis
Comparison with PETL methods. We first compare DAS with a bunch of PETL methods [17, 21, 26, 36, 59] on the VLP models, of which results are given in Tab. 1. Here, the suffix of DAS denotes the number of skipped layers, and "fusion" and "global" refer to the range of network skipping, i.e., only the fusion branch or the complete model. From Tab. 1, the first observation is that existing PETL methods can largely reduce the amount of updated parameters for VLP models. For instance, the prompt-based methods only require about 1M parameters for two VLP models, nearly 300 times less than full tuning. Meanwhile, their performance gap to fully tuning is also marginal, e.g., Scaled PA [17]. However, we can also see that all these PETL methods cannot reduce the computation of VLP models, and some of them incurs obvious increases in FLOPs, e.g. +28.71G by Shallow Prompt [36] on METER. The most computation efficient one is LoRA [21], which applies the re-parameterization technique to merge the additional modules into the VLP model, taking no extra computations. However, its performance obviously lags behind other PETL methods and our DAS. In stark contrast, DAS is the only method that can reduce the computation overhead on the downstream VL tasks, e.g., -11.16G FLOPs by  $\mathrm{DAS}_4$ -Fusion on VQA. More importantly, its updated parameter scale is only slightly larger than Adapter and Scaled PA, while the overall performance is still competitive. These results well confirm the effectiveness of DAS towards PCETL.
Ablation of the number of skipped layers. In Tab. 2, we report the results of skipping different numbers of layers by DAS. In terms of METER, we can first observe that skipping a few layers
Table 2: Ablation study of different numbers of skipped layers by DAS. "Fusion" denotes the skipping is only for the fusion branch, while "Global" refers to the entire METER.
METER
Method	Skipped Number	VQA	NLVR2	Avg.
test-dev	Additional FLOPs	test-P	Additional FLOPs	Per.	Additional FLOPs
Baseline	0	75.28	1.68G	81.28	0.99G	78.28	1.34G
DAS-Fusion	2	74.92	-9.06G	80.07	-2.66G	77.50	-5.86G
4	74.80	-11.16G	80.11	-4.14G	77.46	-7.65G
6	74.67	-17.58G	78.16	-9.97G	76.42	-13.78G
8	73.70	-24.00G	79.30	-11.45G	76.50	-17.72G
DAS-Global	2	75.24	-3.96G	81.37	-2.19G	78.31	-3.08G
4	75.13	-4.51G	81.34	-3.67G	78.24	-4.09G
6	75.02	-5.06G	80.04	-4.22G	77.53	-4.64G
8	74.95	-5.61G	79.61	-8.34G	77.28	-6.97G
ViLT
Baseline	0	70.13	0.73G	76.26	0.73G	73.20	0.73G
DAS	1	69.28	-1.03G	74.89	-1.03G	72.09	-1.03G
2	67.64	-2.79	73.00	-2.79G	70.32	-2.79G
Figure 3: The comparison between DAS and random skipping for ME-TER on the VQA2.0.
Table 3: Computation overhead of different methods for METER on the VQA2.0.
METER
Method	VQATest-dev	Additional FLOPs	Training	Testing
Memory(G)	Time(h)	Memory(G)	Speed(Sample/s)
Full Tuning	77.43	0	>40	N/A	6.8	4.16
LoRA	74.00	0	21.5	27	6.8	4.16 (+0.00%)
Adapter	74.70	+1.64G	22.9	28	7.2	4.09 (-1.68%)
Scaled PA	75.11	+1.12G	23.1	30	7.1	3.95 (-5.04%)
DAS4-Global	75.09	-4.51G	21.7 (search) & 20.6 (train)	10 (search) + 20 (train)	6.5	4.57 (+9.85%)
DAS4-Fusion	74.80	-11.16G	21.7 (search) & 18.1 (train)	10 (search) + 18 (train)	6.5	4.96 (+19.23%)
has limited impact on its performance, e.g., skipping up to 8 fusion layers only has about  $2.2\%$  performance drops, strongly suggesting the redundancy of this VLP model. However, DAS can only reduce about one layer for ViLT without notable performance degradations. To explain, METER is a deep VLP model with two independent encoders to extract the features of image and text, while ViLT processes the multi-modal information from image pixels and word embeddings. In this case, ViLT is more compact than METER for downstream tasks, and this is also reflected in their parameter scales and performance. In terms of METER, we can also see the difference in optimizing the fusion branch and the entire model, i.e. DAS-Fusion and DAS-Global. With the same number of skipped layers, DAS-Fusion can reduce more FLOPs since the length of multi-modal inputs is often larger than the single-modal one. Meanwhile, when evaluating the entire model, DAS often tends to reduce the layers in the language encoders<sup>5</sup>, which also suggests that natural language understanding is often less difficult in the VL tasks. Overall, these results well confirm our motivation about the redundancy of VLP models in downstream VL tasks, especially the ones with independent encoders like METER.
Reliability of DAS. Considering that DAS is an RL-based search approach, we also examine its stability and reliability via comparing to random skipping, of which results are given in Fig. 3. It can be seen that DAS is consistently better than random skipping without regard to the number of skipping layers, well confirming its effectiveness. In particular, when the number of skipped layers increases, the performance deviation of random skipping will become much more obvious, e.g.  $\pm 1.33$  for skipping 8 layers. Instead, DAS is still more stable and superior than the random solution, which also suggests the importance of correct redundancy estimations. Overall, these results further confirm the effectiveness and reliability of the proposed DAS.
Inference Efficiency. We further compare the actual inference efficiencies of DAS and other methods. The computation overhead during both the training and testing stages are reported in Tab. 3. We can first observe that the PETL methods, i.e. LoRA, Adapter and Scaled PA, significantly reduce the computational burden during the training stage. However, during test, these methods lose their efficiency advantage. For instance, Scaled PA is  $-5.04\%$  slower compared to the full tuning method. In contrast, the proposed DAS enhances the efficiency in both phases. Specifically, DAS only takes similar computation overhead to the PETL methods in the training stage and improves inference efficiency by  $+19.23\%$ . Overall, these results well confirm the effectiveness of the proposed DAS in the PCETL task.
The detailed results are given in our Appendix.
Figure 4: The change of the redundancies of METER's fusion layers. The horizontal axis shows the progress of training and the vertical axis represents the degree of redundancy based on Eq. 5. F_Lang and F_Visual denote the language and visual modules in the fusion branch.
(a) Redundancy on VQA when skipping 4 Modules
(b) Redundancy on VQA when skipping 6 Modules
(c) Redundancy on NLVR when skipping 6 Modules
(a) Skipping 4 modules for VQA
(b) Skipping 6 modules for VQA
(c) Skipping 6 modules for NLVR
Figure 5: The optimal subnetworks of METER searched by DAS. Here, the green modules are trainable adapters while the blue ones are frozen during adaption.
5.3.2 Qualitative analysis
To obtain deep insight into DAS, we also visualize its changes of redundancy estimations during search, as depicted in Fig. 4. From these plots, we can observe several patterns of DAS across tasks. The first is that the most redundant layers can quickly emerge during DAS, especially when the number of skipped layers is smaller, e.g. Fig. 4.a. Meanwhile, for some uncertain layers, their redundancies can be gradually determined after a short period of oscillation. We can also see that the numbers of the skipped language and visual layers are similar on VQA and NLVR. However, the preference about the skipping layers is still different on the two tasks, see Fig. 4.b and Fig. 4.c.
In Fig. 5, we visualize the subnetworks of METER searched by DAS, which are also the final results of Fig. 4. As discussed above, the preferences of the hidden layers of METER are still different for two tasks. In terms of VQA, DAS tends to discard the language modules at the middle level while skipping the visual ones on the top and bottom of the network. In contrast, the language layers skipped on NLVR are all the top ones. These search results can somewhat reflect the properties of these two tasks. For instance, VQA is a reasoning task, thus it focuses more on the high-level semantics of the input text, while NLVR needs a detailed comparison between two images and one sentence, so the last few language layers may be more redundant to this task.
Overall, Fig. 4 and Fig. 5 not only confirm the feasibility of DAS towards PCETL, but also yields some interesting findings about the downstream adaption of VLP models.
Currently, DAS has two main limitations. First, it still needs to set the number of layers to skip. In our future work, we will introduce the computation or hardware constraints to DAS for the more
automatic network skipping. Second, DAS only regards the complete Transformer layers in VLP models as the skipping candidates, limiting its potential in pathway routing. In the future, we will extend the search space with more detailed components, such as MHA and FFN.
In this paper, we propose a new problem for vision-language pre-trained (VLP) models termed parameter and computation efficient transfer learning (PCETL). Existing transfer learning solutions for VLP models can only save the parameter expenditure during downstream task adaption, e.g., the PETL ones, while the excessive computation is still a unconquered problem. To this end, we propose a novel approach called Dynamic Architecture Skipping (DAS) towards effective PCETL. DAS can observe the redundancies of VLP modules to downstream tasks via a reinforcement learning based process, and then skip the redundant ones to speed up inference. Meanwhile, DAS also adopts lightweight adapters to serve the hidden feature adaptions and the short-cut connections, thereby reducing the scale of trainable parameters. On two VLP models and three VL tasks, DAS not only shows a great superiority in reducing computation, but is also on par with the PETL methods in terms of parameter overhead and performance.
8 Acknowledge
This work was supported by National Key R&D Program of China (No.2022ZD0118201), the National Science Fund for Distinguished Young Scholars (No.62025603), the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), the Natural Science Foundation of Fujian Province of China (No.2021J01002, No.2022J06001) and the China Fundamental Research Funds for the Central Universities (Grant No. 20720220068).
[1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[2] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[3] Jianda Chen, Shangyu Chen, and Sinno Jialin Pan. Storage efficient and dynamic flexible runtime channel pruning via deep reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), pages 14747-14758, 2020.
[4] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In European Conference on Computer Vision (ECCV), pages 104–120, 2020.
[5] Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. Template-based named entity recognition using BART. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1835-1845, 2021.
[6] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[7] Chaitanya Devaguptapu, Samarth Sinha, K. J. Joseph, Vineeth N. Balasubramanian, and Animesh Garg.  $\Delta$ -networks for efficient model patching. CoRR, abs/2303.14772, 2023.
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171–4186, 2019.
[66] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[67] Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S. Davis, Kristen Grauman, and Rogério Schmidt Feris. Blockdrop: Dynamic inference paths in residual networks. In CVPR, pages 8817-8826. Computer Vision Foundation / IEEE Computer Society, 2018.
[68] Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. Bert-of-theseus: Compressing BERT by progressive module replacing. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7859-7869, 2020.
[69] Hao Yu and Jianxin Wu. A unified pruning framework for vision transformers. Sci. China Inf. Sci., 66(7), 2023.
[70] Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron C. Courville. Can subnetwork structure be the key to out-of-distribution generalization? In Marina Meila and Tong Zhang, editors, ICML, volume 139, pages 12356-12367. PMLR, 2021.
[71] Jeffrey O. Zhang, Alexander Sax, Amir Zamir, Leonidas J. Guibas, and Jitendra Malik. Side-tuning: A baseline for network adaptation via additive side networks. In European Conference on Computer Vision (ECCV), pages 698-714, 2020.
[72] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. Computing Research Repository (CoRR), 2021.
[73] Yunshan Zhong, Mingbao Lin, Mengzhao Chen, Ke Li, Yunhang Shen, Fei Chao, Yongjian Wu, and Rongrong Ji. Fine-grained data distribution alignment for post-training quantization. In European Conference on Computer Vision (ECCV), pages 70-86, 2022.
[74] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16795-16804, 2022.
[75] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV), pages 2337-2348, 2022.
[76] Yiyi Zhou, Rongrong Ji, Gen Luo, Xiaoshuai Sun, Jinsong Su, Xinghao Ding, Chia-Wen Lin, and Qi Tian. A real-time global inference network for one-stage referring expression comprehension. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[77] Yiyi Zhou, Rongrong Ji, Xiaoshuai Sun, Jinsong Su, Deyu Meng, Yue Gao, and Chunhua Shen. Plenty is plague: Fine-grained learning for visual question answering. IEEE transactions on pattern analysis and machine intelligence, 44(2):697-709, 2019.
[78] Yiyi Zhou, Tianhe Ren, Chaoyang Zhu, Xiaoshuai Sun, Jianzhuang Liu, Xinghao Ding, Mingliang Xu, and Rongrong Ji. Trar: Routing the attention spans in transformer for visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2074-2084, 2021.
<sup>1</sup> Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China.
$^{2}$  Institute of Artificial Intelligence, Xiamen University, 361005, P.R. China. {qiong, weiyu}@stu.xmu.edu.cn, zhouyiyi@xmu.edu.cn, shubinhuang@stu.xmu.edu.cn, {xssun, rrji}@xmu.edu.cn
A The detailed skipping results
Figure 1: Architectures of the baseline models (a) ViLT and (b) METER. The blue modules are the default Transformer layers that are frozen during the adaptation, while the green ones are the trainable adapters. "En" denotes the encoding layers. "LEn" and "FEn" represent the encoding layers of METER for texts and images, and "FL" and "FV" are the fusion layers for language and vision, respectively.
The architectures of two based models are given in Fig. 1. We also report their detailed skipping results by DAS in Tab. A. Here, "LEn" represents Language Encoder, and "VEn" represents Vision Encoder. We can first see that ViLT is a relatively compact model to METER, which only has 12 Transformer layers without any modality-specific encoder. In this case, it can only be skipped one or two layers without obviously degrading the performance. In stark contrast, METER is a deep and huge VLP model, of which redundancy is much more obvious. By skipping up to 8 layers, its performance drops are still marginal on all tasks. Meanwhile, we also observe that discarding its visual encoder layers will greatly disturb its training and performance during experiments, thus these layers are not considered as the skipping candidates. From Tab. A, we also have some interesting observations. For instance, the language encoding layers are less important to VQA. This may suggest that most questions in VQA2.0 are shorter and less complex, and the model needs to focus more on the visual understanding and cross-modal interactions. This case is less significant on  $\mathrm{NLVR}^2$ , which requires a detailed comparison between images and texts. Overall, these results confirm that the large VLP models exhibit obvious redundancy to downstream VL tasks. More importantly, the importance of their modules is different to different tasks, requiring proper estimations.
Table A: The kipped layers and performance for different base models and tasks. For VQA, we report the test-Dev as the performance. For NLVR $^2$ , we report the test-P as the performance. For Flickr30k, we report IR/TR R@1 as the performance. "Fusion" refers to only skipping the layers in the multimodal fusion modules of METER, while "Global" denotes the skipping scope of the fusion modules and the language encoder.
METER
Datasets	Candidates	Number of Skipped	Per.	Additional FLOPs	Skipped Layers
VQA	-	0	75.28	1.68G	-
Fusion	2	74.92	-9.06G	FV0, FV4
4	74.80	-11.16G	FL2, FL3, FV0, FV5
6	74.67	-17.58G	FL1, FL2, FL3, FV0, FV4, FV5
8	73.70	-24.00G	FL1, FL2, FL3, FL4, FV0, FV1, FV4, FV5
Global	2	75.24	-3.96G	FV0, LEn6
4	75.13	-4.51G	FV0, LEn10, LEn11
6	75.02	-5.06G	FV0, LEn4, LEn6, LEn8, LEn10, LEn11
8	74.05	-5.61G	FV4, LEn4, LEn5, LEn6, LEn8, LEn9, LEn10, LEn11
NLVR2	-	0	81.28	0.99G	-
Fusion	2	80.07	-2.66G	FL4, FV5
4	80.11	-4.14G	FL2, FL3, FL5, FV5
6	78.16	-9.97G	FL3, FL4, FL5, FV1, FV3, FV4
8	79.30	-11.45G	FL1, FL2, FL3, FL4, FL5, FV0, FV3, FV4
Global	2	81.37	-2.19G	FV5, LEn1
4	81.34	-3.67G	FL2, FL3, FV5, LEn4
6	80.04	-4.22G	FL2, FL5, FL6, LEn5, LEn6, LEn11
8	79.61	-8.34G	FL2, FL3, FL4, FL5, FV1, FV5, LEn4, LEn11
Flickr30k	-	0	81.20/92.40	1.68G	-
Fusion	4	80.12/91.80	-11.16G	FL4, FL5, FV0, FV3
Global	4	80.42/91.40	-6.06G	FL2, FL5, FV0, LEn8
Datasets	ViLT
Candidates	Number of Skipped	Per.	Additional FLOPs	Skipped Layers
VQA	-	0	70.13	0.73G	-
Global	1	69.28	-1.03G	En3
2	67.64	-2.79G	En1, En3
NLVR2	-	0	76.26	0.73G	-
Global	1	74.89	-1.03G	En5
2	73.00	-2.79G	En5, En11
Flickr30k	-	0	62.44/82.10	0.73G	-
Global	1	60.66/80.80	-1.03G	En7
B The results of random sampling
Table B: The detailed experiment results of random sampled subnetworks for Fig.3 in the main paper.
METER
Datasets	Candidates	Number of Skipped	VQATest-Dev	AdditionalFLOPs	Skipped Layers
VQA	Fusion	4	74.24	-11.16G	FL2, FL4, FV2, FV3
74.67	-11.16G	FL1, FL5, FV0, FV3
74.08	-11.16G	FL1, FL4, FV1, FV4
6	74.05	-17.58G	FL1, FL2, FL3, FV2, FV3, FV5
73.26	-17.58G	FL0, FL1, FL4, FV0, FV2, FV5
73.03	-17.58G	FL2, FL4, FL5, FV1, FV2, FV5
8	71.81	-24.00G	FL0, FL1, FL3, FL5, FV2, FV3, FV4, FV5
68.56	-24.00G	FL1, FL3, FL4, FL5, FV1, FV2, FV3, FV4
69.88	-24.00G	FL0, FL2, FL5, FV1, FV2, FV3, FV4, FV5
Tab. B gives the detailed results of random sampling mentioned in Fig.3 of the main paper. We can see that random sampling is not only consistently worse than our DAS, but also varies greatly in terms of skipped layers and performance, especially when the number of skipped layers is large. On the contrary, these results just confirm the effectiveness of the proposed DAS.
C Generalization on Pre-trained Language Model
To validate the generalization ability of DAS, we also apply it to a pre-trained language model called RoBERTa [5], as shown in Tab. C. Due to the time limit, we do not conduct careful tunings for
Table C: Comparison between DAS and PETL methods for RoBERTa on MNLI and SST2. "En" denotes the encoding layers. "Acc." denotes the accuracy.
RoBERTa
Methods	Updated Parameter	Additional FLOPs	MNLI	SST2
Acc.	Skipped Layers	Acc.	Skipped Layers
Full Tuning	124.65M	0.0	87.6	-	94.6
Bit-Fit [6]	0.10M	0.0	84.7	-	93.7
Pre-fix [4]	0.14M	1.20G	86.3	-	94.0
LoRA [3]	0.59M	0.0	87.2	-	94.2
Adapter [2]	0.63M	0.33G	87.2	-	94.2
MAM [1]	0.61M	0.79G	87.4	-	94.2
DAS1	0.63M	-3.71G	86.8	En10	94.1	En10
DAS2	0.63M	-7.74G	86.7	En10, En11	93.9	En8, En10
DAS3	0.63M	-11.77G	86.2	En9, En10, En11	93.8	En7, En8, En10
RoBERTa. The settings of DAS follow the main paper, while the rest are the same with MAM [1]. From this table, we can first see that DAS is also applicable to pre-trained language models. It can also achieve the target of PCETL in terms of computation and update parameter scales, while obtaining limited performance drops. However, we can also see that the competitiveness of DAS to other PETL methods is slightly worse on MNLI, of which objective is close to the pre-training ones. We think that the task gap may be a potential factor affecting PCETL. Overall, these results well validate the generalization ability of DAS on LLMs towards PCETL.