SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 3: Low-resource results are evaluated on full test sets. We report average results across 5 runs with different initialization. Bold marks the best result among all competing methods. Underline marks the best result among all prompt tuning methods. We report the average of accuracy and F1 for both MRPC and QQP, and average of Pearson and Spearman correlation coefficients for STS-B. For all other tasks, we report accuracy.
Method	MPQA	Subj	CR	MR	SST-2	QNLI	RTE	MRPC	STS-B	QQP	Avg
K=100
Fine-tuning (EFL)	86.2±0.4	88.4±0.8	83.7±2.4	81.4±1.0	86.2±1.3	77.7±1.5	84.2±1.2	72.6±3.7	84.1±1.6	78.1±0.4	82.2
Adapter-tuning (Compacter)	81.0±2.9	88.7±0.8	84.7±2.1	83.7±0.7	85.7±0.9	75.6±0.8	84.7±0.6	80.0±0.9	78.1±1.4	77.1±0.6	81.9
prompt tuning	75.9±1.6	86.8±0.8	72.9±1.4	74.1±1.4	82.9±2.0	82.7±0.2	86.5±0.6	80.0±1.3	70.2±3.1	76.5±0.4	78.9
P-Tuningv2	74.3±2.9	89.7±0.8	80.1±1.0	82.5±1.1	85.1±1.6	78.2±0.5	83.6±0.7	80.1±0.6	78.8±3.0	76.8±0.5	80.9
S-IDPG-PhM	79.0±3.7	87.6±1.1	75.0±1.6	76.2±1.3	87.6±1.3	80.4±1.2	86.3±0.5	79.3±0.4	70.9±2.5	76.1±0.6	79.8
S-IDPG-DNN	78.0±2.1	84.2±1.6	76.3±4.5	77.4±0.5	89.6±1.2	81.1±0.8	87.4±0.8	78.8±1.3	70.6±2.8	74.1±0.9	79.8
M-IDPG-PhM-GloVe	76.6±2.0	90.7±0.4	80.6±2.6	83.0±1.5	85.6±0.8	77.9±1.3	84.4±0.9	79.6±0.9	77.8±1.6	76.1±0.7	81.2
M-IDPG-PhM	75.5±4.6	90.5±0.6	80.2±1.5	82.5±1.1	85.9±1.2	78.8±1.6	84.0±0.4	79.9±0.8	79.3±0.4	77.1±0.2	81.4
K=500
Fine-tuning (EFL)	85.1±1.7	94.1±0.4	90.9±0.6	87.6±0.5	92.5±0.6	85.7±0.6	57.5±1.0	82.3±0.6	88.8±0.5	79.0±0.3	84.3
Adapter-tuning (Compacter)	86.0±0.8	94.9±0.2	89.5±1.0	88.5±0.2	91.9±0.9	82.2±0.6	83.9±0.8	82.7±0.5	86.6±0.5	78.9±0.3	86.5
prompt tuning	82.4±1.3	91.2±0.1	86.8±0.4	84.6±0.8	88.6±1.0	86.3±0.4	86.5±0.4	80.0±0.4	77.4±1.9	77.8±0.3	84.2
P-Tuningv2	84.0±1.3	94.6±0.3	89.0±1.8	88.1±0.5	91.3±0.7	84.6±0.8	84.2±1.5	83.2±0.7	83.8±0.5	78.6±0.3	86.1
S-IDPG-PhM	81.6±2.7	91.4±0.7	85.8±2.0	85.8±0.5	88.5±1.3	85.0±0.4	86.3±1.3	81.9±0.8	78.3±1.5	78.1±0.3	84.3
S-IDPG-DNN	84.8±0.7	90.8±0.6	89.7±1.0	86.1±2.8	90.4±1.6	84.8±0.3	87.7±0.7	82.0±1.4	79.1±2.3	77.1±0.4	85.3
M-IDPG-PhM-GloVe	84.0±1.7	95.0±0.2	89.0±1.1	88.1±0.5	90.4±1.3	85.1±0.1	84.0±1.0	82.3±0.5	84.1±0.8	78.2±0.8	86.0
M-IDPG-PhM	85.2±1.1	94.6±0.0	89.1±1.6	88.8±0.4	91.6±1.1	84.9±0.9	83.9±0.7	82.5±0.5	84.2±0.5	78.6±0.3	86.3
K=1000
Fine-tuning (EFL)	87.7±0.7	95.1±0.2	89.8±1.2	89.2±0.5	93.6±0.4	88.0±0.7	87.3±1.3	87.9±0.9	90.8±0.2	79.8±0.3	88.9
Adapter-tuning (Compacter)	88.2±0.6	95.6±0.3	89.9±1.4	90.0±0.3	92.9±0.2	85.2±0.7	86.8±0.7	86.1±0.6	89.6±0.5	79.9±0.3	88.4
prompt tuning	83.9±2.0	92.6±0.4	87.2±1.4	86.7±0.3	89.9±1.0	86.9±0.1	86.4±0.7	82.5±0.3	82.9±1.3	78.6±0.3	85.8
P-Tuningv2	87.0±0.9	95.9±0.4	88.3±1.5	89.5±0.3	93.2±0.5	87.4±0.4	85.1±1.1	82.6±1.1	87.8±0.3	79.3±0.4	87.6
S-IDPG-PhM	83.4±1.7	93.4±0.9	89.2±0.8	88.0±0.9	90.2±1.0	85.5±0.6	86.9±0.6	83.1±0.4	83.9±0.8	78.9±0.4	86.3
S-IDPG-DNN	85.9±0.8	93.3±1.2	89.9±0.8	89.6±1.1	92.2±0.8	85.2±1.3	87.7±0.8	82.5±0.9	84.7±0.9	78.0±0.8	86.9
M-IDPG-PhM-GloVe	86.5±0.7	95.5±0.3	87.7±1.3	89.3±0.4	93.4±0.3	87.5±0.3	84.9±0.9	82.7±0.7	87.6±0.3	79.1±0.7	87.4
M-IDPG-PhM	87.7±0.5	95.6±0.2	89.2±1.2	89.8±0.4	93.7±0.6	87.2±0.5	85.6±0.6	82.5±0.9	87.8±0.8	79.1±0.4	87.8
where  $x_{i}$  is the input sequence with  $k$  tokens  $t_1, \ldots, t_k$ . According to Table 1, using GloVe as sentence encoder to generate prompts doesn't sacrifice much performance over the ten tasks and outperforms prompt tuning and P-tuning v2. It indicates that our model does not benefit a lot from a strong contextual pre-trained LM. Instead, a light sentence encoder such as GloVe can also help the tasks. Also, instance-dependent prompt tuning shows promising improvement over non-instance-dependent prompt tuning models. One of the drawbacks of our method is that it is twice as expensive to run compared to Compacter, even though it uses slightly fewer parameters. Adopting GloVe as sentence encoder would avoid going through the LM twice, thus effectively reducing IDPG's run-time complexity by half.
4.5.2 Prompt Generator: PHM or DNN?
To reduce the tuning parameters, we substitute the DNN layers with PHM layers. An open question we seek to answer is what is the best generation model for prompt regardless of training parameters. Hence, we compare the PHM-based prompt generator with the DNN-based prompt generator, as shown in Table 1. We observe that including DNN as a generator doesn't improve performance signif
Figure 3: Comparison between three different multilayer generator models (S, M, L versions), and comparison between taking layer 0's output or previous layer's output as input.
icantly, with  $+0.1\mathrm{pt}$  gain on average, while adding 87K parameters (with hidden size  $m = 16$ ). On the other hand, this ablation study further verifies PHM layers' efficiency in the generation model.
4.5.3 Multi-layer Architecture Exploration
When applying the instance-dependent generation model  $\mathbf{G}$  into a multi-layer case, the first challenge we face is the considerable increase in training parameters. If each transformer layer requires an independent generator  $\mathbf{G}_i$ , the number of training
parameters increases  $N$  times, where  $N$  is the number of transformer layers (24 in RoBERTa-Large). Assuming  $\mathbf{G}$  has the form  $y = \mathbf{W}x + b$ , there are three alternatives: (i) Smallest version (S version): sharing both  $W$  and  $b$ ; (ii) Middle version (M version): sharing  $W$  and making  $b$  layer-specific; and (iii) Largest version (L version): making both  $W$  and  $b$  layer-specific.
Another way to reduce the training parameters is by adjusting the hidden size  $m$  of the generator. We compare two models with  $m = 16$  and  $m = 256$ . Surprisingly, we find that generator with a hidden size 16 is not far from the large model (92.0 vs. 92.1, respectively, in M version). We hypothesize that the smaller hidden size of 16 is already enough to store useful instance information, and setting  $m$  too large may be less efficient.
Besides, in single-layer prompt generation model, the input to  $\mathbf{G}$  is  $M(x_{i})$  - the representation of input sequence  $x_{i}$ . In a multi-layer case, the input to each layer generator has another option, i.e., the previous layer's output. However, as shown in Figure 3, the experiment results suggest no significant difference between the two input ways. As for the generator selection, the three models perform as expected (S version  $< \mathrm{M}$  version  $< \mathrm{L}$  version). In Table 1, M-IDPG-PhM uses the previous layer's output as input, M version as the generator, and 16 as the generator hidden size. Detailed information for all models' performance on each task can be found in Appendix A.3.
4.5.4 Prompt Insertion: Single-layer or Multi-layer?
P-tuning v2 (Liu et al., 2021a) conducted substantial ablation studies on the influence of inserting prompt into different transformer layers. To boost single-layer IDPG performance, we add supplementary training (cf. Appendix A.4) and conduct ablation studies in Appendix A.5. We come to a similar conclusion that multi-layer instance-dependent prompt tuning model (M-IDPG) is significantly better than the single-layer method (S-IDPG) in both evaluation settings. An interesting finding is that the impact of supplementary training on S-IDPG is high while it is limited for M-IDPG.
4.5.5 How Prompts Help?
Given two sentences, we encode each of them by one of the comparison models and compute the cosine similarity. We sort all sentence pairs in STS-B dev set in descending order by the cosine
Figure 4: The number of pairs of each group in Top-200 cosine similarity ranking. More results can be found in Appendix A.6.
similarity scores and get a distribution for number of pairs in each group that is included in Top-k ranking. We compare a vanilla model without any prompts with M-IDPG-PHM. Both models are finetuned on STS-B training set. As shown in Figure 4, prompts bring the similar sentences closer while pushing the dissimilar ones apart.
4.5.6 IDPG Scalability
We study our proposed model's scalability in this section. In general, the performance of IDPG in downstream tasks improves gradually when using a larger prompt length (Cf. Appendix A.7).
Supplementary Training: Existing works (Phang et al., 2018; Liu et al., 2019) have observed that starting from the fine-tuned MNLI model results in a better performance than directly from the vanilla pre-trained models for RTE, STS, and MRPC tasks. A series of work (SentenceBERT (Reimers and Gurevych, 2019), BERT-flow (Li et al., 2020), SimCSE (Gao et al., 2021b)) explored intermediate training to improve STS tasks. All of them applied pre-fine tuning on NLI datasets. More recently, EFL (Wang et al., 2021) proposed a task transformation paradigm, improving single sentence tasks with less labels using rich sentence-pair datasets.
Adapter Tuning: Adapter tuning has emerged as a novel parameter-efficient transfer learning paradigm (Houlsby et al., 2019; Pfeiffer et al., 2020), in which adapter layers – small bottleneck layers – are inserted and trained between frozen pre-trained transformer layers. On the GLUE benchmark, adapters attain within  $0.4\%$  of the performance of full fine-tuning by only training  $3.6\%$  parameters per task. Compactor (Mahabadi
et al., 2021) substitutes the down- projector and upprojector matrices by a sum of Kronecker products, reducing the parameters by a large margin while maintaining the overall performance.
Prompting: Hand-crafted prompts were shown to be helpful to adapt generation in GPT-3 (Brown et al., 2020). Existing works including LM-BFF (Gao et al., 2021a; Wang et al., 2021) explored the prompt searching in a few-shot setting.
Recently, several researchers have proposed continuous prompts training to overcome the challenges in discrete prompt searching. Prefix tuning (Li and Liang, 2021) and P-tuningv2 (Liu et al., 2021a) prepend a sequence of trainable embeddings at each transformer layer and optimizes them. Two contemporaneous works - prompt tuning (Lester et al., 2021) and P-tuning (Liu et al., 2021b), interleave the training parameters in the input embedding layer instead of each transformer layer. All these methods focus on task-specific prompt optimization. Our proposed method, IDPG, is the first prompt generator that is not only task-specific but also instance-specific.
We have introduced IDPG, an instance-dependent prompt generation model that generalizes better than the existing prompt tuning methods. Our method first factors in an instance-dependent prompt, which is robust to data variance. Parameterized Hypercomplex Multiplication (PHM) is applied to shrink the training parameters in our prompt generator, which helps us build an extreme lightweight generation model. Despite adding fewer parameters than prompt tuning, IDPG shows consistent improvement. It is also on par with the lightweight adapter tuning methods such as Compacter while using a similar amount of trainable parameters. This work provided a new research angle for prompt-tuning of a pre-trained language model.
We thank Brian Lester, Jonas Pfeiffer, Tianyu Gao, Qinyuan Ye, Zhenhao Zhang, Fei Sun, members of the University of Michigan's NLP4Health research group, and all anonymous reviewers for helpful discussion and valuable feedback.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021a. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816-3830, Online. Association for Computational Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021b. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and
Luo Si. 2021. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2208-2222, Online. Association for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119-9130, Online. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online. Association for Computational Linguistics.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021a. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021b. Gpt understands, too. arXiv preprint arXiv:2103.10385.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. arXiv preprint arXiv:2106.04647.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling*. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics* (Demonstrations), pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 271-278, Barcelona, Spain.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 115-124, Ann Arbor, Michigan. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rückle, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503, Online. Association for Computational Linguistics.
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Linguual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational Linguistics.
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Computational Linguistics.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021. Entailment as few-shot learner. arXiv preprint arXiv:2104.14690.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. *Language resources and evaluation*, 39(2):165-210.
Aston Zhang, Yi Tay, Shuai Zhang, Alvin Chan, Anh Tuan Luu, Siu Cheung Hui, and Jie Fu. 2021. Beyond fully-connected layers with quaternions: Parameterization of hypercomplex multiplications with  $1/n$  parameters. arXiv preprint arXiv:2102.08597.
A.1.1 Training hyperparameters
We use RoBERTa-Large (Liu et al., 2019) model implemented by Fairseq (Ott et al., 2019) as our basic model. The detailed model hyperparameters are listed in Table 4.
Note that for both transformer fine-tuning methods, including RoBERTa (Liu et al., 2019) and EFL (Wang et al., 2021), we follow their official training instructions, i.e., using a polynomial learning rate scheduler with  $6\%$  steps to warm up and tuning for ten epochs. For all adapter-based and prompt-based methods, we train them more sufficiently (with fifty epochs) on small datasets (i.e., MPQA, Subj, CR, MR, RTE, MRPC, STS-B).
A.1.2 Model hyperparameters
We report the detailed model hyperparameters for each method in Table 1 and illustrate how numbers in Table 2 are computed.
Compacter: hidden size  $d = 1024$ , adapter hidden size  $m = 16$ , user defined  $n = 4$ , each transformer layer inserts 2 compacters. Down-project  $s_i$  matrix takes  $1024 / 4 \times 4 \times 24 \times 2 = 48K$ , down-project  $t_i$  matrix takes  $16 / 4 \times 4 \times 24 \times 2 = 0.75K$ , hidden bias takes  $16 \times 24 \times 2 = 0.75K$ , up-project  $s_i$  and  $t_i$  matrix takes the same number of parameters as down-projector, the output bias takes  $1024 \times 24 \times 2 = 48K$ , the shared matrix  $A_i$  takes  $4^3 \times 24 \times 2 = 3K$ . Total parameters:  $48 + 0.75 + 0.75 + 48 + 0.75 + 48 + 3 = 149.25K$ .
Adapter: hidden size  $d = 1024$ , adapter hidden size  $m = 16$ . Total parameters:  $(1024 \times 16 + 16 + 16 \times 1024 + 1024) \times 24 \times 2 = 1.55M$ .
Prompt-tuning: prompt length  $t = 5$ . Total parameters:  $5 \times 1024 = 5K$
Prompt-tuning-134: prompt length  $t = 134$ . Total parameters:  $134 \times 1024 = 134K$ .
P-tuning v2: prompt length  $t = 5$ , inserted layers 24. Total parameters:  $5 \times 24 \times 1024 = 120K$ .
S-IDPG-PHM: hidden size  $d = 1024$ , generator hidden size  $m = 256$ , prompt length  $t = 5$ , user defined  $n = 16$  (Cf. Equation 4). First PHM layer  $W_{1}$  takes  $1024 / 16 \times 256 / 16 \times 16 + 256 = 16.25K$  parameters, second PHM layer  $W_{2}$  takes  $256 / 16 \times 5 \times 1024 / 16 \times 16 + 5 \times 1024 = 85K$  parameters, the shared matrix  $A_{i}$  takes  $16^{3} = 4K$  (Note we use one shared matrix in single version IDPG). Total parameters:  $105K$ .
S-IDPG-DNN: hidden size  $d = 1024$ , generator hidden size  $m = 256$ , prompt length  $t = 5$ . Total parameters:  $1024 \times 256 + 256 + 256 \times 5 \times 1024 + 5 \times 1024 = 1.5M$ .
M-IDPG-PhM-GloVe: input vector size 300, generator hidden size  $m = 16$ , prompt length  $t = 5$ , user defined  $n = 4$  (Cf. Equation 4). First PHM layer  $W_{1}$  takes  $300 / 4 \times 16 / 4 \times 4 + 16 = 1216$  parameters, second PHM layer  $W_{2}$  takes  $16 / 4 \times 5 \times 1024 / 4 \times 4 + 5 \times 1024 \times 24 = 140K$  parameters, the shared matrix  $A_{i}$  takes  $4^{3} \times 2 = 128$ . Total parameters:  $141K$ .
M-IDPG-PhM: hidden size  $d = 1024$ , generator hidden size  $m = 16$ , prompt length  $t = 5$ , user defined  $n = 16$  (Cf. Equation 4). First PHM layer  $W_{1}$  takes  $1024 / 16 \times 16 / 16 \times 16 + 16 = 1K$  parameters, second PHM layer  $W_{2}$  takes  $16 / 16 \times 5 \times 1024 / 16 \times 16 + 5 \times 1024 \times 24 = 125K$  parameters, the shared matrix  $A_{i}$  takes  $16^{3}16 \times 2 = 8K$ . Total parameters:  $134K$ .
M-IDPG-DNN: hidden size  $d = 1024$ , generator hidden size  $m = 16$ , prompt length  $t = 5$ . Total parameters:  $1024 \times 16 + 16 + 16 \times 5 \times 1024 + 5 \times 1024 \times 24 = 216K$ .
We provide a detailed information in Table 5 for 10 NLU datasets we used.
A.3 Detailed results for Multi-layer Architecture Exploration