SMoP: Towards Efficient and Effective Prompt Tuning with Sparse Mixture-of-Prompts
Authors: Joon-Young Choi
Junho Kim
Jun-Hyung Park
Wing-Lam Mok
SangKeun Lee
Venue: EMNLP 2023 Main (2023)
Keywords: Natural Language Processing
Prompt Tuning
Parameter-Efficient Fine-tuning
Mixture-of-Experts

Abstract: Prompt tuning has emerged as a successful parameter-efficient alternative to the full fine-tuning of language models. However, prior works on prompt tuning often utilize long soft prompts of up to 100 tokens to improve performance, overlooking the inefficiency associated with extended inputs. In this paper, we propose a novel prompt tuning method $SMoP$ ($S$parse $M$ixture-$o$f-$P$rompts) that utilizes short soft prompts for efficient training and inference while maintaining performance gains typically induced from longer soft prompts. To achieve this, $SMoP$ employs a gating mechanism to train multiple short soft prompts specialized in handling different subsets of the data, providing an alternative to relying on a single long soft prompt to cover the entire data. Experimental results demonstrate that $SMoP$ outperforms baseline methods while reducing training and inference costs. We release our code at https://github.com/jyjohnchoi/SMoP.

SMoP: Towards Efficient and Effective Prompt Tuning with Sparse Nixture-of-Prompts
Joon- Yoon-Yuungaa Choi', Junho K Jun-Hyung Park2, Wing-Lam Mok', SangKeun Lee
'Department of Artificial Intelligence, Korea University, Scoul, Republic of Korea BBK1 FOUR R&E Center for Artificial Intelligence, Korea University, Scoul, Republic of Korea 3 Department of Computer Science and Engincering, Korea University, Scoul, Republic of Korea {johnjames, monocrat, irish07, wlmokac, yalphy)@korea.cm kr
Abstract
Prompt tuning has emerged as a successful parameter-effiiien alternative to the full fine- tuning of language models. However, prior works on prompt tuning often utilize long soft prompts of up 100 tokens improve per- formance, overlooking the inefficiency associ- ated with extended inputs. In this paper, we propose novel prompt tuning method SMoP (Sparse Mixture-of-Prompts) Prompts) that utilizes short soft prompts for efficient training and inference while maintaining performance gains typically induced from longer soft prompts. To achieve this, SMoP employs a gating mechanism to train multiple short soft prompts specialized in handling different subsets of the data, provid- ing alternative to relying soft prompt to cover the entire data. Experi- mental results demonstrate that SMoP oulper- forms baseline methods while reducing train- ing and inference costs. We release our code at https: htg//gituuuu thub. com/ hommm/jooonnoo//S00
Theluris
Taai Prompt Langth
Tull Foopttargg
Figure 1: Accuracy (left) and training memory usage (right) with varying total prompt length on RTE dataset. For prompt tuning (Lester et al.. 2021). increasing soft prompt length improves accuracy, but also results in a significant increase in memory usage. SMoP outper- forms prompt tuning while preserving memory usage by sparsely activating short (length 5) prompts.
single long
layer of the model (Li and Liang, 2021; Qin and Eisner, 2021; Liu et al., 2022), incorporating input- specific soft prompts (Jiang et al., 2022; Wu et al., 2022), or pruning and rewinding soft prompts (Ma et al., 2022). On the other hand, methods such FPT (Huang et al., 2022) demonstrate improved training efficiency of prompt tuning in terms of convergence speed via progressive training.
1 Introduction
Prompt tuning (Lester et al., 2021; Liu et al., 2021) has recently gained attention as a parameter- efficient alternative to the full fine-tuning of language models. By freezing the original lan- guage model parameters and solely tuning the soft prompts (i.e., learnable token embeddings) added the model input, prompt tuning achieves compa- rable performance to full fine-tuning while largely reducing the number of trainable parameters. More- over, prompt tuning stands out for its conceptual simplicity and flexibility among other parameter- efficient fine-tuning methods (Houlsby et al., 2019; Guo et al., 2021; Hu et al., 2022), as it does not require modifications to the model structure.
Although these methods have empirically shown improvements in prompt tuning, they have over- looked the inefficiency associated with the exten- sion of input sequences caused by the inclusion of soft prompts. While increasing soft prompt length (typically up to 100 tokens) is known to bene- fit model performance (Lester et al., 2021; Jiang et al., 2022), it consequently yields longer input sequences, leading to increased computational quirements during training and inference (see Fig- ure 1). Therefore, we aim to investigate the uti- lization of relatively short soft prompts while pre- serving performance gains typically achieved from longer soft prompts.
Since the proposal of prompt tuning, there has been active research enhance its efficiency and effectiveness. On one hand, several approaches pro- pose to improve the performance of prompt tuning by integrating soft prompts into activations in each
To this end, we propose SMoP (Sparse Mixture- of of-Prrmpts), a novel prompt tuning method that utilizes short soft prompts during training and in-

---

Prompts
Input Embeddings
Oulput
Oulput
Pre-trained Language Model
Pre-trained Language Model
Gating
Mechanism
Router
Model
Input Instance
Input Instance
(a) Prompt Tuning
(b) SMoP (Ours)
Figure 2: (a) Illustration of prompt tuning (Lester et al., 2021). A soft prompt concatenated with the embedding representations of an input instance, and the soft prompt is solely fine-tuned. Given soft prompt of 100 tokens, the length soft prompt typically longer or similar the input instance. (b) Illustration our proposed method SMoP. A gating mechanism is employed to route each input instance to short soft prompt.
ference. Given that using a single short soft prompt leads to inferior performance compared to longer soft prompts, our key insight is to train multiple short soft prompts that are specialized in handling different subsets of the data. To achieve this, we draw inspiration from the Sparsely- Gated Mixture- of of-Experts (Shazeer et al., 2017; Fedus al., 2022) that sparsely activates sub-networks (i.e.. experts) increase model capacity without a proportional increase in computation. We integrate this concept in the context of prompt tuning by employing a gating mechanism in SMoP, which guides each in- put instance to one of the short soft prompts based on its embedding representation. Such sparse ac- tivation enables effective utilization of short soft prompts without significant increase in computa- tion degradation in performance.
SMoP sparsely activates short soft prompts via a gating mechanism that routes cach in- stance to one of the multiple soft prompts based on its embedding representation.
Experimental results demonstrate that SMoP outperforms the baselines on T5-base and TS-large while utilizing shorter soft prompts. thereby using less training and inference costs.
Method
2.1 Preliminaries
Full Fine-tuning Assume that we have a sequence-to-sequees model ps(y x) param- eterized by á. Given an instance with a length sequence of embedding representations X {I1, T2. 2n} E Rexe and corresponding label token embedding sequence Y, the objective func- tion for full fine-tuning the model pa is follows:
To verify the efficiency and effectiveness SMoP introduces to prompt tuning, we conduct evalua- tions on six natural language understanding tasks from the SuperGLUE benchmark. Experimental re- sults demonstrate that SMoP outperforms prompt tuning with reduced training and inference costs. In particular, SMoP improves the average perfor- mance of prompt tuning on six SuperGLUE tasks by 2.5%p with T5-base and 3.4%p with T5-large on average while reducing training time, memory, and inference computations.
arg max log Po (Y) X).
(1)
Prompt Tuning If we define a length soft prompt with embedding dimension e as Pe which is parameterized by 0 E [ the objective function of prompt tuning is as follows:
Our contributions are as follows:
1. We propose novel prompt tuning method SMoP (Sparse Mixture-of-Prompe that uti- lizes short soft prompts for efficient training and inference while maintaining performance gains often induced by increased soft prompt length.
arg max log (Y [Po: X]),
(2)
0
where indicates the concatenation of the two ma- trices. Note that the language model parameters @ are no longer updated. Figure (a) depicts the process of prompt tuning.

---

Total
Utilized
Training Costs (.)
Inference Costs (1)
Trainable Params (%)
Average
Model
Method
Prompt
Prompt Length
Time
Memory
FLOOP
Score (
Length
(s/100 steps)
(GBB
(GFLoPsssmmlee)
Full Fine-tuning
100
81.4
14.9
70.3
78.2
T5-
Prompt Tuning
100
100
0,0344
70.7
(13.119)
14.4
(3.4%)
98.1
(+39.55)
73.3
base
P-tuning
20
20
0.1028
64.9
(-20.3%)
12.3
(-17.4%)
76.6
(1909)
75.22.1
SMoP (Ours)
20
5
0,0083
61.0
(-25.1%)
11.9
(-2011%)
71.5
(+1,7%)
75.8,
Full Fine-tuning
100
176.1
29.2
247.8
83,41,3
TS- large
Prompt Tuning
100
100
0.0139
151.2
(-14,15)
29.3
(+0.3%)
378.1
(+52.6911
78.61.4
P-tuning
20
20
0.0407
131,9
(-25.195)
23.6
(-19.256)
291.6
(-177,991
81.31.8
SMoP (Ours)
20
5
0,0033
129.1
(-26.7%)
22.6
(-22.6%)
275.4
(+11.1%)
82.0,.
Table Experimental results on six SuperGLUE tasks. Average training costs, inference costs, and performance for baselines and SMoP are presented. The percentage value next to each cost value indicates relative changes in cost values compared to full fine-tuning, and the subscript of the average score indicates the corresponding standard deviation. The highest performance and lowest cost values among prompt tuning methods bold highlighted.
2.2 SMoP: Sparse ceaturre-ffPromptt
balance among experts during training plays an im- portant role in performance. To ensure load balance among soft prompts by encouraging exploration of inputs over diverse prompts, we apply router per- turbation during the training of SMoP. Specifically, add scaled Gaussian noise 5 ~ (O, 1) to the output value of the router model during training, Therefore, equation (3) is modified as follows:
The goal of SMoP is to train multiple short soft prompts, where each prompt is specialized in subset of the data. To achieve this, SMoP employs a gating mechanism to direct the input instance to one of the soft prompts based on its embedding representations, as shown in Figure (b).
In the gating mechanism, we introduce small linear router model I. parameterized by μ E T which makes decisions regarding which of the soft prompts the input should be routed to. Formally, given & soft prompt embeddings Po., Poz. .o,, Poz which are parameterized by { where 0)- E x the router model takes the average of input embeddings X E E as its input and calculates the routing probability Pt, P2. .... Pk for each soft
(X)) (softmaxLL, (I+ 8)1z. (5)
3 Experiments
3.1 Experimental Settings
Tasks To cover diverse NLP tasks in our experi- ments, we evaluate SMoP and baseline methods six tasks' from the SuperGLUE benchmark (Wang et al., 2019). As the official test sets for Super- GLUE benchmark are not publicly released, we follow Chen et al. (2022a) to use the validation set as the test set and split the original train set into train and validation sets by 90%/10%% proportion.
prompt. Thus, the routing probability of the j-th prompt calculated as:
Px( = [softmax(L,
(3)
The input is then routed to the soft prompt with the highest probability, and the final soft prompt to be utilized is obtained as the product of the routed prompt and the probability value. Therefore, the objective function of SMoP is defined as follows:
Models and Baselines Our experiments are built on the public HuggingFace (Wolf al., 2019) im- plementation and pre-trained checkpoints of T5 (Raffel et al., 2020) two scales: base and large. To demonstrate the advantages that SMoP in- troduces to prompt tuning, we compare SMoP to prompt tuning (Lester et al., 2021). P-tuning (Liu et al., 2021), and full fine-tuning.
arg max log pY [Pe(X)- Po.: XI), 1.0g
(4)
where is the index of the prompt with the highest probability value. Note that in SMoP, while the total prompt length is k- I, the utilized prompt length remains as I.
Evaluation Setup For prompt tuning methods, we experiment on (5, 20, 50, 100) soft prompt tokens, and for SMoP, we sweep through (2, 4, 10, 20) prompts of length 3, 10). We report experimental results on the setting with the best average performance over two or three runs, as
2.3 Router Perturbation
Prior works on Mixture-of off-exsttsss (Chen et al., 2022b; Fedus et al., 2022) demonstrate that load
BoolQ, CB, COPA, MultiRC, RTE, WIC

---

k
Model
Method
BoolQ
CB
Average
Model
2
10
20
SMoP (Ours)
79.4n.
94.6.x
77.5a.2
83.82.
1
72.92.1
74.0
73.51a
73.91.2
perturbation
79.7pa
93.527
76.0h.s
83.1
3
7..22.3
74.0
74.82.4
74.2
T5- hase
Top-2
78.4p
88.11.
69 69.70.4
78.70s
TS-base
5
75.02.
75.8
75.31.8
74.71.7
Gumbel-Softman Softmax
79.2
92.320
75.24.3
82.2g7
Stochastic
78.2n.3
86.92.1
09.21.7
78. 78.
10
1.
74.8
75.21,4
74.12a
Single
78.5um
89.3
69.9p.s
79.21.1
Table 2: Average performance (%) on six tasks from the SuperGI.UTE benchmark with diverse utilized prompt lengths (1) and the number of prompts (k).
Table 3: Experimental results (%) on diverse routing methods for SMoP.
mance. We conjecture that this may be due the limited labeled data for training in several Super-
well as the corresponding standard deviations, We report training time? and memory usage as training costs and inference FLOPs as inference costs.
GLUE tasks, leading to insufficient training of each soft prompt (Wang et al., 2022).
3.2 Results
Given these findings, we primarily report the re- sults of SMoP utilizing 4 soft prompts, each with length of 5 tokens. Note that while SMoP generally demonstrates improvements in prompt tuning, the optimal length and number of soft prompts may vary by specific tasks or datasets.
3.2.1 Main Results
Table presents the performance of SMoP and the baseline methods. Notably, SMoP achieves the highest performance among the baseline prompt tuning methods SuperGL.UE tasks with the least training and inference costs. On T5-base, SMoP demonstrates an average improvement of 2.5%pp while on TS-large, the improvement reaches 3.4%p. The detailed results of SuperGLUE tasks are shown in Appendix D.
3.2.3 Routing Methods
To verify the impact of the routing method the gating mechanism of SMoP, we perform experi- ments on diverse routing methods, including linear router without router perturbation (w/o perturba- tion), taking the weighted sum of two prompts with the highest probability (Top-2). dumbel-Softmnn routing where the output probability of the router is calculated as (Gumbel--oftmax), stochastic rout- ing (Stochastic) which an application of AdaMix to prompt tuning (Zuo et al., 2022; Wang et al., 2022), and no routing (Single) which identical to prompt tuming with length 5 soft prompt.
The fact that SMoP outperforms the baselines with less training and inference costs highlights the significance of utilizing short soft prompts during training and inference. For example. SMoP saves 14.6% training time, 22.9% training memory, and 27.2% inference FLOPs T5-large, compared to prompt tuning with a soft prompt of length 100. It is worth noting that full fine-tuning requires the fewest of FLOPs for inference as no additional to- kens are added to the input, while SMoP introduces the least additional FLOPs.
Table 3 shows experimental results three Su- perGLUE tasks with diverse routing methods. The top-1 linear router with router perturbation, which is our original setting, generally outperforms all other routing strategies. One exception is BoolQ where removing the router perturbation exhibits a slightly better performance. We speculate that in high-resource settings like BoolQ, router perturba- tion may not be mandatory for sufficient training of each soft prompt.
3.2.2 Length and Number of Soft Prompts
To investigate the optimal length and number of soft prompts to employ, we present the experimen- tal results on SMoP with diverse utilized prompt lengths and numbers of prompts in Table 2.
It is observed that increasing the total prompt length over 50 provides marginal performance gains. This finding is aligned with previous re- search (Lester et al., 2021; Li and Liang, 2021; Ma al., 2022) that report increasing soft prompt length above certain threshold brings limited im- provements to performance.
Related Work
4.1 Prompt Tuning
Pre-trained language models (PLMs) have demon- strated remarkable performance on a wide range of tasks in Natural Language Processing (NLP) (Devlin et al., 2019; Liu et al., 2019). However, with the introduction of larger language models
Furthermore, we notice that using 20 soft prompts generally lead to a degradation in perfor-
"Measured with single NVIDIA RTX A6000 GPU.

---

such as T5 (Raffel et al., 2020) and GPT-3 (Brown al., 2020). fine-tuning the entire parameters of the PLM for each specific task has become notably inefficient in terms of training and deployment.
that integrates the concept of Mixture-of-Expertss to Adapter (Houlsby et al., 2019). It follows THOR (Zuo et al., 2022) to employ stochastic routing and merging of multiple adapter modules. Both SMoP and AdaMix have taken inspiration from the con- cept of the coxture--of-xperee structure to improve parameter-effiient fine-tuning, However, their pri- mary motivations are distinct in that the motivation of SMoP is to multiple short soft prompts for efficient prompt tuning, while the motivation of AdaMix to provide multiple views of the given task for better performance. Therefore, SMoP em- ploys a linear router for instance- wiise prompt se- lection resulting in multiple soft prompts each spe- cialized in subset of the task, whereas AdaMix employs stochastic routing and merging, resulting in single adapter module per task.
To address such inefficiency, researchers have proposed parameter-e effficiiettt fine-tuning methods (Houlsby al., 2019; Lester et al., 2021; Pfeiffer et al., 2021; Hu et al., 2022), which involves fine- tuning a relatively small portion of task-specific parameters of the PLM while keeping the other parameters frozen. Among these methods, prompt tuning (Lester et al., 2021) is a simple and effective approach that entails prepending learnable token embeddings (i.e., soft prompts) to the model in- put and solely fine-tume fine-tuning these embeddings. The simplicity and adaptability of prompt tuning have led to several advancements aimed at improving its efficiency and performance by modifying the struc- ture of soft prompts (Liu et al., 2021; Li and Liang, 2021), using instance-specific prompts (Jiang et al., 2022; Wu et al., 2022), adjusting the training process (Huang et al., 2022; Ma et al., 2022). More- over, prompt tuning is known for its capability for task knowledge transfer from source task prompts to target task prompts (Vu et al., 2022; Asai et al., 2022; Wang et al., 2023). These methods have im- proved the overall performance of prompt tuning, but they have overlooked the inefficiency of uti- lizing lengthy soft prompts. SMoP is designed to alleviate this efficiency concern and is orthogonal most of the existing prompt tuning methods.
Conclusion
We have presented SMoP (Sparse Mixture-of- Prompts), novel prompt tuning method that uti- lizes short soft prompts for efficient training and inference while maintaining performance gains as- sociated with increased prompt length. To achieve this, we have employed a gating mechanism in SMoP that routes each instance to one of the mul- tiple short soft prompts. Experimental results have demonstrated that SMoP has outperformed prompt tuning while reducing training and inference costs through the utilization of short soft prompts.
Limitations
4.2 Mixture-of-Experts
Given the same total prompt length, the gating mechanism of SMoP introduces additional param- eters compared to prompt tuning, inducing addi- tional storage requirements. Comparing prompt tuning with soft prompt of length 20 (20,480 trainable parameters) and SMoP with 4 prompts of length (24,576 trainable parameters) TS-base, SMoP adds 20% trainable parameters and such difference increases as more prompts are utilized.
Mfxture-of-Expere is model structure in which the output of the model is computed by multi- ple sub-networks (i.e., experts) conditionally acti- vated by a gating mechanism (Shazeer et al., 2017). This enables increasing the number of model pa- rameters without incurring a proportional increase in computation. Typically, the gating mechanism determines which experts process specific tokens (Shazeer et al., 2017; Fedus et al., 2022), while it can be extended route sequences or batches (Wang al., 2022: Zuo et al., 2022; Pan et al., 2023). In particular, Fedus et al. (2022) presents Switch Transformer that employs the Sparsely- Gated Mixture-of-Exxpetss layer (Shazeer et al., 2017), and Zuo et al. (2022) proposes THOR which utilizes stochastic (i.e., random) routing.
We further note that SMoP is orthogonal to most of the existing prompt tuning methods including prompt transfer learning methods (Vu et al., 2022; Asai et al., 2022; Wang et 2023) as mentioned in Section 4. While our investigation has highlighted the significance of incorporating short soft prompts through sparse activation in conventional single- task prompt tuning, we believe that SMoP holds promise as a valuable direction for augmenting the efficiency of prompt tuning methods in the future.
Recently, Wang et al. (2022) has proposed AdaMix, efrfmeter-efficient fine-tuning method

---

Acknowledgements
William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39.
We thank the anonymous reviewers for their helpful comments. This work was supported by the Basic Research Program through the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2021 12021R1A2CC A2C30104330) and Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019- 0-00079, Artificial Intelligence Graduate School Program (Korea University)).
Demi Guo, Alexander Rush, and Yoon Kim. 2021. Parameter-efffcient transfer learning with diff prun- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), ACI. IJCLNP 2021, pages 4884 4896. Association for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameetr-eeffcient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of Proceedingy of Machine Learning Research, pages 2790-2799. PMLR.
References
Akari Asai, Mohammadreza Salchi, Matthew Pe- ters, and Hannaneh Hajishirzi. 2022. ATTEMPT: Parameter-efficte multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Conference on Empirical Methods Natural Lan- guage Processing, EMNLP 2022, pages 6655-6677 Association for Computational Linguistics.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations. ICLR 2022. OpenReview.net.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models few-shot learners. In Advances in Neural Information Processing Systems, NeurIPS 2020, vol- ume 33, pages 1877-1901. Curran Associates, Inc.
Yufei Huang, Yujia Qin, Huadong Wang. Yichun Yin. Maosong Sun, Zhiyuan Liu, and Qun Liu. 2022. FPT: Improving prompt tuning efficiency via progressive training. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 6877-6887, Association for Computational Linguisties.
Yuezihan Jiang, Hao Yang, Junyang Lin, Hanyu Zhao. An Yang, Chang Zhou, Hongxia Yang, Zhi Yang. and Bin Cui. 2022. Instance-wise prompt tuning for pretrained language models. CoRR, abs/2206.01955
Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. 2022a. Revisiting parameter- efficient tuning: Are we really there yet? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, pages 2612-2626. Association for Computational Linguis- tics.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021.
The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2021, pages 3045-3059. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers). ACL-IJCLNP 2021, pages 4582-4597, Association for Computa- tional Linguistics.
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. 2022b. Towards understanding the cexture-of-experte layer in deep learning- In Ad- vances in Neural Information Processing Systems, NeurIPS 2022, volume 35, pages 23049 23062. Cur- ran Associates, Inc.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLE 2019, pages sociation for Computational Linguistics.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2022, pages 61 Association for Computational Linguis- ties.
As-

---

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. CoRR, abs/2 103, 10385.
adaptation through soft prompt transfer. Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), ACL 2022, pages 5039-5059, Association for Computational Linguistics.
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining ap- proach. CoRR, abs/1907.11692
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for seneral-purposs. language understand- ing systems. In Advances in Neural Information Pro- cessing Systems, NeurlPS 2019, volume 32, pages 3261-3275. Curran Associates, Inc.
Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qi- fan Wang, Wei Wu, Xiaojun Quan, and Dawei Song. 2022. XPrompt: Exploring the extreme of prompt tuning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Longuage Processing, EMNLP 2022, pages 11033-11047. Association for Computational Linguistics.
Yaqing Wang. Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadal- lah, and Jianfeng Gao. 2022. AdaMix: Mixture-of- adaptations for parameter-efficiee model tuning. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, EMNLP 2022, pages 5744-5760. Association for Computa- tional Linguisties.
Xiaoman Pan, Wenlin Yao, Hongming Zhang, Dian Yu, Dong Yu, and Jianshu Chen. 2023. Knowledge- in-context: Towards knowledgeable semi- cari-parametric language models. In The Eleventh International Con- ference on Learning Representations, ICLR 2023. OpenReview.net.
Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogério Feris, Huan Sun, and Yoon Kim. 2023, Multitask prompt tuning enables parameter-efficient transfer learning. In The Eleventh International Con- ference on Learning Representations, ICLR 2023. OpenReview.net.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Tryna Gurevych. 2021. AdapterFusion: Non-destruetive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, pages 487-503. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Authony Moi. Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, and Jamie Brew. 2019. Huggingface' 's transformers: State-of-the-ee natural language processing. CoRR. abs/1910.03771.
Guanghui Qin and Jason Fisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Association
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yux- iao Dong, V. G. Vinod Vydiswaran, and Hao Ma. 2022. IDPG: an instance-dependee tion method, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologiex, NAACL-HILT 2022, pages 5507-5521. As- sociation for Computational Linguistics.
NAACL-HLT 2021, pages for Computational Linguistics.
prompt genera-
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the lim- its of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-61 17.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim. Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. 2022. Taming sparsely activated transformer with stochastic experts. In The Tenth International Conference Learning Representations, ICLR 2022. OpenReview.net.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le. Geoffrey F. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In The Fifth International Conference on Learning Repre- sentations, ICIR 2017. OpenReview.net.
Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, ICML. 2018, volume 80 of Proceedings of Machine Learning Research, pages 4596-4604. PMLR.
Tu Vu, Brian Lester, Noah Constant, Rami AI-Rfou', and Daniel Cer. 2022. SPoT: Better frozen model

---

Appendix
number of layers in the backbone model, as they add an adapter module the internal structure of the original model. contrast, prompt tuning methods including SMoP introduce additional pa- rameters exclusively to the inputs of the model, enabling a parameter-efficient module where the number of trainable parameters doesn't t increase proportionally to model size (Asai et al., 2022).
A Comparison to Adapter-based Methods
To further explore the advantages of SMoP in the realm of parameter-effiiien fine-tuning methods, we compare SMoP and prompt tuning methods to adapter-based parameter-efficien fine-tuning methods, namely Adapter (Houlsby et al., 2019), AdapterFusion (Pfeiffer et al., 2021), and LoRA (Hu et al., 2022). We provide a brief description of each method and present the experimental results on six SuperGLUE tasks with the T5-base model. Adapter-based methods add additional modules to the internal structure of the model. Adapter (Houlsby et al., 2019) adds bottleneck modules af- ter the multi-head attention and feed-forward layer of each Transformer layer, while AdapterFusion (Pfeiffer et al., 2021) adds bottleneck modules only layer. LoRA (Hu al., 2022)
Trainable Params %
Model
Method
SMoP (I=5, k=4)
0.0083 (1.0x)
LoRA (r=l)
0.0496 (6.0x)
T5-base
LoRA (r=2)
0.0991 (12.0x)
LoRA (r=4)
0.1981 (24.0x)
LoRA (r=8) 0.3954 (47.8x)
Table 4: Comparison of trainable parameter ratio be- tween SMoP and LoRA. The value in the parenthesis trainable params denotes the relative difference, with SMoP the reference point.
after the adds a low-rank decomposition of cach of the at- tention matrices, which are directly added during inference. We implement these methods upon the adapter-transsformers library.
B Text-to-Text Templates
We provide the text-to-text templates and verbaliz- ers used in our experiments in Table 6.
Table presents the experimental results of full fine-tuning, adapter-based methods, and prompt tuning methods on six SuperGLUE tasks with the TS-base model. While adapter-based methods gen- erally outperform prompt tuming methods under their best configuration, SMoP is able to reach comparable performance while only utilizing up to 190x smaller number of trainable parameters. In particular, when the ratio of trainable parame- ters narrows to factor of B3x,SMor outperforms Adapter on tasks out of Similar results are observed for AdapterFusion, where SMoP shows inferior performance when the bottleneck dimen- sion is set to 48, but reverses the results when d is reduced to 8.
C Hyperparameters
We train our model for (50, 100) epochs on CB, COPA, RTE, WiC and for (10, 20) epochs on BoolQ and MultiRC with batch size 32, learning rate of (1e-4, 5e-5, 1e-5) for full fine-tuning and adapter-based methods, and learning rate (0.5, 0.3, 0.1 for prompt tuning methods including SMoP. We perform early stopping based on validation per- formance, and terminate training if there is no im- provement for 10 epochs. We train the model with Adafactor optimizer (Shazeer and Stern, 2018), where the weight decay is le-5, and linear lcarning rate decay of warmup ratio 0.06 is applied.
Considering LoRA. SMoP shows slightly bet- ter performance compared to both configurations. One notable result is that using lower rank in LoRA does not yield significant decrease per- formance. However, as shown in Table 4, the level parameter efficiency of SMoP is not attainable with LoRA, as LoRA (r=1) still requires 6x more trainable parameters compared to SMoP. These observations highlight the parameter efficiency of SMoP compared to adapter-base based approaches. In general, adapter-based lightweight methods require additional parameters proportional to the
D Detailed Experimental Results
We provide task-wise results of experiments pre- sented in the paper. Since we experiment with our own train/validation/tsst split, the results may vary with previous works such as Lester et al. (2021).
D.1 Performance
Table and 8 present the experimental results on six SuperGLUF. tasks on T5-base and T5-large.
D.2 Training Costs
Table presents the memory used during training (GB), and Table 10 presents the training time (s/100

---

Trainable Params
BoolQ
CB
COPA
MultiRC
WIC
Average
Model
Method
Score
%
&ccc
%Acc
96Acc
%Fl,
%Acc
AAcc
Full Fine-tuning
100
81.9o.1
96.41.
64.31.5
80.20.2
79.20.2
67.02.
78.21.
Adapter (d=48)
1,5800
81.10.1
94.61.8
62.7p.6
80.2n.2
76.61.6
66.21.6
76.91.2
Adapter (d=8)
0.2806
79.6
89.91.0
59.01.0
80.
75.21.6
63.3p.s
74, 74.8
AdapterFusion (d=48)
0.7963
79.20.4
94.61.8
63.72.6
80.2
79.20.9
66.8p.3
77.31
T3-
AdapterPusion (d=8)
0.1405
79.60.6
92.3a.7
58.30.8
79.90.s
78.02.9
64.90.5
75.52.0
LoRA (r=8)
0.3954
79.00.0
90.51.
60.0p.8
80.0b.o
77.92.9
66.90.a
75.71.3
LoRA (r=2)
0.0991
19.
91.1a.
59.31.2
80.2o.0
77.42.7
66.5p.3
75.61.2
Prompt Tuning (I=100)
0.0344
19.1o.1
86.9a.7
56.72.1
78.30.2
73.21.7
65.61.2
73.31.9
P-Tuning (l=20)
0.1028
78.70.2
91.72.7
58.3g.s
79.30.2
77.31.8
65.9p.7
75.22.1
SMoP (I=5, k=4)
0.0083
79.4n.3
94.61.8
58.32.9
79.6n.1
77.53.2
65.2p.5
75.8
Table 5: Experimental results of diverse parameter-efficiet fine-tuning methods on six SuperGI.UF tasks with TS-hase model. The methods include full fine-tuning, adapter-based methods, prompt tuning methods, and our proposed SMoP. d for Adapter and AdapterFusion indicates the bottleneck dimension and r for LoRA indicates the rank of the matrices. The best performance among adapter-based methods and prompt tuning methods for each task are bold highlighted.
Dataset
Text-to-text Template
Verbalizer
BoolQ
boolg passage: passage" question: "*question"
False, Truc
hypothesis: *hypoihesiss*, premise: "*premise
entailment, contradiction, neutral
COPA
copa choicel: "#choice| choice2: "*choice2"* premise: *"rremise* question: "Qquestion""
choice choice2
MultiRC
multire question: ""question"> 99 *Panswer*,, paragraph: ""paragraph"
False, True
RTB
sentencel: premisse" sentence2: "hypothesiss*
entailment, not entailment
WiC
sentencel: P sentence2: >sgentence2* word: *tword**
False, True
Table 6: Text-to-text templates and verbalizers used in our experiments.
Total
Utilized Prompt Length
BoolQ
CB
COPA
MultiRC
RTE
WIC
Average Score (%
Model
Method
Prompt Length
%eAcc
Acc
Accc
$F1,
%Acc
%Acc
Full Fine-tuming
81.9p.1
96.41.
64.31.5
80.2p.2
79.20.2
67.02.3
78.21.3
78.5p.
89.31.8
54.03.6
79.1
69.9 69.9p.s
64.4p.0
72.5
20
20
78.6p.
86.92.1
55.0s.5
79.2p.2
70.61.
64.3p.2
72.4
Prompt Tuning
50
79.30.1
87.51.
56.04.
78.30.0
70.80.5
65.
72.81.8
100
100
79.1
86.93.7
56.72.1
78.3p.2
73.21.7
65.61.2
73.3
79.0p.1
89.947
59.0%.
79.2q.1
73.8
65.41.3
74.4
20
20
78.7
91.72.7
58.33.8
79.3p.2
77.31.
65.9p.7
75.22.1
P-tuning
50
50
78.8p
90.51.0
59.01.0
79.2g.
75.11.6
65.2pu
74.6n.9
100
100
79.0p.1
89.91.
59.02.
79.2g.0
73.83.5
65.4p.7
74.4
2
79.3p.3
90.70.7
52.74.
78.8p.3
71.52.5
64.71.1
72.92.1
79.0p.1
91.11.8
57.38.2
79.4g.1
72.40.8
65.0p.4
74.0
10
78.6p.0
92.90.
54.71.2
78.9..1
71.53.0
64.3p.5
73.5
T5-
20
78.6p.1
90.51.
57.72.1
79.30.2
72.61.5
64.9p.s
73.91.2
base
78.8p.1
92.91.
54.05.0
79.1
75.71.
64.71.
74.22.8
12
3
79.0pu2
92.91.
53.35.8
79.2
74.60.2
64.9
74.02.4
78.8p.1
94.04.5
56.03.6
79.2g.3
75.50.5
65.60.2
74.82.1
60
3
78.7am
91.71.
56.0x6
79.2a.1
74.71.6
64.7p.1
74.21.7
SMoP
10
78.5p.
92.9am
58.04.
79.4g.
76.4.3
64.9p.s
75.02.0
20
5
79.4n.3
94.61.
58.32.9
79.6n.1
77.58.2
65.2g.5
75.8
50
5
79.3p.1
92.31.0
58.74.2
79.3gm
77.103
65.2p.4
75.3
100
5
79.0p.3
93.420
55.33.1
79.3
76.92.0
64.30.2
74.71.7
20
10
78.701
93.510
59.33.5
79.2
76.01.
64.2p.1
75.1
10
78.6p.1
92.33.7
56.01.7
78.9p.1
76.96.0
66.4p.8
74.8
100
10
78.5n.1
95.810
57.72.
79.2
75.1
64.81.7
75.21.4
200
10
79.0p.
91.11.8
56.0s.5
79.4a.1
74.22.8
64.9p.7
74.12.0
Table 7: Experimental results on baseline methods and SMoP on six SuperGLUE tasks with T5-base, Subscripts of each score represent the corresponding standard deviation over multiple runs.

---

Total
Utilized Prompt Length
BoolQ
CB
COPA
MultiRC
RTE
WIC
Average
Model
Method
Prompt Length
Score (%)
%Acc
&Acc
%Acc
%F1,
&Acc
&Acc
Full Fine-tuning
85.8p
96.4p.a
76.02.
84.5p.1
87.60.1
70.3
83,41,
5
5
83.3p.
89.32.
57.5a.
83.8pa
86.3p.s
68.2p.
78.
20
20
83.1p.1
90.21.3
57.56.4
83.8p.
86.6p.
68.5p.
78.32.7
Prompt Tuning
50
50
83.10.1
91.10.
58.50.7
83.00.1
85.90.0
68. 68.00.4
78.2a.3
100
100
83.
90.51.0
62.0g.0
82.6p.2
87.01.
66.21.0
78.61.4
5
5
83.2p.2
92.0g.7
69.02.8
83.9p.
86.6p.
67.91.1
80.42.
TS-Jarge
20
20
83.40.4
91.72.7
71.73.2
84.20.1
87.61.0
69.20.9
81.31.8
P-Tuning
50
50
83.5p.1
92.01.3
71.04.2
83.70.0
87.01.
67.20.8
80.71.9
100
100
83.1p.1
93.8
67.01.4
82.2pm
87.4p.5
65.7pa
79.91.6
10
5
83.0p.1
97.31.
69.50.7
83.90.0
86.6p.0
66.0p.4
81.1a.6
20
5
83.5p.1
96.40.0
71.73.1
83.9p.2
87.7p.0
68.6p.7
82.01.3
SMoP
50
5
83.
94.72.5
69.04.2
83.9p.1
86.62.5
68.2p.2
80.92.3
100
5
83.6p.s
92.01.
67.56.4
83.9p.1
88.8p.
69.7pu6
80.91.7
Table 8: Experimental results on baseline methods and SMoP on six SuperGLUE tasks with T5-large. Subscripts of each score represent the standard deviation over multiple runs.
steps) for each SuperGLUE task. For BoolQ and MultiRC in T5- Ta-lerge we report the results for step batch size of 16 with gradient accumulation, as using batch size 32 exceeds the memory capacity of single NVIDIA RTX A6000 GPU.
D.3 Inference Costs
Table 11 presents the inference FLOPs (GFLOPs/sample) for each SuperGLUE task.

---

Total
Utilized Prompt Length
Model
Method
Prompt
BoolQ
CB
CОPA
MultiRC
RTE
WIC
Average
Length
Full
27.0
14.3
3.1
27.0
13.9
4.1
14.9
Prompt Tuning
100
100
21.8
16.0
5.0
21.8
15.6
6.2
14.4
T5-hase
P-Tuning
20
20
21.8
12.0
2.7
21.8
11.7
3.5
12.3
SMoP
5
5
21.8
11.3
2.3
21.8
11.0
3.1
11.9
Full
39.7
38.3
8.6
40.1
37.2
11.3
29.2
Prompt Tuning
100
100
30.5
42.9
13.7
30.6
41.8
16.6
29.3
T5-large
P-Tuning
20
20
30.1
32.3
7.5
30.5
31.3
9.8
23.6
SMoP
5
5
30.0
30.5
6.4
30.5
29.5
8.6
22.6
Table Peak memory (GB) during training on SuperGI.UE tasks.
Total
Utilized Prompt Length
Model
Method
Prompt
BoolQ
CB
COPA
MultiRC
RTE
WIC
Average
Length
Full
105.8
92.6
45.8
131.6
76.5
36.0
81.4
Prompt Tuning
100
100
93.1
90.3
37.2
103.7
71.4
28.4
70.7
T5-base
P-Tuning
20
20
84.8
85.9
30.5
108.2
59.0
21.1
64.9
SMoP
5
82.5
74.1
30.8
104.6
54.2
19.8
61.0
Full
228.4
183.1
82.8
338.9
152.0
71.3
176.1
Prompt Tuning
100
100
204.3
169.1
74.9
253.0
137.6
68.3
151.2
TS-large
P-Tuning
20
20
171.2
134.9
51.5
275.9
114.0
43.7
131.9
SMoP
164.7
134.0
47.4
281.0
107.4
39.9
129.1
Table 10: Training time (s/100 steps) on SuperGLUE tasks.
Total
Utilized Prompt Length
Model
Method
Prompt
BoolQ
CB
COPA
MultiRC
RTE
WiC
Average
Length
119.4
86.7
13.8
105.7
78.3
17.6
70.3
Prompt Tuning
100
100
136.9
120.2
40.1
130.7
114.3
46.7
98.1
bS-base
P-Tuning
20
20
124.3
93.4
19.0
114.1
85.5
23.4
76.6
SMoP
5
5
119.2
88.4
15.1
107.3
80.1
19.0
71.5
Full
402.5
334.9
48.4
365.1
274.5
61.4
247.8
Prompl Tuning
100
507.3
633.5
141.0
421.7
400.7
164.2
378.1
Ta-large
P-Tuning
20
20
417.6
499.2
66.8
384.5
299.6
81.8
291.6
SMoP
5
5
406.3
474.1
$3.0
371.7
280.8
66.5
275.4
Table 11: Inference FLOPs (GFLOPsssample) on SuperGLUUE tasks.


Reviews:
[{"review_id": "bpiJDRlZPq", "reviewer": "Reviewer_hgaH", "summary": "", "questions": "(1) Is there any reason why discrete prompts based approaches and adapter approaches are not used in the comparison?\n\n(2) Regarding the number of tuned parameters, can the proposed approach use less parameters compared to adapter based approaches (while maintain competitive accuracy performance)?\n\n(2) For examples, BERT, RoBERTa and GPT are used in the paper of the baseline P-tuning. However, T5 is used to report the baseline P-tuning’s results. If we use BERT, RoBERTa and GPT, can the proposed approach still outperform the baseline P-tuning?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging words and insightful suggestions. Below are our responses, and we are keen to address any further concerns. \n\n**Reasons to reject 1, 2.  and Question 1. Comparison to discrete-prompts-based approaches and adapter approaches.**\n\n- In the context of searching for suitable prompts for a given task, prompt tuning (i.e., continuous prompt-based method) can be interpreted as an extension of discrete prompt-based methods that relax the constraints of discrete prompts [1]. As we have shown our improvements over conventional prompt tuning, we have excluded discrete prompt-based methods from our experiments.\n- While the adapter-based methods do represent a domain of research in parameter-efficient fine-tuning, we mainly considered prompt-tuning methods as our baselines to align with our primary motivation to alleviate the inefficiencies associated with the length of soft prompts during prompt tuning.\n- To address your concerns, we have conducted additional experiments with three adapter-based baselines, namely Adapter [2], AdapterFusion [3], and LoRA [4], and the results are shown below. \n\n| Method | Trainable % | BoolQ | CB| COPA| MultiRC | RTE | WiC | Avg. |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Adapter (m=48) | 1.5800 | 81.0 | 95.5 | 60.3 | 80.3 | 75.8 | 67.0 | **76.6** |\n| AdapterFusion (m=48) | 0.7963 | 79.5 | 94.6 | 58.3 | 79.2 | 79.2 | 65.9 | 76.1 |\n| LoRA (rank=8) | 0.3954 | 79.0 | 90.5 | 59.7 | 80.0 | 77.9 | 66.9 | 75.7 |\n| Adapter (m=8) | 0.2806 | 79.0 | 90.2 | 58.0 | 80.0 | 73.6 | 65.1 | 74.3 |\n| **SMoP (l=5, k=4)** | **0.0083** | 79.4 | 94.6 | 58.3 | 79.6 | 77.5 | 65.2 | 75.8 |\n\n- In this table, \"m\" for Adapters represents the bottleneck dimension of the adapter layer, whereas \"l and \"k\" for SMoP represent the utilized soft prompt length and the number of soft prompts. The methods are listed in descending order of trainable parameters.\n\n**Question 2. Regarding the number of tuned parameters, can the proposed approach use less parameters compared to adapter-based approaches (while maintain competitive accuracy performance)?** \n- Yes, as shown in the experimental results from our previous response, SMoP uses 47 to 190 times smaller number of trainable parameters compared to adapter-based approaches, while maintaining competitive performance. Additionally, when the difference in trainable parameter ratio narrows to a factor of 33x, SMoP outperforms Adapter on 5 tasks out of 6. This highlights the parameter efficiency of SMoP compared to adapter-based approaches. \n\n**Reason to reject 3 and Question 3. Generalizability of SMoP to other backbone models.**\n\n- We have followed Lester et al. (2021) [5] to consider T5 as the main backbone model, and thus have conducted experiments with P-tuning on T5 for comparison.\n- As you have suggested, to validate the generalizability of SMoP to other backbone models, we have conducted additional experiments on RoBERTa-large on 3 datasets as an example. The results are shown below. \n\n| Model | Method | CB | RTE | WiC | Avg. |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n| RoBERTa -large | Full | 98.3 | 83.8 | 69.9 | 84.0 |\n|  | Prompt Tuning | 65.6 | 57.6 | 57.5 | 60.2 |\n|  | P-tuning | 69.6 | **60.3** | 61.0 | 63.6 |\n|  | SMoP | **72.4** | 59.7 | **66.1** | **66.1** |\n\n- In the experiments, we have swept over soft prompt length of {20, 100} for both Prompt Tuning and P-tuning, and used the setting of four prompts of length 5 for SMoP. \n - These results demonstrate that SMoP outperforms P-tuning with a different backbone model, namely RoBERTa-large. We note that there is still a significant gap between the performance of full fine-tuning and SMoP, which we believe is a direction for future work. \n- Due to additional implementations and extensive hyperparameter search required for different backbone models, we have conducted a small portion of experiments as an example. We will proceed with experiments on other tasks and backbone models and include the results in the camera-ready version paper. Thank you for your precious suggestion.\n\nReferences\n\n[1] Liu et al., \"Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing\", ACM Computing Surveys 2023\n\n[2] Houlsby et al., “Parameter-Efficient Transfer Learning for NLP”, ICML 2019\n\n[3] Pfeiffer et al., “AdapterFusion: Non-Destructive Task Composition for Transfer Learning”. EACL 2021\n\n[4] Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, ICLR 2022\n\n[5] Lester et al., \"The Power of Scale for Parameter-Efficient Prompt Tuning\", EMNLP 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The paper studies an emerging and important topic.\n\n(2) The developed solution is well motivated and has good intuitions.\n\n(3) The paper is well written and easy to follow. ", "reasons_to_reject": "(1) The baselines in the paper seem to focus on the soft prompts. The paper may also include baselines based on discrete prompts (which show competitive performance) in the comparisons.\n\n(2) The paper may also consider adapter-based baseline, such as LoRA, in the comparisons.\n\n(3) The paper may be further improved, by considering other model backbones. This can validated the proposed method is more generic. For examples, BERT, RoBERTa and GPT are used in the paper of the baseline P-tuning.", "questions_to_authors": "(1) Is there any reason why discrete prompts based approaches and adapter approaches are not used in the comparison?\n\n(2) Regarding the number of tuned parameters, can the proposed approach use less parameters compared to adapter based approaches (while maintain competitive accuracy performance)?\n\n(2) For examples, BERT, RoBERTa and GPT are used in the paper of the baseline P-tuning. However, T5 is used to report the baseline P-tuning’s results. If we use BERT, RoBERTa and GPT, can the proposed approach still outperform the baseline P-tuning?\n", "paper_topic_and_main_contributions": "Prompt tuning has emerged as a successful parameter-efficient alternative to the full fine-tuning of language models. However, prior works on prompt tuning often utilize long soft prompts of up to 100 tokens to improve performance, overlooking the inefficiency associated with extended inputs. This paper proposes a novel prompt tuning method SMoP (Sparse Mixture-of-Prompts) that utilizes short soft prompts for efficient training and inference while maintaining performance gains typically induced from longer soft prompts.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4lYNzAdExh", "reviewer": "Reviewer_qdfJ", "summary": "", "questions": "1. Could you please explain how you derive the trainable parameters in Table 1, especially for SMoP?\n2. Why do you use the soft prompt with the highest probability only instead of using a weighted average of all learned prompts? Have you considered the strategy used in AdaMix?\n3. In your ablation study, can you provide a comparison to prompt-tuning with the same final length of prompts?\n\nI appreciate if the author could respond to the concerns in the weakness section.\n\n----------\nThanks for the response from the author. With more detailed experimental evidence, I increased my score for the solidness. I also appreciate the discussion from the author for the difference between SMoP and ATTEMPT. However, I still find the SMoP as a simplified and single-task version of ATTEMPT, which limits the whole novelty of this work, so I kept my excitement score unchanged. I believe a discussion between SMoP and relevant multi-task framework such as SPoT, Attempt, and MPT will be neccessary for making this work distinct for future manuscripts.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and insightful suggestions. Below are our responses, and we are keen to address any further concerns. \n\n**Reason to reject 1. Important baselines and critical comparisons are missing, including but not limited to: SPoT, Attempt, and MPT.**\n- Works such as SPoT, ATTEMPT, and MPT are beyond the scope of our work. These works, encompassing multi-task prompt transfer learning, are centered on the transferability of task-specific knowledge from source tasks to a target task. In contrast, SMoP focuses on efficient single-task prompt tuning by alleviating the inefficiency associated with long soft prompts during the training. \n- In essence, multi-task transfer learning methods focus on verifying and utilizing well-initialized prompts via source tasks, while SMoP focuses on efficiently, and effectively initializing the prompts. Thus, we firmly believe that SMoP is rather orthogonal to multi-task prompt transfer learning and the application of SMoP to such work is a promising avenue for future research. \n- We would appreciate any further suggestions on additional baseline methods that will strengthen our paper.\n\n**Reason to reject 2. Limited novelty given the ATTEMPT framework.**\n- SMoP has several novel aspects compared to ATTEMPT. \n1) Motivation: ATTEMPT’s motivation is to leverage information from multiple source tasks via soft prompts [1], while SMoP’s motivation is to alleviate the inefficiency of soft prompt tuning.\n2) Methodology: The attention module of ATTEMPT assumes well-initialized soft prompts that are frozen during target prompt training. In contrast, SMoP's linear router does not necessitate initialized soft prompts, enabling simultaneous training of all soft prompts. This divergence leads to ATTEMPT's two-stage training process (source prompt training and target prompt training), while SMoP operates within a unified, single-stage approach.\n3) Parameter-Efficiency: During training and inference, ATTEMPT with T5-base involves 96k parameters plus the source task prompts per target task, while SMoP with T5-base in our best-performing configuration employs <20k parameters per target task, therefore is highly parameter-efficient.\n- Due to these differences, we believe that it could be misleading to interpret ATTEMPT as an improved version of SMoP.\n\n**Reason to reject 3. The proposed method requires further expensive hyper-parameter search, as shown in Table 2, and the optimal k, l setups are not easy to obtain, which makes the overall training more expensive and redundant than the original prompt tuning implementation.**\n\n- While SMoP does involve managing two parameters associated with soft prompts and requires more experiments to identify optimal configurations, the enhancements offered by SMoP are not significantly impacted by the choice of hyperparameters. For example, as illustrated in Table 2 of the paper, as long as the utilized soft prompt length is 5, SMoP consistently outperforms prompt tuning with a soft prompt of length 100 regardless of the number of soft prompts (74.7 ~ 75.8 vs 73.3). Hence, the increased number of parameters does not necessarily lead to expensive training costs.\n\n**Reason to reject 4. The experiments do not report multiple random seeds, and it appears to me that the improved performance in Table 1 can be due to cherry picking the seed and the exhaustive hyper-parameter search in Table 2.**\n- To clarify, we have reported the average result from the experimental results of three distinct random seeds. The results along with their corresponding standard deviations of our main experiments are presented below:\n\n|   Model  |     Method    | $l$ ($k$) |   BoolQ   |     CB    | COPA | MultiRC | RTE | WiC | Avg. |\n|:--------:|:-------------:|:-------------:|:---------:|:---------:|:----:|:-------:|:---:|:---:|:----:|\n|  T5-base |      Full     |       - (-)       | $81.9_{0.1}$ | $96.4_{1.8}$ |   $64.3_{1.5}$  | $80.2_{0.2}$ | $79.2_{0.2}$ | $67.0_{2.3}$ | $78.2_{1.3}$    |\n|  | Prompt Tuning |      100 (-)      | $79.1_{0.1}$| $86.9_{3.7}$ |$56.7_{2.1}$|$78.3_{0.2}$|$73.2_{1.7}$|$65.6_{1.2}$|$73.3_{1.9}$|\n|  |    P-Tuning   |       20 (-)      |$78.7_{0.2}$|$91.7_{2.7}$|$58.3_{3.8}$|$79.3_{0.2}$|$77.3_{1.8}$|$65.9_{0.7}$|$75.2_{2.1}$|\n|  |      SMoP     |     5 (4)    |$79.4_{0.3}$|$94.6_{1.8}$|$58.3_{2.9}$|$79.6_{0.1}$|$77.5_{3.2}$|$65.2_{0.5}$|$75.8_{1.9}$|\n| T5-large |      Full     |       - (-)      |$85.8_{0.1}$|$96.4_{0.0}$|$76.0_{2.6}$|$84.5_{0.1}$|$87.6_{0.4}$|$70.3_{1.6}$|$83.4_{1.3}$|\n| | Prompt Tuning |      100 (-)      |$83.1_{0.2}$|$90.5_{1.0}$|$62.0_{3.0}$|$82.6_{0.2}$|$87.0_{1.0}$|$66.2_{1.0}$|$78.6_{1.4}$|\n|  |    P-Tuning   |       20 (-)       |$83.4_{0.4}$|$91.7_{2.7}$|$71.7_{3.2}$|$84.2_{0.1}$|$87.6_{1.0}$|$69.2_{0.9}$|$81.3_{1.8}$|\n| |      SMoP     |     5 (4)    |   $83.5_{0.1}$   |$96.4_{0.0}$|$71.7_{3.1}$|$83.9_{0.2}$|$87.7_{0.0}$|$68.6_{0.7}$|$82.0_{1.3}$|\n\n- Furthermore, as stated in Appendix B, we have conducted a hyperparameter search on the baseline methods consistently with SMoP, which include learning rate, batch size, and training epochs.\n- We will report the experimental setup and results regarding the standard deviations in our paper.\n\n**Reason to reject 5. As described by the author in Limitation section, the SMoP method requires 20% more trainable parameters, but the reported training parameters in Table 1 is misleading?**\n- The setting in the limitation is where the total number of prompts is the same (a single soft prompt of length 20 for prompt tuning and four soft prompts of length 5 for SMoP). In contrast, the settings outlined in Table 1 are the configurations that yield the best performance (a single soft prompt of length 100 for prompt tuning and four soft prompts of length 5 for SMoP). \n- For the first case where the total number of prompts is identical, the linear router in SMoP introduces an additional parameter to tune given the same total number of prompts. The specific numbers are 15,360 vs. 18,432 trainable parameters for prompt tuning and SMoP, respectively. \n- For the second case where SMoP utilizes fewer soft prompts, the number of trainable parameters is lower than prompt tuning. The specific numbers are 76,800 vs. 18,432 trainable parameters for prompt tuning and SMoP, respectively.\n\n**Question 1. Could you please explain how you derive the trainable parameters in Table 1, especially for SMoP?**\n- The number of trainable parameters in prompt tuning can be derived as a sum of two components: a) soft prompt embeddings and b) additional parameters to reparametrize/select soft prompt embeddings.\n- Based on this, we provide our calculation process of trainable parameters in Table 1 on T5-base as an example. For T5-base, the embedding dimension e is 768, and the total count of pre-trained parameters is 222,882,048. \n1. SMoP\n- a) Uses four soft prompts of length 5, summing up to a total prompt length of 20. This counts up to 20 * 768 = 15,360 parameters. \n- b) The linear router for prompt selection is of shape e * number of prompts, which is 4 * 768 = 3,072 parameters. \n- The total number of trainable parameters is 15,360 + 3,072 = 18,432 parameters.\n- The ratio is computed as 18,432 / (18,432 + 222,882,048) * 100, which is approximately 0.0083%.\n2. Prompt Tuning\n- a) Uses a single soft prompt of length 100, which counts up to 100 * 768 = 76,800 parameters.\n- b) There are no additional parameters in Prompt Tuning. \n- The total number of trainable parameters is 76,800 parameters. \n- The ratio is computed as 76,800 / (76,800 + 222,882,048) * 100, which is approximately 0.0344%.\n3. P-tuning\n- a) Uses a single soft prompt of length 20, which counts up to 20 * 768 = 15,360 parameters.\n- b) A 3-layer MLP with the width of each FC layer 128, 128, and 768 is used for reparameterization. The number of parameters in each layer is computed as 768 * 128 + 128, 128 * 128 + 128, and 128 * 768 + 768. This sums up to 212,992 parameters.\n- The total number of trainable parameters is 15,360 + 212,992 = 228,352 parameters\n- The ratio is computed as 229,376 / (229,376 + 222,882,048) * 100, which is approximately 0.1028%.\n\n**Question 2. Why do you use the soft prompt with the highest probability only instead of using a weighted average of all learned prompts? Have you considered the strategy used in AdaMix?**\n- To clarify, the ‘Stochastic’ setting in Table 3 of our paper mirrors the methodology of AdaMix. Also, we have considered a weighted sum of soft prompts with top-2 probability, which is denoted as ‘Top-2’ in Table 3. We have observed that both methods underperform SMoP (SMoP 83.8 vs Top-2 78.7 vs Stochastic 78.1). As we have observed that the weighted sum of 2 soft prompts harms performance, we did not conduct any further experiments on the weighted sum of learned soft prompts.\n- We conjecture that during training, each soft prompt becomes an expert to the subset of the data, and the instance-wise selection of a single prompt is advantageous over the approach of combining multiple prompts in a mixture. This is in line with previous literature that highlights the significance of instance-specific soft prompts [2, 3].\n\n**Question 3. In your ablation study, can you provide a comparison to prompt-tuning with the same final length of prompts?**\n- Regarding the final length of prompts as the utilized prompt length, we provide a comparison between prompt tuning with soft prompt length 5 and SMoP with four length 5 soft prompts as an example in the table below.\n \n| Model | Method | Total Prompt Length | Utilized Prompt Length | Trainable Params % | Training Time | Memory Used | Inference FLOPs | SuperGLUE Average |\n|---|---|---|---|---|---|---|---|---|\n| T5-base | Prompt Tuning | 5 | 5 | 0.0017 | 60.7 | 11.9 | 71.5 | 72.5 |\n|  | SMoP | 20 | 5 | 0.0083 | 61.0 | 11.9 | 71.5 | 75.8 |\n| T5-large | Prompt Tuning | 5 | 5 | 0.0007 | 123.5 | 22.6 | 275.4 | 78.1 |\n|  | SMoP | 20 | 5 | 0.0033 | 129.1 | 22.6 | 275.4 | 82.0 |\n\n- To clarify, the memory usage and inference FLOPs are not identical, but the increased cost ratio is lower than 1e-6 which is negligible and thus not presented in the table.\n- We will include this comparison in our work as an ablation study as ‘single’.\n\nReferences\n\n[1] Asai et al., \"ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts\", EMNLP 2022\n\n[2] Wu et al., \"IDPG: An Instance-Dependent Prompt Generation Method\", NAACL 2022\n\n[3] Jiang et al., \"Instance-wise Prompt Tuning for Pretrained Language Models\", arXiv 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This method shows improved training and inference costs for the traditional prompt tuning.\n2. It is particularly designed for long soft prompts and will be beneficial in that case.", "reasons_to_reject": "1. Important baselines and critical comparisons are missing, including but not limited to: SPoT, Attempt, and MPT.\n2. Limited novelty given the Attempt framework. \n3. The proposed method requires further expensive hyper-parameter search, as shown in Table 2, and the optimal k, l setups are not easy to obtain, which makes the overall training more expensive and redundant than the original prompt tuning implementation. \n4. The experiments do not report multiple random seeds, and it appears to me that the improved performance in Table 1 can be due to cherry picking the seed and the exhaustive hyper-parameter search in Table 2.\n5. As described by the author in Limitation section, the SMoP method requires 20% more trainable parameters, but the reported training parameters in Table 1 is misleading?", "questions_to_authors": "1. Could you please explain how you derive the trainable parameters in Table 1, especially for SMoP?\n2. Why do you use the soft prompt with the highest probability only instead of using a weighted average of all learned prompts? Have you considered the strategy used in AdaMix?\n3. In your ablation study, can you provide a comparison to prompt-tuning with the same final length of prompts?\n\nI appreciate if the author could respond to the concerns in the weakness section.\n\n----------\nThanks for the response from the author. With more detailed experimental evidence, I increased my score for the solidness. I also appreciate the discussion from the author for the difference between SMoP and ATTEMPT. However, I still find the SMoP as a simplified and single-task version of ATTEMPT, which limits the whole novelty of this work, so I kept my excitement score unchanged. I believe a discussion between SMoP and relevant multi-task framework such as SPoT, Attempt, and MPT will be neccessary for making this work distinct for future manuscripts.", "paper_topic_and_main_contributions": "This paper proposes a sparse mixture of soft prompts, SMoP, that alleviates the training and inference cost for long soft prompts. It implements a linear router model for assigning the soft prompt with the highest probability as the final prompt. In addition, it introduces a router perturbation trick to encourage the exploration of different prompts in the training phase.  By conducting experiments on T5-base and large on 6 SuperGLUE tasks, it shows improved training cost and marginal performance improvement over the Prompt-Tuning and P-tuning baselines. However, other important baselines that show strong similarity in principle: Attempt [1], SPoT [2], MPT [3], are neither discussed nor compared. Therefore, the soundness of this work needs to be further improved. Moreover, due to the strong overlapping between the proposed methodology of SMoP and Attempt, the novelty of this work is very limited. Though Attempt is designed to focus on multi-task prompt transfer learning, Attempt can be interpreted as an improved version of SMoP with an additional attention module, whereas SMoP only considers a single soft prompt with the highest probability.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Vu T, Lester B, Constant N, Al-Rfou’ R, Cer D. SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer. ACL 2022.\n\n2. Asai A, Salehi M, Peters ME, Hajishirzi H. ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts. EMNLP 2022.\n\n3. Wang Z, Panda R, Karlinsky L, Feris R, Sun H, Kim Y. Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning. ICLR 2023."}}, {"review_id": "Chgw7vG5kv", "reviewer": "Reviewer_k8Ru", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging words and insightful suggestions. Below is our response, and we are keen to address any further concerns. \n\n**Reason to reject 1. Given the numbers are so close for both tables 1 and 3, it might be worth to compute their variance computed over different runs.**\n\n- We have conducted experiments on 3 different random seeds and reported the average value derived from these runs. The results along with their corresponding standard deviations are presented in the tables below.\n\n**Table 1**\n|   Model  |     Method    | $l$ ($k$) |   BoolQ   |     CB    | COPA | MultiRC | RTE | WiC | Avg. |\n|:--------:|:-------------:|:-------------:|:---------:|:---------:|:----:|:-------:|:---:|:---:|:----:|\n|  T5-base |      Full     |       - (-)       | $81.9_{0.1}$ | $96.4_{1.8}$ |   $64.3_{1.5}$  | $80.2_{0.2}$ | $79.2_{0.2}$ | $67.0_{2.3}$ | $78.2_{1.3}$    |\n|  | Prompt Tuning |      100 (-)      | $79.1_{0.1}$| $86.9_{3.7}$ |$56.7_{2.1}$|$78.3_{0.2}$|$73.2_{1.7}$|$65.6_{1.2}$|$73.3_{1.9}$|\n|  |    P-Tuning   |       20 (-)      |$78.7_{0.2}$|$91.7_{2.7}$|$58.3_{3.8}$|$79.3_{0.2}$|$77.3_{1.8}$|$65.9_{0.7}$|$75.2_{2.1}$|\n|  |      SMoP     |     5 (4)    |$79.4_{0.3}$|$94.6_{1.8}$|$58.3_{2.9}$|$79.6_{0.1}$|$77.5_{3.2}$|$65.2_{0.5}$|$75.8_{1.9}$|\n| T5-large |      Full     |       - (-)      |$85.8_{0.1}$|$96.4_{0.0}$|$76.0_{2.6}$|$84.5_{0.1}$|$87.6_{0.4}$|$70.3_{1.6}$|$83.4_{1.3}$|\n| | Prompt Tuning |      100 (-)      |$83.1_{0.2}$|$90.5_{1.0}$|$62.0_{3.0}$|$82.6_{0.2}$|$87.0_{1.0}$|$66.2_{1.0}$|$78.6_{1.4}$|\n|  |    P-Tuning   |       20 (-)       |$83.4_{0.4}$|$91.7_{2.7}$|$71.7_{3.2}$|$84.2_{0.1}$|$87.6_{1.0}$|$69.2_{0.9}$|$81.3_{1.8}$|\n| |      SMoP     |     5 (4)    |   $83.5_{0.1}$   |$96.4_{0.0}$|$71.7_{3.1}$|$83.9_{0.2}$|$87.7_{0.0}$|$68.6_{0.7}$|$82.0_{1.3}$|\n\n**Table 3**\n|  Model  |      Method      | BoolQ | CB | RTE | Avg. |\n|:-------:|:----------------:|:-----:|:--:|:---:|:----:|\n| T5-base |       SMoP       |$79.4_{0.3}$|$94.6_{1.8}$|$77.5_{3.2}$|$83.8_{2.1}$|\n|         | w/o perturbation |$79.7_{0.2}$|$93.5_{2.7}$|$76.0_{1.5}$|$83.1_{1.8}$|\n|         |       Top-2      |$78.4_{0.2}$|$88.1_{1.0}$|$69.7_{0.4}$|$78.7_{0.6}$|\n|         |  Gumbel-Softmax  |$79.2_{0.4}$|$92.3_{2.0}$|$75.2_{4.3}$|$82.2_{2.7}$|\n|         |    Stochastic    |$78.2_{0.3}$|$86.9_{2.1}$|$69.2_{1.7}$|$78.1_{1.6}$|\n\n- We will include the standard deviations of our experimental results with the clarification on our experimental results. Thank you for your valuable comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work presents a clear and well-motivated technique that requires less GPU memory. The overall idea is simple and is derived from past work on sparse mixture of experts models from the DL community. This makes the work accessible to a wider community and increases its applicability.\n- Experiment results are satisfactory and validate the main premise of the paper, namely that SMoP is effective.\n \n", "reasons_to_reject": "- Given the numbers are so close for both tables 1 and 3, it might be worth to compute their variance computed over different runs. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work focuses on improving the efficacy of prompts using short prompts using a sparse mixture. The idea is equivalent to a sparse mixture of models, and in a similar manner, it has a gating mechanism that allows the proposed model to specialize different prompts for different subsets of the dataset. The paper supports its claims with strong empirical performance without compromising the training and inference computation. Furthermore, this work uses memory efficiently and scales well with prompt length.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eH9gd8TEJS", "reviewer": "Reviewer_xer5", "summary": "", "questions": "Please address the concerns in *Reasons to Reject*.\n\nI view the first point of *Reason to Reject* as the biggest concern and I'll adjust my rating conditional on the other reviews and if the authors could provide a satisfactory author response.\n\n## Post-rebuttal comments\n\nI thank the authors for providing a detailed rebuttal. Below, I provide detailed comments:\n1. Comparison with AdaMix: Thank you for the clarification. Please incorporate the changes in the camera-ready version.\n2. Comparison with LoRA: I do not disagree with the authors' saying that prompt tuning is less invasive, but my point was that prompt tuning while requiring no architecture changes, still requires *access* to the parameters, and you still cannot run prompt tuning in models where you may only observe their outputs in a black-box manner, and thus there are only limited scenarios where you can run prompt tuning but not, for example, LoRA -- I am stating this for clarification, not as a point against the authors as they indeed provide LoRa comparison. While I thank the authors for providing additional experiments in a short timeframe, my concerns are: 1) It seems that the difference between LoRA and SMoP is minimal and likely statistically insignificant, and 2) while SMoP is indeed more parameter-efficient in the Table, both LoRA and SMoP involve hyperparameters that significantly affect their parameter efficiency, and it is unknown whether a lower rank value in LoRA would lead to a similarly competitive performance (one key advantage claimed by LoRA authors is that LoRA is not very sensitive to the rank choice).\n\nIn general, I have become more positive about the work and would have increased the excitement score to 3.5 had I had the choice as in previous years. The reason I cannot give an even higher score is stated in Point 2 of my response.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging words and insightful suggestions. Below are our responses, and we are keen to address any further concerns. \n\n**Reasons to reject 1. Relation and discussion with respect to previous works (AdaMix) should be more explicitly stated.**\n- Thank you for your suggestion, we would like to provide an explicit comparison of SMoP and AdaMix, which will be included in the camera-ready version.\n- SMoP and AdaMix have substantial differences in their motivation and methodology. The primary motivation of SMoP is to use multiple short soft prompts for efficient prompt tuning, and each soft prompt is to be specialized into a subset of the task dataset. Therefore, SMoP employs a linear router for instance-wise prompt selection. Conversely, the motivation of AdaMix is to provide multiple views of the given task for better performance [1]. Hence, AdaMix employs stochastic routing and merging, resulting in a single adapter module per task.  Thus, while both works have taken inspiration from the concept of the Mixture-of-Experts structure, their motivations and methodologies are distinct.  \n\n- Additionally, we have empirically observed that directly applying the single-module-per-task methodology of AdaMix to prompt tuning leads to lower performance compared to SMoP. Table 3 in the paper shows the results regarding this observation (SMoP 83.8 vs. Stochastic 78.1). To clarify, the “Stochastic” configuration in Table 3 mirrors the method of AdaMix, including consistency regularization and merging.\n\n**Reasons to reject 2. Clarification in the superiority of prompt tuning and comparison to other lightweight methods (e.g., LoRA).**\n- First, we would like to clarify the superiority of prompt tuning. Prompt tuning approaches modify the input of the model by prepending soft prompt tokens to the original input. This adjustment does not necessarily entail modification to the internal structure of the model [2]. \n- A clear example that emphasizes the difference is where several tasks are processed in the same batch. Prompt tuning can handle this with a single model since different prompts corresponding to the task of each input can be prepended, but adapter-based approaches require copies of the model for each task [3]. \n- Another advantage of prompt tuning approaches is their parameter efficiency. They solely introduce additional parameters to the input of the model without being influenced by the number of layers in the underlying backbone model. \n- As the main motivation of SMoP is to alleviate the inefficiencies in prompt tuning associated with long soft prompts, we have presented a comparison with prompt tuning methods in Table 1 as our main experimental results. To address your concerns regarding comparison to other lightweight methods, we conducted additional experiments with the T5-base model on three adapter-based approaches, namely Adapter [4], AdapterFusion [5], and LoRA [6]. The results are as follows:\n\n| Method | Trainable % | BoolQ | CB| COPA| MultiRC | RTE | WiC | Avg. |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Adapter (m=48) | 1.5800 | 81.0 | 95.5 | 60.3 | 80.3 | 75.8 | 67.0 | **76.6** |\n| AdapterFusion (m=48) | 0.7963 | 79.5 | 94.6 | 58.3 | 79.2 | 79.2 | 65.9 | 76.1 |\n| LoRA (rank=8) | 0.3954 | 79.0 | 90.5 | 59.7 | 80.0 | 77.9 | 66.9 | 75.7 |\n| Adapter (m=8) | 0.2806 | 79.0 | 90.2 | 58.0 | 80.0 | 73.6 | 65.1 | 74.3 |\n| **SMoP (l=5, k=4)** | **0.0083** | 79.4 | 94.6 | 58.3 | 79.6 | 77.5 | 65.2 | 75.8 |\n\n- In this table, \"m\" for Adapters represents the bottleneck dimension of the adapter layer, whereas \"l\" and \"k\" for SMoP represent the utilized soft prompt length and the number of soft prompts. The methods are listed in descending order based on their trainable parameter ratio.\n\n- The experimental results demonstrate that SMoP achieves comparable performance to adapter-based lightweight methods while utilizing less than 47~190 times the trainable parameters. Notably, SMoP outperforms adapter with bottleneck dimension 8 which employs 33 times more trainable parameters than SMoP. These results highlight the parameter efficiency of SMoP compared to adapter-based approaches. We will include these results in our camera-ready version.\n\n**Reason to reject 3. Empirical evidence on orthogonality of SMoP to various prompt tuning methods.**\n- We believe that exploring the application of SMoP to various prompt tuning methods holds significant potential for future research, and we are engaged in conducting investigations in this direction. Thank you for your suggestion. \n\nReferences\n\n[1] Wang et al., \"AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning\", EMNLP 2022\n\n[2] Pfeiffer et al., “Modular Deep Learning”, arXiv 2023\n\n[3] Bari et al., “SPT: Semi-Parametric Prompt Tuning for Multitask Prompted Learning”, arXiv 2022\n\n[4] Houlsby et al., “Parameter-Efficient Transfer Learning for NLP”, ICML 2019\n\n[5] Pfeiffer et al., “AdapterFusion: Non-Destructive Task Composition for Transfer Learning”. EACL 2021\n\n[6] Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, ICLR 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- I think the overall methodology is reasonable and convincing and addresses an important problem (as the authors state on Page 1 that \"they have overlooked the inefficiency\" associated with longer soft prompts). \n- As briefly mentioned in Limitations, I think the proposed method can be largely plug-and-play for various soft prompt tuning methods, as any prompt tuning methods utilising the soft tokens could be potentially broken up into small pieces with a routing network to achieve reduced cost. I think this is another plus in favour of this paper.\n- The experiments are executed satisfactorily. The ablation studies are included, such as the trade-off between prompt length and number of prompts and the use of different routing mechanisms. I think these experiments would greatly improve readers' confidence in the method's working mechanism.", "reasons_to_reject": "- One of my major concerns is that the relation to previous works should be more explicitly stated. In my view, the proposed method is very similar to the previous work of AdaMix. While AdaMix uses a mix of *adapters*, the present paper uses a mix of *prompts*. However, the soft prompts can be deemed as adapters only inserted into the embedding layer of the PLMs, and thus I'd be grateful if the authors could discuss especially w.r.t this very related work. While the paper did cite AdaMix (Wang et al. 2022), the only references are 1) that using many soft prompts leads to performance deterioration and 2) comparing against stochastic routing. I feel that the similarity (at least *prima facie*) at a more fundamental level to this work warrants a more thorough discussion.\n\n- While the authors compare favourably against methods such as prompt tuning and P-tuning, I'm wondering how does the method compare to other lightweight methods that have been recently shown to be very effective, such as LoRA? In decomposing updates into low-rank matrices, LoRA similarly reduces the training cost by effectively reducing the number of parameters to fine-tune during the PEFT procedure, and it also offers an adjustable parameter (i.e., the rank of the update) that allows the user to trade-off the cost and precision. The authors tried to argue on Page 1 that prompt tuning is superior in its simplicity and that \"it does not require modifications to the model structure\" -- I think soft prompt tuning still requires modifying the model structure, just in a less invasive way; the input embedding layer still has to be modified (c.f. hard prompt tuning requiring only modification on the input texts).\n\n- Another minor point related to the Strength I mentioned: while the authors conceptually argued for (and I agree with) that the proposed method is orthogonal to various prompt tuning methods, it would be much more convincing if the authors provided empirical evidence.  However, I do not view this point as a critical point against the authors but rather as a suggestion for future improvement.", "questions_to_authors": "Please address the concerns in *Reasons to Reject*.\n\nI view the first point of *Reason to Reject* as the biggest concern and I'll adjust my rating conditional on the other reviews and if the authors could provide a satisfactory author response.\n\n## Post-rebuttal comments\n\nI thank the authors for providing a detailed rebuttal. Below, I provide detailed comments:\n1. Comparison with AdaMix: Thank you for the clarification. Please incorporate the changes in the camera-ready version.\n2. Comparison with LoRA: I do not disagree with the authors' saying that prompt tuning is less invasive, but my point was that prompt tuning while requiring no architecture changes, still requires *access* to the parameters, and you still cannot run prompt tuning in models where you may only observe their outputs in a black-box manner, and thus there are only limited scenarios where you can run prompt tuning but not, for example, LoRA -- I am stating this for clarification, not as a point against the authors as they indeed provide LoRa comparison. While I thank the authors for providing additional experiments in a short timeframe, my concerns are: 1) It seems that the difference between LoRA and SMoP is minimal and likely statistically insignificant, and 2) while SMoP is indeed more parameter-efficient in the Table, both LoRA and SMoP involve hyperparameters that significantly affect their parameter efficiency, and it is unknown whether a lower rank value in LoRA would lead to a similarly competitive performance (one key advantage claimed by LoRA authors is that LoRA is not very sensitive to the rank choice).\n\nIn general, I have become more positive about the work and would have increased the excitement score to 3.5 had I had the choice as in previous years. The reason I cannot give an even higher score is stated in Point 2 of my response.", "paper_topic_and_main_contributions": "This paper proposes SMoP, a mixture-of-expert method to approach parameter-efficient fine-tuning (PEFT). The authors first note that previous literature has noted the improved performance of using longer soft prompts over short ones, but doing so increases the context length and thus increases the computational expense. SMoP instead tunes multiple short prompts but combines them using a gating mechanism, a network that selects and weighs the different short prompts that are trained jointly. The authors performed experiments on two T5 models and show improvement in cost-performance trade-off compared to previous prompt tuning methods in a suite of SuperGLUE datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work focuses on increasing the parameter efficiency of prompt tuning through the use of sparsely selected prompts. Authors managed to fit many interesting results in a short paper and provide convincing evidence during the rebuttal. All reviewers agree on the usefulness of the method and experimental evidence (which is almost at a long-paper level). I encourage authors to add the relevant work and comparison discussed during rebuttal (e.g. Adamix, Attempt).
Decision: Accept-Main