Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the prompt tuning of Transformer language models in multi-task settings. They proposed a method named multitask prompt tuning (MPT), which aims to enhance the transferability of source prompts (i.e., the learned soft prompts for source tasks). Specifically, they learn a single transferable prompt by knowledge distillation from multiple source tasks and then learn the rank-one matrix for adapting the transferable shared prompt to a given target task -- prompt decomposition, distillation, and adaptation. On the benchmark GLUE and SuperGLUE, they compare the proposed MPT method with many other prompt tuning baselines. The MPT outperforms the baseline methods and has a smaller number of the parameters.  
strength_and_weaknesses: ## Strength

- A novel method for multi-task prompt tuning. The design of the prompt decomposition and distillation is intuitive and reasonable. 
- Great empirical results on GLUE and SuperGLUE. The proposed method MPT outperforms many recent baseline methods for prompt tuning. 
- Comprehensive analysis with ablation studies and qualitative analysis with heat maps. 

## Weakness
- The evaluation does not consider the NLG tasks, such as those in the GEM benchmark. This can be a big limitation. 
- The evaluation is based on the setup where only 6 source tasks are used. This is a pretty small size and it seems that many source tasks are closely related to each other. I would suggest authors use benchmarks such as CrossFit (Ye et al. 2022) to do a more large-scale analysis, where the transferring is more challenging as some source tasks can be relatively less related to the target tasks. 
- The current method design only considers a single shared prompt for transfer. I think when you have a large number of source tasks, this can be a weak point. As it is less likely that a very diverse set of tasks can use a single prompt to share all knowledge.  
clarity,_quality,_novelty_and_reproducibility: As shown in Fig 5, it seems that 300 is still not the optimal prompt length for MPT. Can you use larger lengths and find the optimal length (the one with the best performance)?   


summary_of_the_review: Overall, I enjoy reading the paper. The idea is pretty novel and its performance is very good, especially when we consider the parameter efficiency. It has a few limitations which are not stated and covered, though. Also, I think the evaluation can be further improved according to my above suggestions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a soft (continuous) prompt tuning method called MPT. In traditional soft prompt tuning, prompts are often sensitive to initialization when trained from scratch and performance may still lag behind full model fine tuning. In this work, the manuscript presents a method for multitask prompt tuning where a single soft prompt is learned that can be transferred to target tasks. The authors find a low rank decomposition based on a source task matrix and a task-specific low rank matrix is more performative than sharing the prompts directly across tasks. This decomposition is is learned via a knowledge distillation style approach.

The authors evaluate performance on 21 NLP datasets reflecting a variety of tasks and report significant improvements on SuperGLUE vs vanilla prompt tuning, along with all the smaller training parameter benefits of parameter efficient transfer learning. They further find that MPT performs well in few-shot learning for models in the 60M to 770M parameter space. The paper presents comprehensive ablation experiments
strength_and_weaknesses: Strengths
- Parameter-efficient transfer learning is an important research area 
- The prompt decomposition method is quite straightforward (decomposition + distillation)
- Comprehensive evaluation (21 datasets) and baseline methods
- Nice breadth of additional experiments (few-shot performance, LM param scaling, prompt lenght)
- Ablation studies highlight benefits of combining decomposition + distillation

Weaknesses
- Ideally the manuscript would explore some class of larger language models (3 - 11B param range), though this presupposes some level of compute that is not available to all researchers, so it is not a strong criticism.
- Experiments would benefit from replicates to characterize variance.
- The core methods aren't super novel (decomposition + distillation), but the combination seems to provide empirical benefits.  
- Code is not immediately available. 
clarity,_quality,_novelty_and_reproducibility: - The paper is very well written, with clear notation, figures and exposition. 
- Overall novelty is modest, but the method is simple and provides benefits
- Authors will provide code for reproducible results
summary_of_the_review: I think this paper makes several nice empirical contributions, is very clearly written with comprehensive evaluations and ablation experiments. This covers some reasonable questions in the space of soft prompt tuning and MTL so it merits acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new method for multi-task prompt tuning, which uses source tasks to learn a single shared prompt and then adapts to target tasks with decomposition and distillation. The design of the decomposition makes the resulting prompting learning more performant yet more parameter-efficient. 
strength_and_weaknesses: Summary Of Strengths
- the paper is clearly written and presented;
- the idea of leveraging decomposition is new and insightful, which not only makes the prompt learning more performant but results in fewer parameters. 
- extensive ablations are provided (decomposition, distillation, adaptation strategy, and training strategy) to illustrate the design choices, which may pave the road for future researchers and practitioners in the prompt learning area. 
- the efficacy of the decomposition and distillation for multi-task prompt tuning is verified across benchmarks (GLUE, SuperGLUE, MRQA) and scales (up to 700M); 

Summary Of Weaknesses
- the additional training compute is unclear compared to fine-tuning, is the fine-tuning also conducted using the same schedule as MPT; Also, in the SPoT paper, they found the best results were achieved with multi-task fine-tuning 79.2 (T5-base), is it also a valid baseline to compare with?
- the poor baseline performance 

   (i) It seems the baseline of BitFit / LoRA / Adapter performs worse than the one reported in [1], could the author elaborate on the reasons?

   (ii) Also, is there any intuition  why SPoT yields such worse performance on SuperGLUE, which should be comparable to Model Tuning in the original paper (though they use more source tasks compared to the one used here)

- the adaptation in Table 1 and 2 is still per-task adaption but not in a multi-task manner (the design choice of this is unclearly presented), and how to select the group is unclearly presented; 

- the generalization of the proposed method is uncertain, is it limited to T5-variants, or is it also applicable to GPT (casual mask) models?

- will the variance of different runs also be given in Table 1 and 2, which can help show the significance of the results? 

- for the few-shot setting, is the source prompt learning still using the full set of the source tasks, or is that also few-shot?

[1] Sung, Yi-Lin, Jaemin Cho, and Mohit Bansal. "Lst: Ladder side-tuning for parameter and memory efficient transfer learning." NeurIPS (2022).

clarity,_quality,_novelty_and_reproducibility: - The paper is clearly presented; 
- Though multi-task prompting in NLP has been investigated in [2,3], distillation [4] decomposition of prompting has been investigated in [4], the idea to use low-rank decomposition for multi-task target prompt adaptation is interesting and new. 
- Some implementation details regarding the might also be vital for reproducibility. 


[2] Vu, Tu, et al. "SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.
[3] Sanh, Victor, et al. "Multitask prompted training enables zero-shot task generalization." ICLR (2022).
[4] Zhong, Qihuang, et al. "Panda: Prompt transfer meets knowledge distillation for efficient model adaptation." arXiv preprint arXiv:2208.10160 (2022).
summary_of_the_review: See above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies multitask prompt tuning to learn better soft prompt representations for tasks. Their proposed approach has two stages: 1) They first train a single source prompt representation for each individual task using the conventional prompt training approach; 2) then they learn a shared prompt representation and task-specific prompt representations on all tasks by applying the prompt distillation from the teacher prompt obtained from step #1. They evaluated their approach on a few well-established benchmarks, including GLUE and SuperGLUE, and demonstrated their prompt tuning approach is better than the prior prompt tuning approaches with less parameters. 
strength_and_weaknesses: Strengths:
- A novel multitask prompt tuning approach by the separation of shared and task-specific prompt representations and knowledge distillation. The idea is simple and technically sound.
- Evaluations show their approach achieves better results than prior prompt tuning approaches with fewer parameters on widely-adopted benchmarks. Their approach is also on-par or slightly better than the finetuning baseline. 
- Their approach demonstrates significantly better few-shot capability than finetune baseline and other prompt tuning approaches.
- Code & data will be released.

Weaknesses:
- Compared to Adapter or finetuning baseline, the proposed approach is still worse on certain datasets (table 1 & 2). It would be better to show if the performance gap w.r.t. Adapter can be closed by adding the same number of prompt parameters as the Adapter.
- It would be more convincing if the few-shot experiments can be performed on GLUE and SuperGLUE, instead of 3 separate datasets. Similar to the scaling experiment (i.e., Figure 4)

clarity,_quality,_novelty_and_reproducibility: Please see the comment above. 
summary_of_the_review: The paper proposes a novel multitask prompt tuning approach. Their approach shows better effectiveness and efficiency compared to prior prompt tuning baselines. The results & findings can be more convincing if the weaknesses can be resolved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a novel prompt tuning approach that learns a transferable prompt by distilling from multiple task-specific prompts. The approach is interesting and the experiments verified the efficacy of the method. All reviewers agree that this paper should be accepted.
justification_for_why_not_higher_score: Need larger scale experiments (not necessarily bigger models, but more tasks > 100). I know the authors have attempted to run with more tasks (12 instead of 6) following the suggestion from a reviewer. I also understand that there is a compute resource limitation on the authors side. However, since the model is pitched as a multitask model, I still think this kind of experiments is needed to for a higher score. 
justification_for_why_not_lower_score: Interesting approach that has the potential to be widely adopted.
Decision: Accept: poster