reviews: 
Comments: This paper proposes a mixture-of-prompts (MoP) method for multi-task, multi-source prompting tuning in LLMs. The key innovation of the method is that instead of using a single prompt to tune all tasks, which the authors claim could lead to degraded performance due to interference between tasks, MoP prompt tuning trains multiple soft prompts and uses a dynamic, learnable gating functionality to scale the attention weights in a task-dependent manner among the different prompts. The authors show the effectiveness of their methods in compressed LLMs and both centralised and federated setups and show improvement over baseline prompt tuning methods in the setups.
Strengths: - The paper studies a specific problem setup that is both of great importance and is currently rather less extensively studied. While prompt tuning in different flavours has been extensively studied in the literature, multi-task, multi-source training in compressed and/or federated setup is not. Such a setup is both more challenging and more practical and thus is of great research and practical value to the community (some caveats to this point are stated in “Weaknesses”).
- The research problem is of relevance, the hypothesis is reasonable, and the method proposed is sound. Inspired by the mixture of expert literature, addressing the prompt tuning problem with heterogeneity with MoP is natural and intuitive. The benefits of the key ideas, such as the use of smart gating and its pretraining, are evidenced well by the experiments (such as Fig 2)
Weaknesses: - While I appreciate the investigations in the setup as mentioned in “Strengths”, it seems that the method proposed is generic and not catered specifically to the federated or compressed setup, and these considerations are orthogonal. The experiments, however, exclusively test the MoP method in models aggressively compressed with pruning. I’m not sure if this is due to computational constraints, but I wonder how the method will perform in a less aggressively compressed LLM. It seems that the extent of gain decreases as pruning moderates. The authors made a comment on Page 9 that “if the pruning ratio is too low, there may not be enough room for improvement.” — I think the paper would benefit if the authors could add more explanations as to why this is the case.
- I also have some concerns regarding baseline comparisons and discussions w.r.t. the literature. The authors only consider a baseline proposed by Xu et al., 2023, which is standard prompt tuning on a compressed model after compression takes place. However, several other, arguably stronger baselines addressing the similar problem of prompt interference in multitask prompt tuning setup, as identified in Sec 3, are available, like [Wang 2023] (who seems to show that using a single prompt for many tasks works fine as long as we do low-rank task-specific updates afterwards) and [Vu 2022] — these works do not specifically target the compressed setup, but there’s no reason why they can’t work in it; it would be nice if the authors could discuss these works qualitatively and quantitatively. Additionally, without a comparison, I feel that the authors claim that their methods synergise with compression/federation learning setups particularly well (*“Via experiments, we have observed an emerging ability of MoPs: they work out of the box, regardless of any reasonable model compression ratio or technique (i.e. pruning, quantization)”* is a bit controversial, as the baseline prompt tuning methods may potentially work *generally* very well (note that Xu et al., 2023 that the authors compared against already shows such a trend) and the good performance is not due to the MoP design *specifically*. If it is indeed the case, the argument should be toned down a bit, in my opinion.
    - Several works use the mixture of prompt/adapter ideas that are missing, such as [Wang 2022], although I agree that the motivation or the problem setup is not identical. In many of these works mentioned above and works like Attempt (Asai et al., 2022, which the authors cited but did not discuss in detail), mechanisms similar to the gating function that authors claimed as a key contribution were proposed and I encourage the authors to compare and contrast.
- Better design ablations should be done: for example, the authors proposed to use the LLM intermediate embedding instead of a separate embedding for the gating function, and they decided to use the embedding after the 10th layer. These design decisions seem a bit ad-hoc; it’d be nice if ablations were done to demonstrate the impact of different design aspects on the final performance.


### Minor:
- It is better to dedicate Sec 2 entirely to background and related work for clarity of writing, but the authors blended their own proposals, such as the injection of prompts in the middle layers in this section. Separating the two components would help with clarity and flow.


## References
[Wang 2023] Wang, Z., Panda, R., Karlinsky, L., Feris, R., Sun, H., & Kim, Y. Multitask prompt tuning enables parameter-efficient transfer learning. ICLR 2023.

[Vu 2022] Vu, T., Lester, B., Constant, N., Al-Rfou, R., & Cer, D. Spot: Better frozen model adaptation through soft prompt transfer. ACL 2022

[Wang 2022] Wang, Y., Agarwal, S., Mukherjee, S., Liu, X., Gao, J., Awadallah, A. H., & Gao, J. AdaMix: Mixture-of-adaptations for parameter-efficient model tuning. EMNLP 2022
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good