Kati8/ICLR_2023_2025 huggingface dataset:
authors: Chen Dun
authors: Mirian Del Carmen Hipolito Garcia
authors: Guoqing Zheng
authors: Ahmed Hassan Awadallah
authors: Anastasios Kyrillidis
authors: Robert Sim
reviews: 
Comments: This paper introduces a "mixture of prompts" technique, which selects relevant prompts for a prediction task based on a pre-trained routing module. In experiments, they find their method outperforms a prompt tuning baseline in a standard and federated learning setup interms of perplexity on Dolly and Super natural instructions.
Strengths: - The idea of routing relevant prompts is useful -- it could make learning a new task quite rapid and only requirese inputting new prompts 
- The presented results are promising in terms of improving performance. I think there are benefits of this method more generally in terms of ease of use -- it seems simpler to use than a prompt finetuning strategy.
Weaknesses: There are some discussion points / experimental evaluations that would be useful to consider:
- A conceivable alternative to this method is including all available prompts in the context window of the language model for few-shot learning -- what's the motivation for using something like this method over this? Does it mitigate the "interference" described in the paper? Are there additional memory + computational constraints?
- Moreover, there are many peft techniques that have shown promise in learning from minimal labeled examples. I understand it may not be possible to baseline all of them, but including the most frequently used techniques, like LORA, would make the results more compelling to many readers to consider this method. I understand there are efficiency motivations here as well, but it would be useful for many readers in the centralization training I believe.
- There are a few simple baselines to consider for this method. In general, it seems like the method upweights relevant prompts in the weighting module for use for the task. It's fairly common just to select prompts for a particular task using KNN on embeddings from a larger set. How would this compare to the current technique? Or, is it expected to be worse for some reason?
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 2 fair
reviews: 
Comments: This paper proposed gating function is able to identify relevant skills for the current task and dynamically select and combine prompts accordingly. The authors claim that the proposed method can overcome prompt training interference from multi-tasks across centralized and federated learning scenarios.
Strengths: This paper utilized a mixture of prompts to deal with heterogeneous tasks and data distributions. The performance seems to improved a lot compared with the baseline.
Weaknesses: 1. A mixture of prompts has been proposed in the NLP field in 2021 [1].

[1]. Qin, Guanghui, and Jason Eisner. "Learning how to ask: Querying LMs with mixtures of soft prompts." arXiv preprint arXiv:2104.06599 (2021).

2. The presentation of this paper makes me confused. Is the "Mixture of Prompts for LLM Task Adaptation" designed specifically for federated learning? I mean whether the method is designed for multi-task, multi-source scenarios. The multitask prompt tuning method proposed in [2] also enables parameter-efficient transfer learning. Is the method in this paper an incremental setting based on [2].
[2]. Zhen Wang, "Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning", ICLR 2023.

3. How do you observe the "aggressively compressed LLMs"? It seems that the authors utilized SparseGPT to get the compressed LLM. If so, any contribution to this point?

4. I think this paper combined many existing technologies into a good technique report.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper proposes an innovative method for improving the efficiency and adaptability of large language models (LLMs) in handling diverse tasks and data distributions. This method, called Mixture of Prompts (MoPs), uses smart gating functionality to dynamically select and combine different groups of prompts, acting as 'experts,' based on the task at hand.
Strengths: + This paper introduces a novel approach to prompt tuning using Mixture of Prompts (MoPs) with a smart gating function. This is a significant contribution as it addresses the challenge of task and data heterogeneity, which is particularly relevant for real-world applications of LLMs.

+ The paper demonstrates that the MoPs method is resilient to various model compression techniques, indicating that it can maintain performance even when computational resources are limited or when efficiency is a priority.

+ The proposed method shows a substantial decrease in final perplexity compared to baselines in both federated and centralized scenarios. This empirical performance is well-documented through experiments, providing strong evidence for the effectiveness of MoPs.
Weaknesses: - While the paper introduces an innovative approach, it does not thoroughly discuss the complexity involved in implementing the MoPs method. Practical application details are crucial for adoption, and this could be an area requiring further clarification.

- The experiments conducted to demonstrate the efficacy of the MoPs method are limited to only two datasets. A broader range of datasets and scenarios would provide a more comprehensive understanding of the method's performance and generalizability.

- Although the paper mentions the mitigation of model drift problems, there is a lack of in-depth analysis of how the MoPs approach ensures the model's stability over time, especially in federated learning scenarios where model drift is a significant concern.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 2 fair
Clarity: 3 good