Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose the progressive prompts that add a new prompt for each task and fix all other parameters from both feature extractor e.g. BERT, T5 and the prompt from previous tasks.

The new method should be able to both transfer knowledge from the previous task to new task, as well as reduce the forgetting as the prompt for previous tasks are unchanged.
strength_and_weaknesses: Strength:
1) The writing seems very good that makes paper easy to follow.
2) The forgetting can be alleviated as the prompt for previous tasks are unchanged.
3) Transfer of knowledge from the previous task to new task is achieved by the new approach.
4) A lot better results on the experiments.

Question:
1) Should we compare the AdapterCL or other adapter model that could add a new part / parameters for each task, for the empirical results?
2) For the eval results in Section 5.1 and 5.2, the progressive prompt on T5 has much larger gap than the other base lines. The BERT model with progressive prompt has not as large enhancement. Are there some root cause? Let me know if my understanding are not good.
clarity,_quality,_novelty_and_reproducibility: Quality and clarity seem both good.

Novelty seems to fall under incremental from technical perspective -- the method to use expansion on model parameter for CL may be explored before by a few researchers.
summary_of_the_review: The recommendation would be accept but marginally above the threshold. My reason for not a higher score results from the somewhat incremental on the technical novelty. My reason for not a lower level results from large gap on the performance compared to other method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper concerns language models in the pretrain-then-prompt-tune paradigm, and aims to introduce a prompting method to allow for  solving multiple tasks.  This can be described a continual learning setup, where the task ID is known to the model at both training and inference time.  The method, called Progressive Prompts, concatenates all prompts learned for previous tasks and learns a new task-specific prompt for each new task; thus the full prompt for task $N$ is the concatenation of the (previously-learned) prompts for tasks $1$ through $N-1$, concatenated with the newly-learned prompt for task $N$.  

Evaluating their method on text classification tasks, the authors find favorable performance compared to their chosen set of baselines, which include both methods that fine-tune the full language model (with or without data replay/buffer) and other purely prompt-based approaches. 

An additional finding of the paper is that parameterization of the prompt matters for performance; specifically, the authors parameterize the  (soft) prompt via a residual MLP and find via ablations that this parameterization improves performance.
strength_and_weaknesses: Strengths:
- The paper is well written and generally easy to follow.
- The observation that parameterization of the prompt matters so much for performance is one that I had not appreciated before.
- The novel observation in this paper, that Progressive Prompting enables forward transfer is interesting.

Weaknesses:
For me, the main weakness of this paper---which prevents me from recommending acceptance---is that the connection to catastrophic forgetting (and thus, the motivation for many of the baselines reported in the paper) does not make sense.  Let me state this objection/confusion as clearly as possible so the authors may let me know if I have misunderstood, in which case I would be happy to amend my score.  As I see it, the problem of catastrophic forgetting in language models is a problem that only occurs in a fine-tuning setting.  In this work, the authors have specified the (common in continual-learning) setup in which the model knows the identity of the task both during training and during inference.  Since prompt tuning (either soft prompt-tuning or manual natural-language prompt tuning) has  now become standard practice for adapting pre-trained language models to a particular task, the standard thing to do for adapting a pretrained language model for multiple tasks is to learn a separate prompt for each of the tasks, and at inference time (since the task identity is assumed to be known) simply select the prompt corresponding to the given task and run the model using that prompt.  In such a setting, there is no forgetting since the model parameters themselves do not change.  Thus in some sense it seems like this paper is attempting to solve a problem which already does not exist in the prompt-engineering paradigm.  The baselines which involve retraining a full model (e.g. the baselines "finetune", "EWC", "A-GEM", "Experience replay", "MBPA++", "IDBR") don't seem relevant to me.

Having said the above, the "standard thing" that I mentioned is one of the benchmarks that the authors consider (Per-task prompts).  In my mind, the fact that the ProgressivePompts beats Per-task prompts is *the* interesting contribution of this paper, and (as the authors say) indicates some forward transfer between prompts.  This could be useful in many scenarios, especially in the low-data regime for subsequent tasks.  If the paper had been written with this as the main selling point (with all of the natural ablations and sanity-checks one would want), I would likely have been much more positive on it.

I think there are two ways in which I would be convinced to change my score:
1. The authors explain why my above argument is wrong, i.e. I have some misunderstanding and the problem of catastrophic forgetting is actually relevant in the prompt-engineering world; or, 
2. The authors reframe their paper as being about the forward transfer phenomenon, and conduct more thorough experiments to explore that phenomenon (e.g., how does the transfer vary with dataset size for the subsequent tasks, how does it compare to a single prompt of length 2M (where M is the length of each individual prompt), etc.).  

clarity,_quality,_novelty_and_reproducibility: Clarity: the paper was generally quite clear and well-written.  I have some minor comments here, like in Figure 1, the "snowflake" indicates parameters being frozen, but this wasn't explicitly mentioned anywhere; or in the setting about baselines, it's not explicitly mentioned which ones require finetuning and which ones don't.  But these are very minor and could easily be addressed between acceptance and final camera-ready version.

Quality and Reproducibility: No issues, these are both fine.

Novelty: This is related to my comment above regarding the weaknesses.  I find the observation of forward transfer in prompts to be novel.  The observation that catastrophic forgetting is not an issue with prompt engineering (as opposed to finetuning) seems not novel to me, as it's not really a contribution of this paper but more a contribution of the entire prompt engineering body of work prior to this paper.
summary_of_the_review: I do not currently recommend the paper for publishing because its observations are presented as a solution to catastrophic forgetting, which does not seem to be a problem in the prompt-engineering paradigm.  

---

Update post rebuttal: The authors have sufficiently addressed my concerns, and I have updated my score to an accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new prompting based method for continual learning for language models. The method learns a sequnce of task specific prompts which allows it to overcome the forgetting problem for long sequences. The methods achieve state of the art performance on long range continual learning tasks.
strength_and_weaknesses: Strength
The method proposed in the paper is an efficient way to continually learn language tasks, requiring to train a very small number of parameters as compared to the overall model sizes (< 0.1%). The method is quite simple and intuitive, and is explained quite well in the paper. 

Weakness
There are a couple of questions that the paper raises:
- Section 2.3 mentions the framework that the identity of each task is known a priori. Instead of concatenating the prompt vectors over time, how would the following baseline perform: the prompt has only one theta_p. update \theta_p for each task starting from the previous one as initialization? If this alrternative is nearly as good, this might save a lot fof time at inference.
- Another alternative is to directly train a prompt of size m \times |\theta_p| instead of one sequential vector per task. How would this perform as a baseline?
- For the results in Section 5, can you also provide numbers for per task fine tuning as a baseline? This would help calibrate how much transfer can you actually get from other tasks.
- In section 5.4, the training procedure to obtain the numbers in Table 4 is modified from the one described before. Why is this the case?
clarity,_quality,_novelty_and_reproducibility: Clarity
The paper is overall well written

Novelty 
The method proposed in the paper is novel, it is a combination of exisiting ideas but the current formulation reduces the memory footprint by a big margin. 

Reproducibility
There is no mention of code release by the authors. 
summary_of_the_review: The paper proposes a new method for continual learning which is intereting. I believe some baseline comparisons are actually missing from their experiment setup which have been pointed out in the weakness section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies continual learning in the NLP domain, focusing on pre-trained language models with transformers. The authors use various techniques to improve the continual learning performance, such as prompt tuning and embedding regularization. The proposed method, Progressive Prompt (ProgPrompt), learns a prompt for each new task. In addition, the base language model is kept frozen while a small number of parameters is added for each task. Since prompts and per-task expanded parameters are separate, this allows for reducing catastrophic forgetting. The paper then empirically shows the effectiveness of ProgPrompt for BERT and T5 models on various NLP benchmarks.
strength_and_weaknesses: **Strengths** 


- The paper is well-written and easy to follow. The idea of prompt tuning is also intuitive.


- The proposed method shows strong performance.




**Weaknesses**  


- One major of the work is the assumption of accessing task identifiers (for separating prompts and task parameters). I believe this assumption (similar to the majority of task-incremental works), is very relaxed for practical scenarios. For example, task boundaries and identifiers do not always exist in real-world streaming setups.



- In addition, although very small, the computational costs grow as more and more tasks arrive. This is also the case for classical expansion-based CL algorithms and, at some point (e.g., hundreds of tasks), not negligible. 



- The main focus of the paper is on the final average performance (e.g., accuracy, F1) of the models/algorithms. However, it is important to compare continual learning methods from different perspectives that measure the evolution of the accuracies and forward and backward transfers.  




**Questions** 


- Could the authors provide a more fine-grained analysis of the methods on at least one benchmark (e.g., with many tasks)? For instance, how the average accuracy evolves as each task is learned, how much forgetting (if any) we have, etc. ?


- Does ProgPrompt require task identifiers at the test time?


- Could the author provide more fine-grained performance measures 


- As explained in Section 3, ProgPrompt allocates different prompts and parameters for each task. Could the authors provide the total parameters/time/computation comparison of ProgPrompt vs. other methods for the experiments with a large number of tasks?




clarity,_quality,_novelty_and_reproducibility: 
**Clarity & Quality:** Overall, the paper is well-written. However, I find the methodology section very brief and a few important details missing. 


**Novelty**: The majority of the fundamental ideas, such as prompt tuning, progressive training, and embedding reparametrization, are borrowed from previous works. However, I do not find this a major concern. 


**Reproducibility**: The authors explained most of their experimental setup.



summary_of_the_review: **Initial**  
Overall, while I find the proposed method intuitive with strong results, I believe many details are missing from the paper. In addition, I find the assumption of accessing task-identifiers very relaxed. While I mentioned from the novelty perspective, the work heavily relies on previous ideas, I do not find this a major limitation.

**Update**  
I would like to thank the authors for their response. While the proposed method still has a limitation regarding the task-boundary assumption, given the clarifications and new results, I increased the score, and I encourage the authors to add the full result to the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper focuses on continual learning in LLMs using prompting (in-context learning)

Strength:
- The writing seems very good that makes paper easy to follow.
- The forgetting can be alleviated as the prompt for previous tasks are unchanged.
- Transfer of knowledge from the previous task to new task is achieved by the new approach.
- strong empirical results
- The novel observation in this paper, that Progressive Prompting enables forward transfer is interesting.

Weakness:
-  The task-boundary assumption is a limitation of the paper
- The main focus of the paper is on the final average performance (e.g., accuracy, F1) of the models/algorithms. However, it is important to compare continual learning methods from different perspectives that measure the evolution of the accuracies and forward and backward transfers.


justification_for_why_not_higher_score: The paper is the first step towards doing continual learning in LLMs and although interesting does not fully solve the problem or provide new insights in general for CL. The assumption of having access to task boundaries is a limitation.
justification_for_why_not_lower_score: See strength section of the meta review
Decision: Accept: poster