Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper defines the problem of lifelong few-shot language learning (LFLL) where the goal is to continuously learn a model with new few-shot tasks without forgetting the previous ones. Towards this problem, the paper introduces a prompt tuning-based framework that augments the pre-trained language model (T5) with continuous prompts (trainable prompt embeddings). Prompts are simultaneously optimized for task solving and data generation. During continual learning of the new domains, pseudo-examples of the previously seen domains are generated for episodic rehearsal and further KL-based consistency regularization is implemented to prevent drifting of the model. Lastly, for new tasks, separate prompts are appended with the existing prompts to enable the transfer of knowledge from previous domains/ tasks. The main contribution of this paper is to showcase the effectiveness of the recently proposed prompt tuning-based method for lifelong language learning. Empirically, this paper reports superior results over EWC and MAS methods when evaluated on text classification, NER, and summarization tasks.
main_review: Strengths:

- The paper attempts to solve an important problem of LFLL in the context of the pre-trained Transformers. With the rise of larger models, it is important to consider the scalability of these models for lifelong learning. Given that this paper builds upon the recent developments in parameter-efficient transfer learning, the number of parameters in the proposed prompt tuning-based method would not grow dramatically with the number of tasks.
- Empirical results showcase that the proposed approach is superior to existing methods like EWC and MAS when evaluated on the sequence of 2-4 domains/tasks.


Weaknesses:

- For the LFLL problem, this paper proposes to use an existing prompt tuning-based transfer learning method where the underlying model is frozen and a few additional task/domain-specific parameters are learned. Furthermore, it repurposes existing lifelong learning methods like deep generative replay [1], distillation-based methods [2], dynamically expandable network [3] as a unified framework when applied to prompt tuning. Thus, the contributions of this paper are marginally novel/ significant.
- Another major limitation of this paper is that it completely ignores comparison with other parameter-efficient transfer learning methods like adapters [4]. Concretely, AdapterFusion [3] work proposes to learn a task-specific composition of adapters from previous tasks. How does the prompt tuning-based method compare with adapter fusion when applied to the LFLL problem? Is there a consensus that prompt tuning-based methods are better than adapters for few-shot learning? 
- By definition, the lifelong learning paradigm deals with a large number of tasks in sequence. However, the paper evaluates the proposed approach on the sequence of 2-4 tasks. So it is unclear whether the gains persist as we increase the number of tasks to 10-15 (towards a more realistic evaluation setup). As we increase the number of tasks, does it enable positive forward/ backward transfer or increase the chance of negative transfer?
- Overall, the paper is written in good style. However, there are key experimental details that are missing. This paper defines prompts as a series of tunable tokens but then it is unclear how many tokens are defined per task prompt. This is important because as we scale the number of tasks, the input sequence length scales linearly, and with that self-attention scales quadratically. This issue limits the practical applicability of the prompt tuning-based method for lifelong learning.
- What does it mean to use validation sets for hyper-parameter selection? Ideally, during lifelong learning, we do not know the sequence of tasks apriori. It is unfair to freeze a particular task order, then search for optimal hyper-parameters, and then report results on multiple runs with the same task order. At least, report the results on unseen task orders? Or search on a subset of disjoint tasks [5]. The paper should consider updating their results with a fair hyper-parameter selection strategy.

Additional clarification question(s):
- The paper trains a single prompt for both task solving and data generation. It is unclear whether this is an optimal strategy when it comes to generating pseudo-samples? Did the paper try learning a separate prompt for task solving and data generation? Does it help in addressing the quality of generated pseudo-samples? Furthermore, how effective is the current pseudo-sample generation strategy? How many samples are discarded to get valid samples?
- There are a few hyper-parameters that are set without any justification: 16-shot learning? How does the method perform with varying the number of training examples? How does the performance change with varying the number of pseudo-samples? Does it hurt if we have more pseudo-samples (generating them is feasible and cheaper)?
- In terms of the multi-task experiments (MT-PT), do we have separate prompts or a single prompt being trained with all data? If it is a single prompt then is it fair to compare it with LFPT5 with more capacity? If there are multiple prompts then how does one ensure that there are dedicated prompts for each task?

[1] Sun, Fan-Keng, Cheng-Hao Ho, and Hung-Yi Lee. "LAMOL: LAnguage MOdeling for Lifelong Language Learning." International Conference on Learning Representations. 2020.

[2] Li, Zhizhong, and Derek Hoiem. "Learning without forgetting." IEEE transactions on pattern analysis and machine intelligence 40.12 (2017): 2935-2947.

[3] Pfeiffer, Jonas, et al. "AdapterFusion: Non-Destructive Task Composition for Transfer Learning." Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021. 

[4] Houlsby, Neil, et al. "Parameter-efficient transfer learning for NLP." International Conference on Machine Learning. PMLR, 2019.

[5] Chaudhry, Arslan, et al. "Efficient Lifelong Learning with A-GEM." International Conference on Learning Representations. 2018.

summary_of_the_review: Overall, I rate this paper marginally below the acceptance threshold. It is interesting to see the efficacy of prompt tuning for lifelong language learning. However, with the current experiments, it is unclear how they compare with adapters? Also given this paper unifies existing approaches and adapts them for prompt tuning setup, it is marginally significant/ novel. There are several open questions (see weakness section + additional clarification questions) that need to be answered before understanding the overall benefits of the proposed framework. I am looking forward to the author's response in the rebuttal period and will consider updating my scores accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to leverage prompt tuning on large pre-trained language models to achieve Lifelong Few-shot Language Learning (LFLL). Lifelong learning and few-shot learning have been considered different ML paradigms. The authors argue that with the emergence of modern large-scale pre-trained language models, AI models now have the capability to achieve these 2 at the same time. This work attempts to formally define the LFLL problem and benchmarked a strong pre-trained LM (T5) on 3 tasks (NER, text classification, and summarization) over 9 different datasets (defined as domains in LFLL).
main_review: 
=================================
Strengths:
  - Combining lifelong and few-shot learning is a new setting.
  - Experiments contain 3 different tasks and each has datasets from different domains.
  - Experiments are well-designed: many baselines are implemented to compare proposed method with traditional lifelong learning methods.
  - Paper is well-written.

=================================
Weaknesses:
  - The actual method is simple combination of existing ideas.
  - The number of tasks and domains is minimal setting. To really become a benchmark to measure the progress of LFLL, more tasks/datasets will be needed.


=================================
Additional Suggestions/Questions:
  - Question Answering and NLI are also very important NLP problems and many datasets exists from different domains. Perhaps the authors can consider adding them to setup a more comprehensive benchmark.
  - Perhaps adding more pre-trained LMs such as GPT-2 and different sizes of T-5. The community will be interested how the model type / scale affect the LFLL capability.
  - How will the pseudo data generation amount affect the learning / forgetting performance? Since the data is generated by T-5, couldn't we generate as much as we want? Maybe adding another ablation on this would be a good idea.



summary_of_the_review: Overall I'm leaning positive about this paper. The proposed LFLL problem, in my opinion, is very important for the community to think about: how to adapt large-scale pre-trained LM to different tasks/domains with few samples? This paper sets up a strong baseline and benchmark for research along this direction. Although prompt tuning, KL regularization, and pseudo data generation are existing ideas, combining them and formulate this LFLL problem/benchmark is non-trivial contribution. I think this work has potential, and if the authors can try to add more tasks/datasets/baselines, it may create more impact to the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper extended the use of prompt tuning to the scenario of life-long language learning. To prevent catastrophic forgetting, the paper proposes to generate pseudo samples as part of the training data when adapting to a new domain, and adding new set of prompt tokens when adapting to a new task. Using the T5 model, the paper demonstrated the effectiveness of the proposed methods using text classification, named entity recognition (NER), and summarization tasks. The method achieved better results for these tasks using the proposed life-long learning mechanisms, compared to fine-tuning, prompt tuning, and their combinations with regularization-based life-long learning methods such as elastic weight consolidation (EWC) and memory-aware synapses (MAS).
main_review: The paper has several strengths that make it a promising submission:

1. The idea of using prompt-tuning for life-long language learning is a promising direction and the methods proposed in the paper constitute a novel and effective way in bridging the gap between these 2 methodologies. The discussion on few-shot learning also clearly demonstrated the difference of the proposed methods and a major collection of prior work using prompt-tuning.

2. The methods proposed in the paper are effective in utilizing the assumption of language modeling, such that tasks are formed as prompt-label pair sequences, and autoregressive sequence generation is used to anchor the model to previously learnt knowledge.

3. The experimental setup is representative of actual uses of life-long language learning, with 3 different tasks included. Comparisons to fine-tuning, prompt tuning, and their combinations with elastic weight consolidation (EWC) and memory-aware synapses (MAS) are solid choices to demonstrate the effectiveness of the proposed methods. In most of these comparisons, the proposed methods achieved better results.

There are a few weaknesses that I hope the paper can address to improve the scoring:

1. In the paragraph below equation 4, the loss form is an addition of 3 values: 1) the task loss L^{task}; 2) the language modeling loss L^{lm}; 3) label consistency loss L^{KL}. Both 1) and 2) are reasonable, but 3) is new in the paper and it is unclear why it is absolutely necessary. In particular, the paper should include an ablation study on a range of \lambda_{KL}, not just a comparison between models with and without L^{KL}.

2. Some statistics that are useful for understanding the propose methodology are missing in the experimental section: 1) it is probably beneficial to include some numbers on the state-of-the-art results on methods for the original tasks and datasets. They may not be fair statistics in a life-long scenario, but they are helpful for the readers to understand where the methods stand in a larger picture of machine learning methods. 2) For the domain order experiments in section 5, it could be useful to also include accuracy for each of the datasets separately. This will provide the readers with additional knowledge on the relationship between task ordering and individual task performance.

Finally, some suggestions to the paper, but these will not affect the scoring:

1. Remove the quote in page 1.

2. The paper should include a brief introduction to the 2 regularization-based life-long learning methods that it compares with. These are elastic weight consolidation (EWC) and memory-aware synapses (MAS).

3. Try to reduce the use of abbreviations in tables. For example, by typing fine-tuning and prompt-tuning instead of FT and PT, it could make the table much easier to read without the need to refer to the paper text.
summary_of_the_review: Good novelty in extending prompt-tuning to life-long language learning using the generative properties of language models. Good experiments demonstrating the effectiveness of the methods in preventing catastrophic forgetting. Some parts of the method are not sufficiently studied in the experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper explores lifelong few-shot language learning. Their framework trains the model as a task solver and data generator. They use pseudo data for new domains, and additional prompt embeddings for new tasks.
main_review: 
Strengths
- Few-shot setting and lifelong learning are important settings.
- I can't give any more strengths at this time because I had a hard time understanding the paper.

Weaknesses (reasons to reject)
- The main contribution of saying that LFPT5 "can be seen as a vital step towards general language intelligence" is surely an overclaim, right? If I ask 100 researchers, how many of them will see this paper as a vital step towards general language intelligence?
- I did not understand why the proposed method is restricted to the few-shot setting. Do gains go away in the full data setting?
- The experiments are OK but not probably not good enough to support the general claims about lifelong learning. E.g., it would be great to show more than 3 tasks.
- I had a hard time understanding the paper, see below (and I will remove this comment from reasons to reject once it is clarified):
	- The key part that I did not understand is why prompt tuning isn't already a sufficient life-long learning method. My understanding of prompt tuning is that no model weights are modified, only the prompt is optimized, and optimized prompts are stored for each task. Then given a new task, a new prompt can be trained and stored. This works because the size of prompts is many orders of magnitude smaller than the size of the model. Maybe another way of asking this question is, how is the following different from regular prompt tuning: "while adapting to a new task type, LFPT5 includes and tunes additional prompt embeddings for the new task"?
	- Another part I do not understand is why pseudo samples need to be generated. Why can't you just use the prior examples?
	- I am missing why this idea of retraining on previous domains each time is a good one. Let's say we're up to 1000 tasks. How much weight does the new task get in this mixture? Won't this take a long time?

Weaknesses (not reasons to reject)
- The introduction could make it more clear why the second two limitations of LLL are indeed limitations. Why NER in particular, instead of other tasks that haven't been explored thoroughly? The negative transfer point needs some more evidence as well, as I believe the decaNLP challenge showed positive transfer, right?

Minor comments
- Is the finetuned BERT-Large a few-shot finetuned or with the full dataset?
- PT is not defined at first use in the abstract
- All the acronyms are pretty confusing
- Typo: review socre 5
summary_of_the_review: This paper tackles an important task. I didn't understand why their method specifically applies to the few-shot scenario. I also struggled to understand why the proposed method solves lifelong learning over regular prompt tuning. One sentence in the paper seems like an overclaim to me.
review_title: null

Meta-review: This work defines the new problem of lifelong few-shot language learning where the goal is to continually learn new few-shot tasks and use those to benefit future tasks while not forgetting previous tasks. With larger models, this is an important goal due to the cost of updating and retraining these models. The work also shows superiority to existing approaches like EWC and MAS. After the author's rebuttal, the experimental section is also thorough with evaluation on a good range of tasks and approaches such as adapters showing good results. While this setting appears simpler than the full lifelong-learning setting and the approach combines existing ideas, this work's contribution to the definition and thinking about this problem is valuable. However, the authors should more clearly state the advantages of their approach vs standard prompt tuning (with an emphasis of benefiting future tasks) since two reviewers seem caught up on this point. The other two reviewers comments were addressed by the rebuttal as they stated in their comments.
Decision: Accept (Poster)