Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a meta-RL algorithm that is focused in particular on the learning of sequential tasks without revisiting previously seen tasks. This approach, as stated in the paper, could be particularly beneficial in real-world applications and robotics where it might be unfeasible to revisit previously seen tasks. The paper builds on a successful meta-RL algorithm (MAML) and appears to leverage on the use of off-policy methods to replay previous data from previous policy without re-engaging with older tasks. 

main_review: My main observation is that the paper reveals possible novelty and innovations very late and unclearly. The abstract explains the problem that the algorithm attempts to solve, but reveals no insight about how this happens, and thus does not provide any concrete claim. This issue continues in the introduction where the contribution part focuses on what problem is being solved, but not how. What are the main scientific and technological innovation that are introduced with respect to existing approaches? Eventually, section 4 (CoMPS overview) enters in the description of the proposed algorithm, but even the following sections 4.1 and 4.2 do not clearly state what specific novel aspects are introduced with respect to existing methods, and which are re-used or adapted, which makes it very difficult to appreciate the novel contributions and originality of the work. In section 4.3, it is mentioned that the proposed method uses "both an improtance-sampled policy gradient estimator and an importance-sampled value estimate for the baseline in the policy gradients” and it claims that the following ablation study demonstrate it’s importance. Is this one of the major contribution of the paper? 

Similarly, the idea that the novel algorithm uses an alternation of off-policy (for the inner loop) and imitation learning (for the outer loop) transpires without sufficient clarity and related claims. An interesting hypothesis, e.g. is mentioned in section 4.2 : " In contrast to methods that are concerned with forgetting, the parameters produced by this meta-RL training can quickly learn new behaviors that are similar to the high-value policies from previous tasks and, if enough prior tasks have been seen, likely generalize to quickly learn new tasks as well. However, this is not sufficiently followed up in the experimental analysis. How does this mechanism affect performance in the various tasks? What can we learn from the fact that CoMPS performs better than the baselines on the first three tasks, but it's comparable to PPO+TL in the arguably more challenging MetaWorld tasks? 

summary_of_the_review: In summary, the paper appears to have solid basis in the literature and tackles an important problem in meta-RL, and it appears to propose an interesting combination of known approaches to solve the stated problem of sequential-task meta-RL without task repetition. The experimental results are encouraging. However, it fails to make clear claims or clearly explain the key novel aspects of the algorithm, how and why they are sufficiently novel.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method that incorporates fast adaptation into continual learning. The main challenge is that previous tasks cannot be revisited during continual learning, and the policy has to be trained on limited offline data. The authors address this challenge by utilizing importance-sampling policy gradient and value estimators in the inner loop and behavior cloning in the outer loop, which enables meta-training with fully offline data. Experimental results show that their method achieves superior forward transfer compared to prior methods.
main_review: ## Merits

1.	This paper studies the problem of continual Meta-RL, which is important and deserves more attention from the community.

2.	The general idea of the proposed CoMPS method is sensible and conceptually simple for addressing the challenges of continual Meta-RL. CoMPS meta-learns on previously seen tasks to acquire a good parameter initialization that fast adapts to new tasks. These adapted policies in turn collect data for meta-learning. The main challenge is that previous tasks cannot be revisited during continual learning, and the policy has to be trained on limited offline data. This challenge is addressed by utilizing importance-sampling policy gradient and off-policy value estimator in the inner loop and behavior cloning in the outer loop. The whole framework is reasonable, and, in some sense, CoMPS can be viewed as an approximate implementation of the FTML algorithm [1].

[1] Finn, Chelsea, et al. "Online meta-learning." International Conference on Machine Learning. PMLR, 2019.

3.	Experimental results look impressive and show that the proposed method achieves superior performance in several sequences of continuous control tasks.

4.	This paper is generally well-written and structured.

## Limitations and Concerns

1.	One major concern is that the technical contribution of the proposed method is very limited. CoMPS is a simple adaptation from the previous method GMPS by using an off-policy variant of PPO in the inner-loop.

2.	Another concern is that CoMPS may not handle well for some badly-learned tasks, as its outer loop objective uses behavior cloning. If previous tasks are not learned well, and the skilled experience dataset $D^*$ contains bad trajectories, the meta-learning procedure may learn an initialization that quickly adapts to imitate the bad trajectories, which harms learning new tasks.

3.	Backward transfer: although the authors claim that they do not evaluate backward transfer or forgetting, it is one of the core challenges in the field of continual learning. It is interesting to see how CoMPS performs on previous tasks, which may contribute to a deeper understanding of CoMPS.

4.	Related works: most related works have been adequately discussed. However, several works incorporating meta-learning into continual supervised learning are missing [2,3]. A discussion over these works should be added.
[2] Joseph, K. J., and Vineeth N. Balasubramanian. "Meta-consolidation for continual learning." arXiv preprint arXiv:2010.00352 (2020).
[3] Gupta, Gunshi, Karmesh Yadav, and Liam Paull. "La-maml: Look-ahead meta-learning for continual learning." arXiv preprint arXiv:2007.13904 (2020).

5.	Experimental details: environment descriptions of Ant Direction and MetaWorld in the caption of Figure 5 do not seem to correspond to Appendix A, which is confusing.

6.	Other minor things: at the start of each meta-learning procedure (Algorithm 1), how is $\theta$ initialized? Do you use the $\theta$ achieved at the end of the last meta-learning procedure, or do you use the parameters updated after the RL procedure?

summary_of_the_review: Overall, this paper is an interesting step towards solving the problem of continual Meta-RL. Its presentation is clear, and the proposed general framework is reasonable. One major concern is its novelty, which resembles the previous approach GMPS. Although several key limitations have been identified in the discussions of this paper, addressing some of them is important and can significantly strengthen this paper, particularly requiring the data storage of previous tasks and no mechanism for handling forgetting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new method for continual meta-reinforcement learning called CoMPS. The objective is to quickly achieve a high reward over any sequence of tasks even in non-stationary task distributions. Unlike previous methods in meta-RL, CoMPS uses a hybrid meta-RL approach, where experiences from past tasks are used to learn a fast adaptation procedure (using a meta-RL objective with off-policy inner loop updates and BC outer loop updates). The model is then adapted to any new task using a standard on-policy RL algorithm like PPO. The authors show that this approach outperforms generic meta-learning and transfer learning approaches across various environments for the specific settings considered in the paper.
main_review: **Strengths**

1. The paper presents a novel, potentially useful setting/objective for the meta-RL/transfer learning communities to focus on, which in some sense bridges objectives pursued by the meta-RL and transfer learning communities.

2. The paper also presents a novel approach to tackle the corresponding problem which would act as a strong baseline for methods tackling the said problem.

3. The paper is also really well written with clear diagrams and algorithm figures which make it very easy to read. 

4. I also liked some of the ablations and the choice of baselines as they provide a nice peek at what some of the critical components of the model were that made it click. 

5. The paper also does a nice job of tuning the baseline hyperparameters to make sure the results are competitive. I would suggest maybe moving some of those details from Appendix D to the main paper as it might serve as a nice reference for how to perform a fair and competitive evaluation of the baselines!

**Weaknesses**

1. I would like to see more ablations. It was unclear to me from reading the methods section as to why certain design decisions were taken : for example, 
    * Why use an on-policy method for online adaptation instead of an off-policy method here as well given you are storing the experiences anyway? 
    * Why the specific choice of off-policy method over so many others?
    * What happens when you replace BC with another offline/off policy RL method? Etc.
 
    In essence I want to understand how much each of the design decisions contribute to the overall performance and which ones were more important than others.

2. I would also like some analysis on why the authors think PPO+TL was able to perform as well as CoMPS (or slightly better) on the meta-world non-stationary tasks. It does seem like a surprising result to me and I think some analysis or discussion there would have been nice! Some hypotheses I have here, for example, are that, maybe the specific non-stationarity ends up providing a kind of curriculum to PPO -TL? Maybe the off-policy updates have too much variance? Maybe the trajectories used for BC update aren't good enough in these types of problems? etc..

3. Related to the point above, I would also like some discussion on the broad class of scenarios where the approach presented in the paper could provide a genuine boost over approaches like PPO+TL, and scenarios where it’s uncertain and why the uncertainty exists (perhaps due to some empirical or numerical issues with the algorithm presented?)

**Other comments**

1. It is also not entirely clear to me if the scenario or motivations presented in the paper for real world applications of the continual RL setting are reasonable? For example, for the home cleaning robot example, it’s hard to imagine that you would ever actually deploy a robot in the real home to collect experiences and learn using RL directly in the sequential setup. You would probably either have some sort of sim2real setup or offline experiences collected in some other (safer?) way used to train the model. I don’t hold this as a major weakness of the work, but I think it would be nice to provide motivations which are more grounded. For example, the disassembling example given at the end of the appendix seems more reasonable to me and might be useful to think along those lines instead!

2. I would also be curious to know how effective the meta-learning objective would be for learning to explore in the new environment when trying to learn the new task (something like “Rothfuss, Jonas, et al. "Promp: Proximal meta-policy search." arXiv preprint arXiv:1810.06784 (2018).”).

3. It might also be a good idea to slightly change how the confidence intervals are computed using suggestions from https://www.google.com/url?q=https://ojs.aaai.org/index.php/AAAI/article/view/11694&sa=D&source=docs&ust=1635372130978000&usg=AOvVaw1BsO05shZBrpwhKwk76dSN (say bootstrap etc)

4. typos/grammatical errors
    * at the same time still adapt to ->  at the same time adapt to
    * rehersal-based methods ->  rehearsal-based methods 
    * and the implementation are given ->  and the implementation is given
    * utilize both an improtance-sampled -> utilize both an importance-sampled
    * furthest from previously chosen location -> furthest from the previously chosen location 
    * we provide futher analysis -> we provide further analysis
    * behavior of CoMPS is statistical different -> behavior of CoMPS is statistically different
    * due to the increasing difficult of tasks -> due to the increasing difficulty of tasks
    * accelerates acquisition of new tasks -> accelerates the acquisition of new tasks
summary_of_the_review: I like the paper overall, it is clearly written and I think it tries to address an interesting problem. I would however, have liked to see some more ablations and analysis of the approaches and maybe some better motivations too.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers  the challenges in sequential multi-task learning, in which the agent will visit all the tasks in a sequence, and will not revisit the previously learned task. This paper proposed a continual meta-RL framework, named Contunual Meta-Policy Search (CoMPS), so as to extend the traditional meta RL framework, GMPS(Guided Meta Policy Search) to a continual Meta-RL setting. Different with GMPS, CoMPS firstly adopts self-imitation learning as the outer optimization instead of a true imitation learning and secondly adopts an importance sampling-based policy gradient for off-policy inner optimization, which is suitable for the continual Meta-RL. The empirical results show that CoMPS can outperform the prior continual learning or meta learning methods on both stationary and non-stationary task distributions.
main_review: strengths:
- this paper considers a more realistic and difficult scenario, continual Meta-RL learning, and proposed an extended meta-rl framework that adapts this setting;
- the experiments are sufficient and the performance improvment is significant;
- the writing is good and key details are clearly described.

weakness:
- This paper can be viewed as a combination of several existing techiniques. In particularly, this paper claims that it addressed a type of continual learning problem but did not consider one of the main characteristic of contual learning - i.e., the forgetting issue. Hence a reasonable baseline to use the past experience is the offline RL learning instead of consider it as a meta learning problem. Overall，it looks to me that the whole method is a kind of  MAML + off-policy RL + online RL. 

- though the paper claims that this is the first to  formulate and address the continual meta-learning problem, this problem setting has already been discussed before, some work can be referred to:
[1] Online Meta Learning. Chelsea Finn, et.al.
[2] Continuous adaptation via meta-learning in nonstationary and competitive environments. Maruan Al-Shedivat, et.al.

some questions and typos:
1. the inner optimization for each sequential task is actually separate, so why not use a recurrent neural networks structure to avoid the forgetting by utilizing the historical information?
2. the subscripts of both summation and quadrature symbols in Eq.2 have problems.
3. the caption of Figure.7  is wrongly written as '... the average average return across the episodes ...'.
summary_of_the_review: The issues considered and the solutions proposed in this paper sound reasonable, and the proposed method does achieve the better performance. But some details of the whole pipeline need more justification, and the optimization between sequential tasks seems to be independent. I'm also not sure that this is able to address the issue about catastrophic forgetting.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper propose a continual meta-learning setup where the agent is faced with a sequence of tasks and has to adapt quickly to new tasks based on the experience it gathered in previous tasks. In addition, at future iterations the agent can only interact with the previous tasks by doing off-policy learning on the data it collected during its interaction with these tasks and cannot further interact with these previous tasks. The authors then propose an algorithm that combines RL on the current task with offline meta RL on the previous tasks. The authors then construct setups that fit this problem setup, evaluate their method on it and compare to previous baselines. 
main_review: The main novelty of this work is the proposed continual meta RL setup and the design of the experiments. Regarding the algorithm, I find it to be reasonable application of existing ideas to this setup. 
To give a few examples: 

The ablation study indicates that the method performs better by using an existing algorithm for doing off-policy learning. While this is nice to know, it is not very surprising, and we can learn very little from looking at figure 6 about that. A more carful investigation of this question will include an experiment that is centered on the off policy questions in this setup, will report some more statistics (e.g., the IS weigths as the tasks change) and will compare to more baselines. 

The baselines that were chosen are complete methods, that were not specifically designed for this new setup and are not high performing in it for that reason. A much more interesting comparison would be a carful ablation analysis on top of COMPS showing which components make it superior and when. For example, I don't learn a lot from the comparison to PEARL as a whole, and I would appreciate a comparison that uses COMPS with SAC as the RL agent. I am interested to know what is needed in this setup and why, I am less interested why papers that were designed for different setups are being outperformed (while it is good to know that this is the case). 

To summarize, I feel that the paper will have to make stronger algorithmic contributions and more carful ablation study for it to get accepted to ICLR. In this form, I would recommend rejecting it. 
summary_of_the_review: Interesting meta RL setup and nice experiment construction. At the same time, the algorithmic novelty is questionable and the ablation study is not convincing enough.
review_title: null

Meta-review: This paper develops a novel continual meta-reinforcement learning algorithm that focuses on learning sequential tasks without revisiting previous tasks. The setting is compelling, and the method is well-developed with good empirical results. The initial version of the paper included a variety of issues, especially lack of clarity in some aspects and the contributions, that were remedied through discussion with the reviewers and subsequent revisions. The discussion among the reviewers seems to have settled on leaning toward a weak accept overall, with one low score that should be dismissed claiming lack of novelty (which isn't correct - the paper certainly is sufficiently novel).  There do remain some concerns by two reviewers that although "the paper has enough meat to be accepted, ... [it] needs more careful and well thought out ablations and analysis to be truly valuable." Although the authors have revised the paper to address this issue of a precise analysis, adding material into the appendices with some changes to the main text, they are encouraged to make certain that these aspects are integrated and clear throughout.
Decision: Accept (Poster)