Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper evaluates 7 lifelong learning methods from supervised learning for reinforcement learning. It uses a credible, minimal lifelong learning framework based on SAC and a bespoke manipulation benchmark, based on ROBEL D'Claw. 
strengths_and_weaknesses: Strengths
---
* Presents important experiments which establish a baseline for lifelong RL research
* High coverage of different LL methods and techniques
* Considers important LL questions, such as task ordering
* Simple paper which seeks to do one thing well

Weaknesses
---
* Editing and writing is sometimes weak; many sentences are awkward, grammatically incorrect, or not very idiomatic; missing labels on plots; awkward use of whitespace (use the ~ for spacing references, i.e. `Fig.~\ref{fig:my_figure}`!)
* I would like to see some more concise take-aways in the discussion section. As it is, it is quite dense.
summary_of_recommendation: I think evaluative and empirical work is very neglected in the field, and I would love to see more papers like this in the future. 

I support accepting this paper because it contains important data and analysis for authors and reviewers working in lifelong reinforcement learning. The authors are correct that there are precious few resources with any information on the performance of popular LL methods on reinforcement learning, and especially few which do so using robotics challenges. This paper, in its best form (see my comments below and above) would be an asset for those studying LLRL for robotics.

I rated this paper weak accept because I believe it is held back by a few presentation issues (see my comments above). I think this paper could use a lot of attention in the discussion section, to provide readers with a few very clear take-aways from the analysis (and perhaps a summary table, if appropriate). Papers like this have the most impact as a quick reference for future researchers. Additionally, a general once-over on presentation, typesetting, grammar, and style would do wonders for the digestability of this work.
issues: See comments above.

Detailed comments:
Line 95: I don't think it's ever possible for a method (which works well) to be "too simple and straightforward". The best method is no method at all! Please reconsider your phrasing here.

Figure 3: Please provide axis labels!

Table 1: Please have a clear heading for this table, and consistent use of rows/columns.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work provides a framework for evaluating various life long learning techniques in a robotic reinforcement learning context. Several approaches (regularization, memory, gradient, and expansion based) are compared. Each approach must learn a set of 10 knob turning tasks in sequence, and performance on all 10 is recorded afterwards.   The memory and expansion based approaches outperform the regularization and gradient based approaches at the cost of requiring replay buffer storage of past episodes or a growing policy size.
strengths_and_weaknesses: Strengths: 
The evaluation setup from this paper is well motivated and implemented.  It captures a temporal nature of tasks changing over time, different from a multi-task setup where multiple tasks are learned concurrently.  

Weaknesses:
- This work doesn't propose a novel method, only a benchmark and comparison of existing methods. 

Perhaps another interesting dimension in addition to the forgetting of previous tasks would be number of rollouts required to learn a new task to some threshold success rate. For example, one would hope a capable life long learning agent with more past experiences would be able to learn a new task more quickly than an agent with less past experiences to leverage.  
summary_of_recommendation: The framework for a life long learning benchmark is a great idea.  That being said, the benchmark articulated in this work leaves a good bit to be desired. It would be interesting if other dimensions beyond catastrophic forgetting where to be discussed.  For example, looking at how number of rollouts needed to learn a new task changes overtime.  

It is also a bit unclear to me how well a set of 10 very similar tasks captures the requirements needed for a life long learning system. 


issues: 
- Table 1: Could you elaborate on what a negative score means? Is the knob spinning in the opposite direction?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The main contribution of this paper are summarized as follows:
The manuscript proposes a novel framework which could modify traditional lifelong learning algorithms in supervised learning into reinforcement learning.
The manuscript proposes a manipulation benchmark for lifelong learning with ten different manipulation tasks. 
The manuscript evaluates Soft-Actor-Critic with the state-of-the-art lifelong learning algorithms from four different categories using proposed benchmark, and evaluates forgetting performances and robustness to task orders in experiments. In the end, the manuscript analyzes the potential of different lifelong learning algorithms applied in RL based on experiments.

strengths_and_weaknesses: The main advantages of this manuscript are two aspects. First, this manuscript proposes a novel reinforcement lifelong learning framework in order to make reinforcement learning algorithm owns the ability of overcoming catastrophic forgetting for previously learned tasks, and owns the ability of speeding up the convergence rate for new task. Second, this manuscript proposes a manipulation benchmark for lifelong learning with ten different manipulation tasks. 
However, there are some technical problems left at current version, some of which are given as follows.
1) The manuscript states that "Though recent lifelong learning research has promising performances on supervised learning, little evidence has proven that they could directly be applied in robotic RL tasks." in introduction. However, lifelong reinforcement learning has been studied for several years. For example, "Online Multi-Task Learning for Policy Gradient Methods". It would be better to pay more attention in investigation.
2) The novel reinforcement lifelong learning framework in this manuscript is similar with overcoming catastrophic forgetting algorithm framework in "Learning without Forgetting". It would be better to provide more details about difference between novel reinforcement lifelong learning framework and the framework in paper "Learning without Forgetting".
3) It would be better to provide the comparative experiment with more details for novel reinforcement lifelong learning framework and state-of-the-art baselines. Current tables and figures in manuscript could support the advantage of manipulation benchmark for lifelong learning rather than the advantage of performance for proposed novel reinforcement lifelong learning framework.
4) It would be better to unify proper noun. For example, the manuscript states that " Reinforcement Lifelong Learning " in title, and states that " Lifelong Reinforcement Learning " in content.
summary_of_recommendation: The related investigation was not enough. And current tables and figures in manuscript could not support the advantage of performance for proposed novel reinforcement lifelong learning framework.

issues: 1) The novel reinforcement lifelong learning framework in this manuscript is similar with overcoming catastrophic forgetting algorithm framework in "Learning without Forgetting". It would be better to provide more details about difference between novel reinforcement lifelong learning framework and the framework in paper "Learning without Forgetting".
2) It would be better to provide the comparative experiment with more details for novel reinforcement lifelong learning framework and state-of-the-art baselines. Current tables and figures in manuscript could support the advantage of manipulation benchmark for lifelong learning rather than the advantage of performance for proposed novel reinforcement lifelong learning framework.
3) It would be better to unify proper noun. For example, the manuscript states that " Reinforcement Lifelong Learning " in title, and states that " Lifelong Reinforcement Learning " in content.
4) The manuscript states that "Though recent lifelong learning research has promising performances on supervised learning, little evidence has proven that they could directly be applied in robotic RL tasks." in introduction. However, lifelong reinforcement learning has been studied for several years. For example, "Online Multi-Task Learning for Policy Gradient Methods". It would be better to pay more attention in investigation.
review_title: null

Meta-review: **Updated meta-review after rebuttal**
After careful consideration of the reviews/rebuttals and paper revisions, I recommend accept.

This paper is focussed on an empirical evaluation of continual learning approaches applied to RL (continual learning approaches are typically proposed for the supervised learning setting, and not for the RL setting, so we have little insight on how these approaches would fare in the RL setup.) I agree with the reviewer that states that not proposing a novel method is not a good reason for reject (which is the major concern of the other 2 reviewers). The manuscript initially did have some issues on properly portraying the scope of the presented evaluation, but that was fixed.

**Initial meta-review**
**Summary**:
This paper benchmarks lifelong learning algorithms in the reinforcement learning on manipulation tasks. Current SOTA continual learning algorithms are adapted to the reinforcement learning setting, and then evaluated on a sequence of tasks. 

**Strengths**:
Lifelong learning is an important problem of robotics, and this manuscript provides important insights on how current lifelong learning algorithms that were mostly developed for supervised learning problems perform in RL settings.

A good coverage and evaluation of existing methods

**Weaknesses**:

The reviewers point out that the metrics evaluated in this work are limited to measuring forgetting. However, lifelong learning is also about understanding how quickly a task can be (re-)learned given previously learned representations. The authors should consider adding such metrics to their evaluation.

Clarity and crisp take aways are lacking in a few places. The authors should carefully discuss and analyze the results, and improve the clarity of the manuscript

There is prior work on lifelong reinforcement learning (for simulated robotic tasks), and the authors should discuss such work. Given that this benchmark is also done in simulation, the authors should maybe not put too much emphasis on “little prior work for robotic lifelong RL exist” , but put more emphasis on “benchmarking of lifelong RL algorithms has received little attention”


Decision: Accept (Poster)