Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: A benchmark for continual reinforcement learning is provided, where the main task contains 20 subtasks. A certain number of choices are made such as the agent being aware of task change, not carrying forward the replay buffer, and learning policies with separate output heads per task. The evaluation was overall performance, forgetting, and forward transfer. Seven learning methods are compared extensively and results are provided.

main_review: The setup is not satisfying but it is also true that finding a continual learning benchmark acceptable is extremely hard. As a first start, I think this benchmark can be acceptable and an interesting place to start.

There is an emphasis that the different goals in continual learning are conflicting. However, that is neither a surprise nor unrealistic. Animals face the same tradeoffs, and these are exactly why continual learning is so challenging.

It seems the focus on computational and memory resource capacity is forgone. In the experiments and comparison, it was not taken into account. Different algorithms took different amounts of computations and memory, which were not described. If they took widely different amounts of computations, it is not clear that aligning their performance on a per time step basis for comparison is fair anymore.

In Table 1, all forgetting numbers are positive. Equation (3) suggests they should be negative. If that is not the case, then can you clarify how this forgetting number is calculated? And, does 0.73 for fine-tuning mean a high level of forgetting?

It is not clear what is meant by regularizing critic. It is also not clear whether the critic is being learned from scratch or not or whether it depends on the method.

*** post rebuttal ***

Thanks for the clarification.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new benchmark for continual reinforcement learning. It modifies the Meta-World robotics benchmark that’s designed for meta- and multi-task RL evaluation by learning the tasks in sequence rather than together. This work is very timely as there lacks a common benchmark for continual/lifelong RL.

The benchmark consists of 10 tasks from Meta-World. To evaluate catastrophic forgetting, these 10 tasks are repeated for a sequence of length 20. For quicker iteration, there are also different triplets of tasks where the middle task might inhibit transfer to the last task.
main_review: Originality: 
- Continual World modifies the evaluation protocol of the existing Meta-World benchmark. 

- As noted by the authors, there is already work that re-purposes Meta-World for the continual RL setting and evaluates various algorithms on it [33]. Additionally, [33] performs the evaluation on the full set of 50 tasks, while Continual World only looks at 10 tasks.

Quality:
- The 10 tasks in Continual World were carefully selected so that there is a good range in the difficulty of transfer between these tasks. They also include a transfer matrix, which can aid researchers when selecting the ordering of the tasks and understanding which tasks facilitate easier or harder transfer.

- They also evaluate a diverse range of continual learning methods on the benchmark. Interestingly, continuous fine-tuning gives the best forward transfer performance, but even that is far below the reference forward transfer value which is the highest transfer value from a single source task.

- Other methods I’d be curious to see the forward transfer performance of are: progressive neural networks and meta-RL methods adapted to the online setting.

Clarity:
- The paper is overall well-written and organized. The related work section also provides a thorough survey of the existing benchmarks. 

Significance:
- A benchmark for continual RL is highly relevant to the NeurIPS community as there is no commonly used benchmark yet.

- The forward transfer results, when compared to the forgetting scores, really highlight transfer as an important direction to study in the context of continual RL.

Questions: 
- In Table 1, is the Performance column reporting the success rate?
limitations_and_societal_impact: Yes. See main review for other limitations.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Summary. The paper proposes a robotic benchmark for continual reinforcement learning built on top of meta-world benchmark. It analyzes the difficulty in the problem of continual RL (such as catastrophic forgetting) and proposes metrics (average performance, forward transfer, forgetting) to measure performance of continual RL algorithms. Additionally, the paper evaluates existing continual RL algorithms on their benchmark and discusses the limitations of the proposed benchmark in detail as well.
main_review: Originality: The paper doesn’t propose any new method but rather presents a new robotic continual RL benchmark.  

Quality: The paper appropriately analyzes the proposed benchmark, evaluates existing methods on it and discusses the limitations of their proposed benchmark in detail. I do think that absence of visual continual RL tasks makes the benchmark limiting as it inhibits study of interesting forms of transfer involving visual features.

Clarity: The paper is clearly written.

Significance: The paper is important because it provides a robotic benchmark for continual RL along with evaluating existing methods on the proposed benchmark which makes it easier to test new continual RL algorithms.
limitations_and_societal_impact: It's not discussed in the paper
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a suite of continuous control tasks for studying continual reinforcement learning. The paper includes a comprehensive study evaluating prior continual learning methods, both the ones designed for RL and for other tasks. It also highlights the challenges of applying existing RL methods to continual learning setting.
main_review: Key contribution
- A suite of continuous control tasks for continual reinforcement learning (CRL) (built on Meta World tasks)
- Implementations of existing continual learning (CL) methods, both the ones designed for RL and other tasks --- require non trivial adaptation.
- A fairly comprehensive study using existing methods on the proposed benchmark.
- Highlights CRL-related challenges (Sec. 6.1) --- 1) existing actor-critic methods may not be the best for CRL settings and 2) we may need to rethink the standard RL training procedure, i.e., exploration and how to populate replay buffers, in the CL setting.

Comments
- From my understanding, the key selling point of the proposed benchmark is “more realistic robotics tasks”, but practically, I’m not sure if they are “realistic” in the sense that they are still individual tasks with clear assumptions about task boundaries. Learning is harder for sure, but I’m not sure about realism.
- In addition, I'm not sure about why the previous supervised CL benchmarks “are not geared towards measuring forward transfer” (line 65). As far as I could tell, the forward transfer measures the learning speed / sample efficiency as a result of transferring the knowledge from the previous task. It would seem natural to also measure the sample efficiency as a form of forward transfer in supervised CL?
- It would be interesting to see what the harder tasks bring to the table. Ideally, the paper should conduct a small-scale experiment and apply the same methods on “easier” tasks such as MNIST sequence and see if the conclusion still holds, since the main selling point is that the easier tasks cannot show the challenge of “more realistic settings”
- Triplet experiment: ideally PackNet should not be affected by the prior tasks (i.e., task B) since it isolates parameters for each task, but it doesn’t seem to be the case in this experiment. More analysis would be helpful.


Minor:
- Fig 1 no axis
- Including Multi-task learning + PopArt in Fig 3 as a curve is somewhat confusing, since the training curve means an entirely different thing here (learning all tasks simultaneously). Better include the asymptotic performance as a horizontal line?

limitations_and_societal_impact: The "limitation" section is adequate for technical contributions. It'd be interesting to see a short discussion on societal impact of studying the continual RL setting.
review_title: null

Meta-review: I think reviewers were generally satisfied that this was a useful contribution.  I found the reviews generally fair and the paper itself generally meritorious.  Obviously there are some concerns, mostly related to the inherent difficulty of producing a satisfactory continual learning benchmark.  This paper is somewhere near the current pareto-frontier of the tradeoff between "doing something useful but hard" and "doing something in a completely satisfactory way".   I don't completely know if that frontier is close enough to the absolute quality threshold for NeurIPS, but I tend to think it is.  
Decision: Accept (Poster)