Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: ​
The authors investigate specific design choices of SAC in affecting transfer efficacy in continual learning on the Continual World benchmark. They then take their findings to achieve state of the art results on Continual World using a relatively simple set of extensions upon SAC.
strengths_and_weaknesses: ## Strengths
​
**Experiment Thoroughness and Analysis:** The authors perform a great job of summarizing their detailed findings intuitively and concisely in the text, and presenting trends across experiments instead of attempting to detail every experiment.
​
**Writing/Clarity:** The paper is written well, and contains a logical flow that makes it easy and pleasing to read.
​
**Motivation and findings:** The paper contributes useful knowledge about continual learning in RL, and is an interesting read and its results should be useful for the community.
​
## Weaknesses
​
**Environment-dependent results:** As the authors allude to in the limitations section of the paper, they demonstrate their results on only the Continual World suite of environments. CW seems to serve as good testbed for the experiments they performed, but the authors should test on at least another continual learning task (e.g Procgen) to show that their findings can also hold in another environment suite (without needing to re-perform most of their testing experiments).
​
**Detailing specific failure cases in the main paper:** The paper results would be more clear if the authors detailed/analyzed some individual specific failure cases in the main paper. For example, why was task 7→9 transfer with the critic so poor in Fig2b?
​
**Minor Issues or comments:** 
​
- L81: [28] analyses → [28] analyzes
- In L183-190, when explaining why the deterioration in forward transfer occurs when reusing the features only, it could be also because there is not enough diverse data when just transferring from one task to another in the experiment the authors perform. This is  in contrast to feature transfer in vision, for example, where pre-training on something like Imagenet allows the network backbone to learn very general image-features.
- It would be nice to have a figure of the CW benchmark, to help contextualize the results, in the main paper.
questions: How do the authors think these results will transfer to different types of observation spaces? E.g. image-based RL?
limitations: Sufficient limitations coverage.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents an empirical evaluation on how to improve knowledge transfer in the continual RL (CRL) setting. The authors consider SAC as the RL algorithm of interest and perform their evaluations on the Continual World benchmark. First, the paper investigates 3 different ways of transferring knowledge obtained by the SAC agent in a 2-task setting. Second, the authors conduct experiments in CRL setting in which they combine SAC with different CL algorithms. They explore different choices for an exploration strategy, data rehearsal and regularising the critic. Finally, they combine all their findings to create a SAC agent which outperforms the competitors.

strengths_and_weaknesses: The paper takes a step in an interesting direction, as knowledge transfer is an important and relatively overlooked property in CL. For the most part, the paper is well written an easy to follow.

In terms of originality, in my opinion, the paper does not introduce novel ideas, but evaluates different combinations of ideas previously found in the literature. The experimental setting and the measurements are also not novel. It would have been helpful if the paper’s claimed contributions were listed in the introduction. Nevertheless, the experimental findings of this paper are interesting and might be beneficial to people doing research in CRL with SAC. I’m not sure how the findings apply to other RL algorithms, aside from SAC, and I did not see a discussion on this. 

Overall, while the findings are interesting, I can’t decide if they are significant enough to warrant a recommendation for acceptance.

questions: q1: Could you briefly list the contributions of this paper?

q2: what knowledge do you think can be transferred between actors? You refer to the expected transfer of feature representation in supervised learning, however, in your setting, the input already provides an abstract representation of the environment.

limitations: The discussion on the limitations of this approach is adequate.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: **Summary:** The paper aims to study different factors that improve transfer in continual learning. It focuses on continual world benchmark and SAC as the underlying algorithm. It concludes the following:

  1. Transfer of critic alone matters more than that of actor and exploration. Furthermore, the benefit (i.e. increase in performance and forward transfer) from transferring multiple components is usually close to the sum of benefits gained when those components are transferred alone.

  2. Regularizing actor loss with BC (using expert data from prior tasks) helps and it’s best to not regularize critic loss

  3. Using previously trained policy (as opposed to random policy) for exploration on new tasks helps

The paper uses these findings to come up with a modified form of SAC named ClonEX-SAC which gives state-of-the-art performance on CW10 and CW20 benchmarks. 

strengths_and_weaknesses: **Strength:**
  1. The experimental analysis in the paper is quite thorough. I like the fact that it uses pair of tasks to come up with initial hypothesis and then tests those hypothesis on CW10/CW20 benchmarks.

  2. The final proposed method nicely follows from the analysis done in previous section and is overall simple while still achieving state of the art performance in CW10/CW20 

**Weaknesses:**
  1. The paper mostly focuses on SAC as the underlying algorithm and CW10/CW20 benchmarks. Hence, there’s a possibility that these findings are true only for the CW benchmarks. I do appreciate the authors being very upfront about the limitations of the work. 

questions: N/A
limitations: The authors were very upfront about the limitations of the work (Section 7) which I appreciated.
review_title: null

Meta-review: metareview: All reviewers appreciated the importance of investigating continual reinforcement learning, and the throughout experiments conducted in the paper. While the paper does not introduce new methods, it evaluates existing methods and offers surprising insights. While the experiments focus on a single algorithm and environment suite, these insights are still valuable to the community. For these reasons, I recommend acceptance.
confidence: Certain
Decision: Accept