Venue: ICLR 2023 Workshop RRL

Reviews:

final_score: rating: 4: Good paper, strong accept
review_content: Summary:
The authors focus on transferring pre-trained decision transformers from MetaWorld to ContinualWorld, in both one-step and continual reinforcement learning settings. They propose a hybrid prompting and modulation-based approach, combining prior works on this topic, which they show yields both good downstream performance and is robust to forgetting.

The paper provides a thorough set of baseline comparisons and ablations. The authors find that directly fine-tuning the model yields good single-task performance on downstream tasks, with performance clearly increasing as more parameters are fine-tuned. However, naive fine-tuning-based approaches lead to very poor performance when training sequentially on ContinualWorld tasks, apparently due to catastrophic forgetting. Fine-tuning only a restricted set of parameters yields both poor downstream performance and significant forgetting.

Relevance and significance: Very relevant to this workshop. The authors directly propose methods to reuse prior computation for expensively pre-trained models to solve new tasks. Good performance and thorough ablations indicate that this is a significant contribution.

Quality and clarity: Good. The paper is well-written and clear, with a very thorough appendix.

Additional feedback:
It might be good, just for the sake of completeness, to include a baseline where the agent is trained simultaneously on MetaWorld and ContinualWorld during fine-tuning. Since DT is inherently an offline algorithm this is fairly well-defined, and re-using buffers of previous tasks is a common technique in continual learning to mitigate catastrophic forgetting. This would be roughly equivalent to the FT-multi-task-pre-trained baseline but with MT-40 data also included.
review_title: Interesting paper, relevant to the workshop

final_score: rating: 4: Good paper, strong accept
review_content: Overall, this is a strong workshop paper on an interesting subject. I think it could be a good full conference paper submission in the future and I think the authors could improve some things for that so I've written more detailed comments.

# paper summary

## overview

- finetuning pretrained RL models isn't as performant as NLP, Vision
    - propose that the issue is [catastrophic] forgetting
- create a toy robot task to demonstrate forgetting in finetuning
- show that continual learning methods (rehearsal, regularization) work here as well

## method and experiments summary

regularization (L2 and EWC)
- L2 loss between pretrained and finetuned weights
- either a standard sum (L2), or EWC weighted

replay (behaviour cloning)
- gather buffer B with pretrained model
- minimize KL on action distribution between pretrained and finetuned models
- *note* mitigating forgetting in critic isn't necessary

first experiment is a toy task, 1D grid env with two tasks in sequence
- model pretrained on task 2 will forget it if task 1 is long enough

second experiment is a robotics task, a sequence of 4 smaller tasks from meta-world, 2 easy then 2 hard
- model pretrained on last two tasks (hard), and must learn the first two without fully forgetting its pretraining
- EWC and BC both help the model retain its pretraining skills and outperform naive finetuning and from-scratch training baselines, L2 is worse somehow

# review

The paper is novel in the connections it makes and I believe it makes a strong and bold claim that these continual learning (CL) problems are what is holding back reincarnated RL. The clarity could use some work as a couple experiments and conclusions are unclear, but the overall significance of this makes it a strong workshop paper. With some improvements, I can see this being a strong conference paper in the future.

What this paper does well is try to bring together the CL and RRL fields. I think a longer related work with equivalents between the two would help. The main robotics experiment does a good job of demonstrating skill forgetting in a practical way, and though it is toy-ish it is hard to dispute the main argument. The CL methods are simple but arguably even more convincing because they are simple and effective. 

The main improvements would be to generally be clearer with what each experiment is testing and more strongly justify the conclusions drawn from each one. There are also some easy improvements to make the experiments more convincing that I suggest below. Finally, the paper is a bit light on overall results, which is fine for a workshop paper, but should definitely be improved for a full conference submission. I think just making the connection between CL and reincarnated RL is interesting but the thing that will sell the paper is justified, actionable, takeaways. The toy experiment feels a bit obvious and I would dedicate less time to them. The robotics experiments are great and do a better job of demonstrating the same points in a "toy-ish" way since the setup is a bit unrealistic (knowingly pretraining on the last two tasks). My personal view is that the paper needs to lean into either demonstrating this issue for a real reincarnated setup (e.g. Agarwal et al already showed BC works for Atari, demonstrating the EWC works too would be a strong contribution) or creating a benchmark CL+reincarnated task (such as the robotics task) and release models / eval protocols to allow future work to test which CL methods are beneficial for reincarnated RL. 

Below are more specific comments for improvement.  


## 3

## 3.1
confusion formulation for compositional MDP
- initially 
    - $\tilde M$ is a subset of $M$ 
    - $\pi_\theta$ is pretrained on $\tilde M$ 
    - the goal is to solve $M$
    - this implies that $\tilde M$ is the easier initial distribution, $M$ is the distribution you want to transfer to
- later
    - getting to $\tilde S$ is not trivial
    - $\pi_\theta$ doesn't manage to reach states $\tilde S$ despite being pretrained on them
    - 

I am not sure that this is necessarily ``compositionality'' since you are not recomposing a set of tasks/skills you learned but rather extending your set of tasks while remembering previous ones. Compositionality in RL is more in line with the recently proposed Zipfian RL than this. I would recommend starting from the idea of temporal splits in your MDP and defining your setup around that with clear language that relates to your toy task setup e.g.
    - MDP $M$ is made up of multiple tasks $m_1 \ldots m_n$ e.g. picking, placing, opening box
    - policy $\theta$ is pretrained on sub-MDP $\tilde M$ that contains only a subset of those tasks e.g. picking, placing
    - the goal is to maintain learned skills on tasks in $\tilde M$ when transferring to $M$ e.g. maintaining picking even if opening box comes earlier in the MDP $M$

## 3.2

$c$ is never defined
- I assume it is the current space the agent is on
- why is it a parameter to the environment if the agent starts at $x = 0$
- why do you have a variable $x$ that seems identical to $c$?
- I would expect $c$ to be a constant if it is a parameter to the env but that would make solving the problem impossible since it needs to know state
- Figure 3a says it sets $c = 0.5$, what does this mean given the previous context?

Deceptive X-axis in Figure 3 (right)
- left and center go to 500 steps
- right goes only to 50 steps
- this is confusing and showing the full 500 would be better
- e.g. it seems like M = 35 solves the full environment but only has a 15% chance of getting to Phase 2 at the end of training

I didn't fully understand why magnitude of $s1 / s2$ will make a big difference

why don't you demonstrate your CL methods on this?

nitpick: just use an affine model and make the state $c$, this is clearer and simpler

## 4

Figures 4 and 5 are great, very clear

If you want to do even better, you might consider randomizing orders of envs and taking average to get an overall score for "best" CL method


## Related Work

There is a substantial work on RL for language models that deals with this problem of continual learning, usually called "alignment tax" or "robustness" in that literature. Specifically, you can see the KL divergence to a pretrained model and re-running on pretrain data (replay) as CL methods, and they are standard in RLHF literature (KL introduced by Jaques et al, standard RLHF paper is Ziegler et al). There is also a smaller literature on "Countering Drift" in emergent communication that tackles a similar problem in RL (e.g. Lee et al). I don't think these works diminish the novelty of yours at all though, since these scenarios are really bandits and use RL for gradient estimation, essentially, whereas you are dealing with full MDPs and possibly more than two tasks (they do LM pretraining and finetune on a single task).

A comparison of CL and RRL methods, showing which ones are the same or have strong similarities would also be great.



review_title: Good Workshop Paper, Improve for Full Submission

Decision: Accept(Oral)