Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a replay buffer strategy to help model-based RL models effectively adapt to local changes in the environment. When local changes happen, the traditional first-in-first-out (FIFO) data buffer will interfere with the model training due to the out-of-data samples still being stored in the buffer. To fix this, the authors proposed to update the buffer by only replacing the data in the local neighborhood of the newly observed samples. Contrastive learning method is used to learn the representation of states and to force consecutive states to have smaller Euclidean distance, thus neighborhood can be determined by applying a threshold on the distance measure.
strength_and_weaknesses: ### Strength
The proposed method is easy to understand and efficient. Experiments showed expected results.

### Weakness
- Major
    I doubt whether the Local Change Adaptation (LoCA) is a suitable testbed for the proposed purpose. In phase 2, the agent is trapped around the T1 area, thus bringing a sharp distribution shift in the collected data during this phase. Data collected during this phase have a very poor state coverage, which hurts the world model learning severely. This designed small local change in the environment results in a huge shift in data distribution, which is a significant effect for MBRL that learns a policy during imagination. A more realistic case would be the case where only the reward function changes, while the agent movement not being limited in any way. 
- Minor
    - Could the authors explain or provide experiment results analyzing the sensitivity of the hyper-param, $D_{local}$ and $N_{local}$ used in the experiments?
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. Based on the details provided in Appendix, I think the results can be reproduced with reasonable effort.
summary_of_the_review: This paper proposed a replay buffer strategy to help model-based RL models effectively adapt to local changes in the environment. However, the extreme cases designed in the experiments can not provide a general use for the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a simple method for adaptation of model-based RL agent to new environments. Starting from the replay buffer filled with transitions from the old environment, the method works by gradually adding in the new transition from the new environment while removing data that are close to the states in the new transitions. In this way, the old data near the newly explored states in the new environment would not negatively hurt the learning of the world model while the old experience far away could still provide a good prior for the world model in the unknown states to prevent catastrophic forgetting. The closeness between two states required in this method is obtained by contrastive learning on a set of transitions collected by a random behavior policy in the new environment. The paper shows that the proposed method can improve upon the standard first-in-first-out (FIFO) replay buffer of different size on top of various existing model-based RL approaches in multiple custom tasks.
strength_and_weaknesses: *Strength*
- The proposed method is conceptually simple and can be combined with existing model-based RL methods 
- The paper is well written with great clarity on the proposed method that seems reproducible. 

*Weaknesses*
- I have doubts over the difficulties of the tasks tested in the paper. In particular, it is unclear to me that whether agents need to use the knowledge from the Phase 1 (environment with old reward function) to perform well in Phase 2 (environment with new reward function) at all. I cannot find the performance of training the agent from scratch directly in Phase 2 (e.g., in Figure 3 and Figure 5). 
- The related work section is very terse and there are not many discussions how this work is similar or different from prior approaches in the continual learning literature (e.g., [1, 2, 3]). Some of them could be valid baselines. For example, instead of training the world model on the full replay buffer, we can train on a small subset of the most informative data from the Phase 1 environment (e.g., coreset [2]), or even discard the data from the Phase 1 environment completely and fine-tune the world model with parameter regularization on the data from the new environment only (e.g., EWC [1]).
- In the last paragraph, the authors wrote that "this is the first time–to the best of our knowledge that adaptivity for the deep MBRL methods has been shown." -- I think this statement is incorrect (see [4])

[1] Kirkpatrick, James, et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.
[2] Nguyen, Cuong V., et al. "Variational continual learning." arXiv preprint arXiv:1710.10628 (2017).
[3] Rolnick, David, et al. "Experience replay for continual learning." Advances in Neural Information Processing Systems 32 (2019).
[4] Nagabandi, Anusha, Chelsea Finn, and Sergey Levine. "Deep online learning via meta-learning: Continual adaptation for model-based RL." arXiv preprint arXiv:1812.07671 (2018).
clarity,_quality,_novelty_and_reproducibility: I found that the biggest quality issue of the paper is lacking proper empirical evaluations to demonstrate the usefulness of the approach in practice:
- In the LoCA setup, the only change between the two tasks (from Phase 1 to Phase 2) is the reward function. A possible (simple to implement, potentially strong) baseline would be to keep dynamics model (e.g., $P(s'|s, a)$) learned from Phase 1 and fine tune it in Phase 2, and train the reward prediction from scratch in Phase 2. This would allow all the data from the Phase 1 about the states to be still correct while discarding all the potentially incorrect reward information. On the flip side, I don't see any conceptual hurdles that prevent the proposed approach from working in the setup where both rewards and dynamics change (though I could be missing some details that left me with this wrong impression). Generalizing the LoCA setup to allow both dynamics and reward changes might make the paper more appealing as it solves a more general problem. 
- In Figure 3, I found it to be a bit odd that the baselines used in the comparison are different across tasks. For example, the authors plotted against FIFO with only very large replay buffer sizes (3e6 and 4.5e6) in MountainCarLoCA, but used a wide range of replay buffer sizes in MiniGridLoCA (ranging from 2.5e4 to 1.8e6). Although the authors did mention that 4.5e6 worked the best for MountainCarLoCA, it is unclear what other replay buffer sizes have been attempted before for the MountainCarLoCA. 
- It is possible that I am missing some detailed text, but I could not find the replay buffer size description for the Reacher experiments (for FIFO baseline). Was the best replay buffer size being reported in Figure 5? What replay buffer sizes have been considered?
summary_of_the_review: While the proposed adaptation approach is conceptually simple and the idea of local forgetting is appealing, it is unclear to me if the proposed approach is practically useful since the paper is missing the baseline of simply retraining a new RL agent from scratch in the new environment (Phase 2). In addition, the paper is missing important discussions and comparisons of the proposed method in relation to existing continual methods (which could serve as potentially strong baselines). Therefore, I would not recommend for acceptance of the paper at its current state. 

A set of more thorough experiments with harder tasks where Phase 2 is difficult to solve without the experience of Phase 1 could potentially make the paper much stronger.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies model-based reinforcement learning in the context of changing environments. More specifically, the algorithms are evaluated in the Local Change Adaptation (LoCA) setting. Well known methods like Dreamer struggle in this setting. The paper proposes a simple method that modifies the first-in-first-out rule of the replay buffer and instead removes old data if it is very similar to a newly added state. The similarity is measured by a function learned through contrastive learning. The proposed method shows improved results over the baseline.
strength_and_weaknesses: The paper is very well and clearly written. The proposed idea is rather simple - which is not a negative aspect - and makes conceptually sense to solve the studied problem. It is shown that the proposed method increases performance in the environments used for evaluation. However, as also mentioned by the authors, the environments are rather simple and it is not clear if the results would transfer to more interesting environments. Currently, the similarity function is trained on random trajectories collected at the beginning of the training. This will not work for more complicated environments, where most parts of the state space have not been observed in the random trajectories. This point limits the significance of the work. Further, while domain adaptation is an important topic it is only studied in the context of one specific benchmark that accounts for scenarios of domain adaptation but not for others.


Are the random trajectories used to train the similarity function counted as environment steps in the shown plots? 

Minor things:
p.5 the 5th to last row has an unnecessary space before a comma 
I find it irritating that there is textof the main manuscript between Fig.2 and Fig. 3. I suggest the Figures should be just underneath each other. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and executed well. The idea is simple but interesting for the studied problem.
summary_of_the_review: The proposed method is interesting and the quality of the paper is very good. However, the limited evaluation makes the paper less significant. Still, it is a good first step and hence the paper is for me just over the acceptance threshold.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a replay buffer update strategy for adaptive RL. The strategy is conceptually simple and makes sense: instead of adopting a FIFO update rule to the replay buffer, the authors propose to replace samples closest to the newly observed samples based on some measure of similarity. Empirically, the paper demonstrates benefits on the LoCo environments proposed in recent work.

On the positive side, the reviewers appreciated the clarity of the work and the fact that the proposed method is conceptually simple and achieves the expected results. However, there were uniform concerns among the reviewers on the significance of the LoCo environments for more practical settings and consequently, how engineered is the proposed update rule for these environments. To this end, the paper could be improved on two major fronts: (a) experiments on conventional benchmarks without the LoCo variant --- the proposed strategy is presented as a plug-and-play since it only modifies the update protocols for the replay buffer, so it should be relatively easy to test (b) more baselines for updating replay buffers, e.g., prioritized/attentive experience replay --- the authors state these techniques as complementary, but without experimental evidence, their combined effects are far from guaranteed to add up. As it stands, the LoCo environments should inspire model-based RL techniques that can transfer to realistic settings rather than ones that can solve only the LoCo environments in isolation. With the above set of experiments, we expect to better understand the significance of the proposed buffer update rule in more realistic settings.  I encourage the authors to follow up on these suggestions for a stronger future version.
justification_for_why_not_higher_score: Proposed technique seems engineered for a somewhat restrictive set of evaluation environments. Needs more extensive evaluation to justify broader utility.
Decision: Reject