Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The method studies the generalization capabilities of unsupervised RL empirically, and develops a hybrid planner with strong asymptotic performance and high sample efficiency in standard benchmarks. 
strength_and_weaknesses: Strength
* This work offers a large-scale benchmarks for different URL techniques and compare the usefulness of different pre-trained components. 
* The developed method has strong empirical performance. 

Weaknesses
* The main weakness is novelty since most components in this work come from existing works.
* The paper pinpoints two decisions to make to apply the proposed framework for tasks outside URLB in Section 3.4. Within URLB, these decisions are made given empirical benchmarking results. But more insights or explanations of when and why different techniques work in different settings, and how much alignmenent of the pretraining and downstream task the proposed mechanism can handle, will further strengthen the contribution of this work.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. The method itself is built upon existing methods in the literature, but novelty itself is not a limitation given the strong empirical performance. 
summary_of_the_review: The empirical findings from the work are insightful for the community, and the proposed method stemming from these findings achieves strong performance in standard benchmarks. Both are valuable contributions to the community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors study model-based methods for tackling the unsupervised reinforcement learning benchmark (URLB). First, the authors show that substituting DrQ (a model-free agent) with DreamerV2 results in increased performance and scaling on URLB. Second, the authors investigate the transfer of specific components (model, actor, critic) from pre-training to fine-tuning, finding that transferring model and actor is beneficial but additionally transferring critic compromises performance. Third, the authors propose Dyna-MPC, a planning procedure that builds upon model predictive path integral (MPPI) control, using the actor as an additional source of sampled trajectories, and the critic and model to score trajectories. This is shown to further improve fine-tuning. These improvements altogether result in dramatic improvement on URLB. Finally, the authors assess transfer under distribution shifts via the real-world reinforcement learning benchmark (RWRL), and find that their proposed model without Dyna-MPC performs best.
strength_and_weaknesses: Strengths

The results are particularly strong. The progression of experiments building towards the proposed method is well structured and informative. The set of methods, comparisons, and ablations is extensive.

Weaknesses

The gains provided by model-based learning are probably significantly inflated due to the simplicity of the three URLB environments, which enables in-imagination training and planning to actually work. In contrast, in Atari, which has far more visual variation per game despite arguably simpler dynamics, model-based methods have still yet to come close to model-free methods, except when the allowed amount of environment interaction or compute is limited. Thus, while impressive, the improvements coming from this work should be taken with a grain of salt, and might not hold for environments with more realistic amounts of variation and complexity.

The application of model-based learning for unsupervised pre-training for RL has already been known to be effective (e.g. Plan2Explore). I'm not sure why the URLB authors did not benchmark model-based baselines like Plan2Explore, but regardless, the improvements over the DrQ-based runs is expected given Plan2Explore, especially given the environments in Plan2Explore and URLB are both sourced from the DeepMind Control Suite. Regrettably, it seems that the authors formulated their main hypothesis without realizing the above. I recommend rephrasing the contributions in the abstract and intro to convey that the results in this work corroborate and extend upon the findings first provided by Plan2Explore.


clarity,_quality,_novelty_and_reproducibility: Clarity

The writing was very clear and well-structured.

Quality

The execution of the work appears sound. The empirical results are impressive.

Novelty

As the authors admit, the work does not propose a novel method, but rather demonstrates the outsized benefit of applying existing methods to a particular problem setting. As noted above, this benefit is itself not particularly novel, as the point was already previously made in the Plan2Explore work. However, this work does provide quite a few minor novelties over Plan2Explore, including swapping out Dreamer for DreamerV2, assessing using a benchmark not available to Plan2Explore, assessing using a variety of self-supervised reward mechanisms during pre-training, investigating the transfer of various components, the Dyna-MPC planning mechanism, and robustness results on RWRL.

Reproducibility

I'm satisfied with the level of detail provided for reproducibility purposes.
summary_of_the_review: This work is rather tricky to assess. While its proposed methods do provide dramatically improved empirical results on an existing benchmark, as argued above there are important limitations to both the novelty and significance in this improvement. However, the thoroughness with which the empirical evaluation was conducted is admirable, and there are a couple of genuinely new insights provided on the application of model-based learning to unsupervised RL. I currently lean towards rejection, but look forward to discussion with the authors and other members of the reviewing team to inform my final recommendation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies the problem of reward free learning, where the setup involves a pre-training stage (when an intrinsic reward is used based on different prior methods) and a fine-tuning stage (when a particular extrinsic reward is provided). The claim is that model-based methods perform better than model-free methods in such a setup. The authors perform certain ablations where different components of the model-based method (in this case the actor, critic, and dynamics model of the Dreamer algorithm) are shown to affect fine-tuning performance differently. Finally, the authors deploy a MPC style planning algorithm instead of using the learnt actor when fine-tuning to new rewards. 
strength_and_weaknesses: Strengths

- Important and relevant problem setup
- Fairly thorough experiments

Weakness
 
- Lack of a coherent story
- Unclear primary contributions
- Somewhat hard to read through
clarity,_quality,_novelty_and_reproducibility: The paper is unclear to read in some parts, is fairly high quality, lacks quite a bit on novelty, and is decently reproducible. 
summary_of_the_review: The paper lacks a coherent story and it’s hard to parse the clear contributions it makes. The Dyna-MPC algorithm as well as the ablations are not the main contributions in my opinion. As far as the main claim goes, I am not sure why this is surprising, i.e. shouldn’t any model-based method be expected to perform better than model-free method when the reward function is changed? I could not understand why Figure 2 has a flat line for the Drq@100k and Dreamer@100k cases. What does 100k refer to here? Can you provide some commentary on why, in the setup you consider, model-based methods should outperform model-free ones by such a huge margin? I think this is crucial to the reader’s understanding of the contributions and motivation of the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers the task of unsupervised pretraining of the reinforcement learning (RL) agents.
It analyses and compares different design choices for pretraining and fine-tuning the pretrained components; it shows the improvement of using proposed model-based reinforcement learning over existing model-free technique, DrQ.
strength_and_weaknesses: Strengths:
- The paper is really well written: the narrative is shaped around choosing different components for pre-training and fine-tuning, with the structure of the overall approach neatly summarised in Section 3.4
- It contains compelling evidence of benefits of using proposed model-based reinforcement learning method over existing state-of-the-art model free technique 
- The analysis includes necessary justification of the architecture: choice of the best possible pretraining model

Weaknesses:
- The model is engineered out of the existing components (apart from incremental, in a positive sense of this word, contribution of Dyna-MPC); this is not, however, a problem at all in my opinion as we need large-scale studies showing the impact of architectural choices, and this is a good contribution for the community and this very knowledge of how to improve architectural choices is a novel aspect of the paper
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is really well-written, with the overall structure summarised concisely in Section 3.4

Novelty of the work is in a large scale evaluation; although the work is engineered out of the existing components, the reviewer thinks that it is important that the work shows the impact of architectural choices and therefore serves a good contribution.

Comments:
- In figure 2, it appears that while the proposed procedure wins over results from Laskin et al (2021) by a big margin,  the results still keep improving even after 2M frames pretraining; do the authors know what would happen beyond 2M frames, and if there is a point where no more pretraining iterations give improvement in fine-tuning? Does it stabilise, or does it lead to some form of overfitting degrading performance after certain point?
- Section 3.2: what is the impact of finetuning just actor, without fine-tuning the model?
- In Figure 6, is it possible to add the baseline of the proposed method ablation, where there is no pretraining phase, to match up with DrQ @100K? That would help complete the picture of the contribution of pretraining to the overall process. 
-  Did the authors think of using DynaMPC in conjunction with the DrQ model? Is it possible (I guess not because if the world model requirement which is would make such modification model-based), and could it bring a fraction of the proposed benefits?
- In Algorithm 1, it looks like the world model $\phi$ is listed as the input but never explicitly referenced in the algorithm body; could the authors clarify whether this is correct and whether the explicit reference is needed?

summary_of_the_review: Based on the merits of the evaluation and experimental contribution, as well as the quality of writing, I recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper performs an extensive study of model-based approaches for the URLB benchmark. The findings show that replacing a model-free approach with Dreamer V2 improves the performance. The paper performs further analysis by using pre-training or fine-tuning for the model, actor, and critic. The findings of the experiments leads to an approach that provides significant gains on URLB. The paper also demonstrates results on Real-Word RL benchmark to test generalization to noisy environments.

Strengths:
- The experiments are very well structured and thorough.
- The paper is also well written and clearly demonstrates the progression of the experiments towards building the proposed approach.
- The paper shows a massive improvement on the benchmark.

Weaknesses:
- Some of the findings of the paper (e.g., model-based methods outperforming model-free methods when the dynamics of the environment stays the same and reward changes) are expected. 
- The results are limited to URLB, which is simplistic compared to more sophisticated benchmarks used by other unsupervised RL works. So, the findings will probably not generalize. 

justification_for_why_not_higher_score: The reviewers raised concerns regarding the generality of the claims and also the novelty of the claims. The AC considers these concerns as valid and cannot recommend acceptance.
Decision: Reject