Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper deals with the problem of hyper-parameters optimization in model-based reinforcement learning (MBRL) Specifically, the paper tries to answer the question of how to schedule hyperparameters such as real data ratio, model training frequency, policy training iteration, and rollout length to achieve better performance. Supplementals materials are provided. 

There are 2 contributions in the paper:
- The paper provides some analysis to show that real data ratios affect the performance of MBRL. Specifically, they show that increasing the real data ratio improves the performance of MBRL.
- They propose a new MBRL algorithm that learns to schedule hyper-parameters on the fly. 

main_review: Main Strengths: 
- The paper makes an effort to analyze the effect of real data ratio on the performance of MBRL. The analysis is based on the one by Munos and Szepesvári but the author did a good job in modifying the proof in the mixed data setting. 
- The author proposes a hyper-parameters tuning method based on Reinforcement Learning. Although the idea is not new, the author did a good job by choosing the appropriate MDP setup. 
- The experiments show good results of their proposed method. 
- Overall, the paper is well written. The way the authors present the paper is easy to follow. 

Main Weaknesses: 
- There is one problem with the comment that the author made about their bound. Since Nreal = N · |A| · β, the ratio β/Nreal  is a constant. Thus it is not clear how gradually increasing the real ratio β is promising to achieve good performance according to the upper bound as what the authors claim.
- The proposed algorithm and the theoretical analysis do not have any relation. The analysis only shows that the real data ratio plays an important role in MBRL. Besides, the theory does not provide any hints in the design of the proposed algorithm. However, the authors claim that their algorithm is inspired by the theory, which is not totally correct in my humble opinion.
- The idea of using RL to tune hyper-parameters is not new and so not a novel contribution. Intuitively, the paper might get trapped in a no-ending route. In detail, the purpose of the paper is to improve MBRL by dynamically tuning hyper-parameters. However, the authors again propose another RL algorithm to achieve this purpose. How are you going to efficiently tune the hyper-parameters of this new RL algorithm? Please let me know if I miss something here. 

Question: Have you tried your model on Humanoid and if so what was the result? 

limitations_and_societal_impact: The authors did not provide limitations and societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work seeks to bring some clarity the hyperparameter management of Dyna-style deep RL (i.e. MBPO). 
The main parameter of interest being the real data to synthetic data ratio.
It does this through theoretical analysis which is followed up by an automl framework in the style of 'RL for RL' (RoR). 
Training an outer policy to schedule policy performs better than standard MBPO across several tasks.
main_review: The MBPO algorithm and DYNA-style RL algorithms are very popular at the moment, so I can see this work being of interest to the deep RL community. 
The outer policy does seem to have a widespread improvement on standard MBPO in the mean and the theoretical analysis looks correct. 

I see this paper more as an analysis into hyperparameters rather than a practical MBRL algorithm.
The need to train a PPO outer loop to me throws away all the sample efficiency that MBRL is supposed to provide, and judging form the hyperparameters in the appendix the full algorithm requires ~1e6 samples that model-free RL usually requires. 
I would be useful to see the 'full' training curves (including the outer loop) to see who effective the outer RL is.

The paper often refers to this method as 'simple yet effective'. I would disagree that a bilevel deep RL problem is 'simple'.
The phrase 'simple yet effective' also a suggests that its a surprise that this method works, whereas for me it is more a question of whether the extensive computational requirements are worthwhile for the task in hand. 

It is also discussed that the empirical results verify that the theory is correct and that it is optimal to slowly increase the data percentage from low to high. However, in the experiements it looks like the initial value of the ratio is set to 0.05 (Table 4), so the policy appears biased to start low.  I think a more uniform initial exploration is required before that claim can be made. 

A rather specific observation I would like some clarification on: on HopperBullet the ratio quickly goes to 1.0, which suggests that MBPO should behave the same as SAC(20) within 10% of training (Fig 3), yet in the performance (Fig 2) AutoMBPO is worse that SAC(20) for the first ~10% but then is better for the remaining 90%. Shouldn't the performance be the same?   

Given the amount of computation required to run this bi-level optimizaiton, it seems the performance increase is not really statistically significant apart from on WalkerBullet. I think some discussion of the statistical significance of the results (since requires so much more computation) would benefit the reader. 

Regarding the baselines, I would appreciate more of a discussion in the appendix regarding how PBT and RoR are implemented as baselines. 

Bayesian optimization is a popular method for HPO. I wonder if, rather that optimizing a neural network policy, you could define a 'schedule spline' (or some similar parameterization) that is only a few parameters but covers a sensible space of possible schedules. 

Minor
line 4 of algorithm 1, typo on target
limitations_and_societal_impact: No issues
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper is about learning a hyper-parameter setter for some of the hyper-parameters used to train a model-based reinforcement learning algorithm. Theoretical results are provided to show that there exists an optimal ratio of real-data to model-generated data during training. Experiments show that setting hyper-parameters with a parametric policy improves performance over manual hyper-parameter selection or hand-crafted schedules.
main_review: The paper is good at conveying its ideas. The introduction section clearly states what we are going to be dealing with. The related work section does a good job of summarizing the different ways in which models have been used in MBRL.

I am not sure if the theoretical analysis should be placed in such a prominent place, given that it is somewhat dense with notation. Perhaps the results of the analysis can be stated earlier on, or be highlighted more from the rest of this section, such that the first-time reader does not immediately have to confront the math right at the stage when they are most eager to find out about the implementation of the ideas presented in the introduction.

The methods section does a good job of setting up the hyper-MDP. However I have some questions about the execution:

* Algorithm 1 starts with "initialize MBPO". Does that mean the model and policy parameters are re-initialized periodically during the lifetime of the whole algorithm?
* The advantage calculation uses the average return of MBPO trained with the same amount of real data from original parameters. Does this mean using AutoMBPO requires logs from a previous MBPO training run (or a separate process training plain MBPO in parallel and providing such logs)?

The experiments section is well-designed. The comparative results are in favor of the method. The hyper-parameter schedules and the ablation study over the hyper-parameters are both welcome inclusions.

That being said, some points are unclear to me:

* A little confused by the experiments. Does AutoMBPO here mean an experiment where both MBPO and its hyper-controller are being trained in tandem, or has AutoMBPO been trained in advance, and only being applied here? If the two are being trained jointly, shouldn't we see periodic drops in performance, since MBPO gets re-initialized every time a new hyper-episode is started, according to Algorithm 1?
* In case we are indeed training MBPO and its hyper-controller in parallel, I am a little surprised that the scheduling curves in fig. 3 can make sense throughout. I'd expect the hyper-controller to make errors in the beginning of training. Can you comment on this?
* How were the hyper-parameters set for plain MBPO? Is it the result of a hyper-parameter search, or carried over from previous work? If they are carried over, I would be curious about the improvement of AutoMBPO over a near-optimal fixed hyper-parameter setup found by a hyper-parameter search. This is especially true of the bullet-based environments which MBPO was not originally applied to.

All in all, this is a good paper and it would benefit the research community to accept it.

**Other comments**

* Do the theoretical results extend to stochastic transition functions?
* Is it right to refer to the hyper-parameter setting problem as an MDP, given that the state does not fully describe all of the information in the system? I think POMDP would be more accurate here.

limitations_and_societal_impact: This is alright.
review_title: null

Meta-review: Reviewers agree that the paper is well-motivated, well-written and the the proposed method, motivated by theoretical analysis, shows convincing experimental results on hyperparameter tuning of model-based RL, a hard yet important problem to address. On the other hand, minor concerns remain mainly about whether the theory is connected well with the practice, whether another outside RL loop over model-based RL is "too heavy" in practice and requires high sample complexity.  Authors have address many of the concerns raised by the reviewers and I am happy to accept this paper. 

Decision: Accept (Poster)