Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper describes meta-method and metrics for solving iterated batch reinforcement learning problems. The paper concludes with experiments on an "acrobot" simulation. 
main_review: Strengths
1. The iterated batch RL motivation is interesting
2. The scope of ablations is above average

Weaknesses
1. While I find the first paragraph or two of the introduction to be enjoyable, the informal, chatty approach to writing the rest of the paper detracts from the content. As an example: "Our main contribution is a Dyna-style GUIDE&EXPLORE strategy (Fig 2). The gist is to learn a guide policy ξ using a model-free RL technique on the model p and on the traces collected on the real system T". This is the main contribution! What is a guide, what is Dyna-style, why are we talking about a "gist"? Ordinarily I would avoid commenting on writing style in a review, but it must be addressed.

2. Further, if we are to understand the above comment as the main contribution, the issue is that "the guide" is a concept, not a new concept at that (just a renaming), and this paper doesn't clearly state a hypothesis for any particular implementation.  The paper spends a section addressing terminology in the description of the guide we are told that sometimes the guide is trained on data from the model of the system (mostly in fact), but also sometimes ("when possible") trained with off policy data.

3. Again in the "terminology section" the following appears: "In that sense, through the data from the **real system**, planning is part of training the guide". It appears that here the paper refers to the OpenAI Gym as the **real system**, the paper should clarify that a better experimental setup would include a real, real system, a simulator of the real system, and a learned model of the real system. Others have done physical acrobot experiments (c.f. https://arxiv.org/pdf/1210.0888.pdf, http://youtu.be/FeCwtvrD76I)

4. Why do NaNs appear in some of the tables?

5. What is the contribution? I see the list at the end of the introduction, but none are testable/falsifiable statements.
summary_of_the_review: This paper explored a variety of approaches for iterated batch reinforcement learning. There is no obvious contribution other than the demonstration that experimentally both planning and exploration are important to improving the performance of Acrobot policies.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors study model-based reinforcement learning applied in the scenario of iterated batch reinforcement learning. There exist other namings of the studied in the paper scenario of iterated batch reinforcement learning, I found out that there exist several other synonymous names of such setting: growing batch RL/ micro-data RL / micro-data model-based RL. The authors motivate the problem informally using several bullet points listed in the introduction.

The main finding claimed by the authors is that the Dyna-style MBRL approach is an appropriate choice for the studied setting by enhancing it with decision-time planning based on a model-free policy and dynamically tuning exploration. 

The algorithms are tested in the problem of resource-limited Acrobot; a detailed ablation study is provided for algorithm features (called here interchangeable algorithmic bricks for iterative batch RL. The performed ablation study helped to find the combination of a neural model, a bootstrapping DQN guide, and a heating explorer, which led to a 10% jump in state-of-the-art on the resource-limited Acrobot system. 
 

main_review: The studied problem on an approach to sample-efficient MBRL is clearly a very significant problem in modern research on RL. The authors have nice ideas, they enhance the traditional MBRL approach with two models (called the guide and the explorer), the guide is based on a Dyna-style policy, applying of which is known to be challenging in a long horizon setting. The paper is nicely written and easy to follow. I think that the designed methodology has a significant potential of being useful in the research on MBRL, but only after the presentation and experimental evaluation are mature enough.

I would like that the researchers address my concerns listed below and related mostly to correct mathematical formalization of results, and validating the methodology in a more broad set of benchmarks and environments.

### 1. Lack of proper formalization of the setting.
After reading the studied problem statement in the introduction, it is still unclear for me what precisely can be characterized using the term ‘iterated batch RL’ coined by the authors. In particular, why it is necessary to define and name an MBRL study subgenre, rather than use the existing framework of online/offline MBRL. 
I appreciate the informal and visual presentation of the studied setting using the example of John, a telecommunication engineer. However, I would appreciate a formal statement and assumptions behind the studied iterated batch RL setting. Especially that it has been studied in the literature under different names. This critique extends further, in my impression the paper lacks mathematical rigor necessary in the setting of such a problem like acrobot (authors showing) being extremely sensitive to: the formulation of observation vector (sincos vs pure angles), initial condition (instability and chaotic behavior), employed model (probabilistic / multivariate / stochastic /deterministic).  

### 2. Disputable significance of the reported improvement over the state-of-the-art in a single benchmark environment
One of the main selling points of the paper is the 10% improvement for the resource-limited acrobot (in terms of Mean Asymptotic Reward after the fixed number of system access steps. Basing the paper contribution on an improvement like that for a  single benchmark can be misleading. First, the issue of ensuring a sufficient number of runs with different random seeds seems to be addressed by averaging the result from 10 different runs (standard in RL literature) and reporting confidence intervals. However, the 10% improvement may be the result of a simple hyperparameter optimization.  It is now known that exhaustive hyperparameter optimization in MBRL may result in significant performance gains, and even solving for exploitative solutions, see B. Zhang et. al. _“On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning”_ AISTATS 2021.  Therefore, it is not unexpected that an exhaustive hyper-parameter search could significantly boost the MAR/MRCP scores of the other evaluated algorithms and change the final leaderboard. Table 3 given in the appendix indicates that the authors did not, in fact, perform an exhaustive hyperparameter search. Another concern is related to the fact that the DynaZero benchmark algorithm achieves higher MAR, and when hyper-tuned could become also the clear leader considering the other utilized metrics.

### 3. Not clear motivation for choosing the Acrobot system as the studied benchmark
As for the motivation of choosing the Acrobot system as the benchmark, it is not clear at all. Page 8 starts with the sentence _“Acrobot is a small but relatively difficult and fascinating system, so it is an ideal benchmark for
continuous-reward engineering systems”_. But I am not convinced, there is only a single other reference given providing the previous benchmark result K´egl et al. (2021). I would like to see more support for the claim that the Acrobot is the ideal system to study in the studied setting with adequate references.  A relation to classical optimal control theory would be a convincing example, and it could also serve as a baseline. 

Nonetheless, even if the Acrobot system is an ideal benchmark it seems to be simplistic when compared to the usual continuous robotics RL benchmark environments studied in literature like Hopper or Half-Cheetah. I see significant differences between the Acrobot and the usual RL benchmark environments - the Acrobot does not require to deal with nonlinear forces generated by friction and contacts with the scene & other objects like in the other continuous RL benchmark envs. 

### Other remarks:

* What is the ‘heating explorer’ used in the abstract,
* Please clarify the terminology used for the studied setting, I found in the paper and related literature several names, I assume, for the same problem:  iterated batch RL/ growing batch RL/ micro-data RL / micro-data model-based RL. I know that the field is new, but this naming discrepancy can become confusing in the long run.
* Please clarify what does precisely NaN reported in Table 2 is supposed to mean ?
* p.7, fig.3, l.5: I do not see how to interpret the notation with brackets T^{(i^*)}[1,2]?

summary_of_the_review: I am on the fence regarding the significance of the paper results. On one hand, it adds new ideas and establishes an interesting line of research. Eventually has a significant potential of being useful in the research on MBRL. 

However, the presentation of the paper is not mature enough. I think the paper has a few significant drawbacks including: lacks a proper mathematical formalization of the studied problem, and validation within a varied set of benchmarks and different scenarios. I explained my concerns in three paragraphs above. Therefore, I am leaning towards its rejection. An improved version of the paper may have a major impact.

More adequate score is 4.5
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents extensions and modular additions to dyna-style algorithms, especially involving bootstrapping and improved exploration.  These changes are studies on the classical, but difficult, Acrobot domain and it is shown empirically that the changes improve convergence speed and asymptotic performance. 
main_review: The paper empirically shows that the proposed changes involving exploration, bootstrapped returns, and other additions improve performance of dyna algorithms in Acrobot.  However, this seems like a limited result as currently presented.  Why were these extensions not studies in other environments, especially more realistic ones like those described in the introduction?  The paper spends a lot of space describing hypothetical industrial applications but never presents domains of that kind.  In addition, the changes proposed here, especially the automated exploration techniques are not theoretically investigated.  Are there any guarantees about the convergence of Dyna+these extensions in the tabular case?  

The main contribution of the paper are the empirical results on Acrobot.  But in the main paper, only a few algorithms are actually shown in Table 2 and Figure 5 (for instance where are the XXX variants?).  And the overall results in Figure 5 show virtually no difference in the asymptotic behavior of the different variants and only a small difference in convergence speed, which seems within the margin of error for the two planning variants and even comparing to current state of the art baseline (purple) there seems to be overlap by the error bars shown. I also question the use of a single percentage (70% here) in calculating the MRCP.  Ideally one would calculate convergence rate for several values, say 70%, 80%, and 90%, to make sure that convergence is proceeding at a fast pace throughout the learning process.  With only a single domain studied and marginal gains I think more studies are needed here.

The paper also does not provide any theoretical analysis of how the proposed changes will affect convergence.  Does the new “heating” exploration guarantee the optimal policy is preserved?  Is it guaranteed to converge in the limit?  What is the expected sample complexity of using this approach?  

And what about the approach described on Page 6 where the planner “dynamically chooses the right amount of randomization” for exploration?  How is that actually implemented?  Is the randomization part of the available action space?  What kind of convergence guarantees are available there?  Page 9 claims that the paper shows “allowing the planner to choose the right amount is a robust and safe approach” but I do not really see evidence of that nor a concrete algorithmic description of how that procedure operates.
Lack of theory and concrete description of the hyper-parameter-less exploration

Organizationally, the paper seems unfocussed. The first page is spent on hypothetical business cases that the paper never returns to.  The related work section lists many different dyna-style algorithms but it is hard to find most of them in the actual algorithm results. I also found the pseudocode blocks quite hard to follow when comparing the different algorithm variants.  A better way to convey this information would be using a matrix where each row has an algorithm name and the columns describe the exploration type, the planner used, etc.

summary_of_the_review: The new algorithm is interesting and seems to make some improvements over some of the existing approaches, but the gains seem marginal, some variants re not shown in the main results, and the reported statistics could use improvement.  there is also no theoretical justification or proofs provided for these changes,
review_title: null

Meta-review: The paper studies dyna-style MBRL in a resource-limited setting. It is evaluated on an acrobat task where it shows very promising results.

The reviewers appreciated the extensive replies, but they did not fundamentally change their opinion. In particular:
- Lack of formal problem statement and definitions
- The experiment on a single task (and that being a non-standard version) isn't sufficient to demonstrate the general merits of the method

While the ideas are very promising, the paper cannot be published in its current form. We'd hence like to highly encourage the authors to revise the paper and to re-submit at a different venue.
Decision: Reject