Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This algorithm proposes an algorithm for iterated batch reinforcement learning. The algorithm uses model-free RL to learn a guide policy, and then uses decision-time planning to improve the policy. The decision-time planning uses some exploration method and a rollout procedure to get a good action.
strength_and_weaknesses: Strength:

The paper is overall well-written and studies an important problem.

Weakness:

The paper is very similar to the AlphaZero algorithm by Silver et al. (2017), therefore, I don't think this paper has enough novelty.
The guide policy is similar to the prior policy in AlphaZero as the authors mentioned in Section 3.2. Instead of learning the guide during MCTS training, this paper learns the guide using simpler model-free RL algorithms.
The decision-time planning is also similar to AlphaZero. In fact, AlphaZero uses a more advanced MCTS technique whereas this paper uses a simple rollout procedure and chooses the action that leads to the best return. In other words, this paper only uses one step of MCTS.
Therefore, I think this paper is a simple modification of the AlphaZero algorithm and thus it is not novel enough as an ICLR paper.
clarity,_quality,_novelty_and_reproducibility: The paper is overall clearly written and easy to follow. The experiments are conducted using simple agents in simple environments, so I think it is likely that the experiments can be reproduced.
summary_of_the_review: The paper lacks novelty since it is very similar to AlphaZero, and in fact, it is almost a simplified version of AlphaZero. The experiments are performed only in simple environments. Therefore, I don't think this paper makes a significant contribution to the community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper explores approaches to iterative batch reinforcement learning, whereby relatively low-dimensional data is collected from physical systems and relatively simple actions/decisions must be made in response to these observations. This includes consideration of how the agent may only be given limited real-world decision making experience, but potentially more access to off-policy data. The paper shows that a novel "Guide&Explore" strategy achieves better performance than the baselines on two standard tasks: acrobot and cartpole.
strength_and_weaknesses: # Strengths
## Novelty
- The heating explore based on using a set of different temperatures to create more exploratory policies seems novel and effective
## Significance
- The general classes of real-world problems considered by this paper are widespread, and this is an interesting investigation into using deep reinforcement learning to tackle them
## Clarity
- The algorithms and results figures are clear and informative
- The text is well edited and makes good use of references to prior works; both of these are well appreciated

# Weaknesses
## Novelty
- Novel algorithms and/or algorithmic changes are not clearly marked, making it harder to identify what is novel
## Significance
- It is hard to see how performance on cartpole and acrobot relate at all to the motivating example of a telecommunications engineer; cartpole and acrobot do not seem to reflect the considerations listed in section 1
- This approach seems to rely on determining the optimal temperature set for exploration -- it is not clear how one could feasibly identify a temperature set to use for novel problems
- There are no new theorems or proofs presented in this paper to rigorously explain why these problems are difficult or unsuited to other deep learning methods, or why the proposed methods are better
## Clarity
- The problems discussed in the first half of the paper at a high level and in vague terms, making it unclear what this paper is precisely trying to contribute or how its insights can/should be used by future works
clarity,_quality,_novelty_and_reproducibility: This paper is well written and has interesting ideas and insights but I do not believe ICLR is the right venue it in its current form. The contributions listed in subsection 1.1 are very vague and high-level compared to other works that appear in ICLR. Guide&Explore is the main novel algorithm presented, but the classical annealing algorithm it uses, augmented with a temperature set rather than a single temperature, does not seem significantly novel or well justified -- while there is discussion on how guide policies tend to underexplore, there is no theory based justification or discussion on why this type of heating method is the right change to make. In fact, it seems like learning the right set of temperatures to use for a given problem would require hyperparameter tuning and ultimately be very sample inefficient -- methods to identify the temperature set should be discussed in the paper. It is also unclear what specific kinds of problems this paper is specifically trying to address; the case studies on cartpole and acrobot seem vastly different than the types of use cases discussed initially. I think the missing piece is a rigorous link between the complexities discussed in section 1, the model formulation in 3.3, and the experimental setup in section 4.

I believe this paper requires significant revisions to rigorously define and characterize the specific classes of problems it is trying to address and to create a mathematical formulation that provides hard insights into the challenges of the problems considered. With this, the experiments could be better justified as being well suited for evaluating algorithms in such tasks. The paper also requires a more rigorous comparison to relevant state-of-the-art methods beyond DQN and SAC. For example, after reading the introduction I was expecting a comparison against imitation learning methods, since the premise seems to focus on problems wherein a human operator had spent significant time performing the task and thus there would be a huge amount of data to learn from.

As an aside, I think the telecommunications engineer example in the beginning is entertaining and relatable but could be greatly abbreviated for this short conference-style format.

The paper includes a reproducibility statement that suggests the results will be easily reproducible with released code, which is well appreciated.
summary_of_the_review: While the paper presents an interesting look at using and modifying RL methods for iterated batch RL problems that have some similarities to real world intelligent control problems, it does not yet have the precise and rigorous problem setup and mathematical theory-backed analysis expected for ICLR papers in this area. It could also use more experimental results ideally exploring more diverse test problems and more state-of-the-art baselines (e.g. in imitation learning and other fields).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed an efficient guide and exploration strategy to improve the performance of iterated batch RL. A Dyna-style explore, and guide method is used where an exploration model is learnt first using a model-based guiding approach that is then used to collect future trajectories. An end-to-end algorithm is proposed that learns a model free guide policy, learn exploration policy, and then collect optimized trajectories. Experimental results on Acrobat and Cartpole tasks demonstrate that the proposed method outperforms existing methods like DQN, SAC and RSActor.
strength_and_weaknesses: The paper extends the research on combining model-based and model-free RL methods for improving sample efficiency and resource utilization. Combining model-free RL with MPC is a well-studied problem but exploring this in an iterated batch RL setting is the key novelty of this paper. This is an extension of Kegl et. al. (2021), where rather than using simple random shooting, a Dyna-style approach is used to learn a dynamic model from gathered data that is used to collect future imaginary traces. Experimental results support the claim that such Dyna-style approach with efficient exploration improves the performance over random shooting.
While I appreciate the approach of using sophisticated Dyna-style approach for generating future traces, I have some reservations regarding the contributions of the paper. Firstly, both Dyna-style approach, and using the concept of model-based exploration are well-studied; this paper just used these ideas in the context of iterated batch RL setting. The paper lacks any theoretical results that can demonstrate the efficacy (in terms of improving two evaluation metrics) of the proposed models in a generic setting. The experiments are conducted on synthetic Acrobat and Cartpole problems, so without theoretical guarantees, it is not clear to me whether the claims are generalizable to other real-world offline RL problems. 
clarity,_quality,_novelty_and_reproducibility: The paper is written well in general, but the presentation might need some improvement. The introduction is well motivated with practical examples. The technical details and connection between different components are clearly mentioned. The experimental results section is easy to follow, except some minor issues (e.g., bold numbers in Table 1 do not seem to the best ones). 
The related work section looks exhaustive and solid to me (albeit some recent work on how exploration or evaluation is done in offline RL is missing). Some additional minor comments:
1.	Some guidance is needed on how an expert would pick the values of input parameters L, n.    
2.	Details of hyperparameter tuning, e.g., temperature, T_i and probabilities, \epsilon_i would be useful.
3.	In experiments, how a model-free RL is trained to find the guide policy?

summary_of_the_review: Iterative batch RL is an interesting and growing research area that can make significant impact in adoption of RL. The ideas presented in the paper are interesting and novel in the sense that an efficient exploration with a learned guiding model can accelerate performance. Experiments demonstrate performance improvement in Acrobat and Cartpole environment. It would be really appreciated if some theoretical results were presented in terms of performance vs. organizational time constraints. Experiments on other real-world domains will further ground the claims. Adding ablation studies with different parameters (e.g., L, n) would be great as well, as my main concern is that choosing an optimal value for these parameters would be challenging.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the  iterated/growing batch RL problem. Authors proposes a Dyna-Style Guide & Explore strategy, which most novel component seems to be an explorer (heating explorer) based on a temperature parameter. The proposed strategy is validated in two classical simple problems: Acrobot (discrete actions) and Cartpole Swing-Up (continuous actions).

strength_and_weaknesses: Strengths: 
- Motivation
- Relevance of the problem being addressed

Weaknesses:
- The novelty of the proposed strategy needs to be better explained. I suggest to improve sub section 3.4
- The two problems used for the validation are rather simple. I recommend to use a much more complex problem to validate the proposes strategy.
clarity,_quality,_novelty_and_reproducibility: Authors' motivation and the problem being addressed are clearly explained. However, the novelty of the proposed strategy is not clearly presented, nor its main components. I suggest to improved the text of sub section 3.4
Given that two standard problems are addressed (Acrobot & Cartpole Swing-up) the results are reproducible.
summary_of_the_review: In the context of iterated/growing batch RL, authors propose a Dyna-Style Guide & Explore strategy, which most novel components seems to be an explorer (heating explorer) based on a temperature parameter. The proposed strategy is validated in two simple, simulated problems: Acrobot (discrete actions) and Cartpole Swing-Up (continuous actions). The simplicity of these problems do not allow to assess properly the relevance of the proposed approach.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a model-based method for iterated batch RL in which an agent learns a model of the transition function which it uses within search to improve a base policy. In comparison to existing Dyna-style approaches, the proposed method uses decision-time planning when selecting actions; in comparison to pure MPC, it learns a guide policy using model-free techniques. The method is evaluated in two domains, Acrobot and Cartpole Swing-Up, showing that it outperforms pure Dyna, MPC, and model-free baselines.

While all reviewers felt the paper tackled an important problem and that the approach was well-motivated, they had concerns about the novelty of the approach and the sufficiency of the evaluation. On the novelty side, the method is a new combination of existing parts (Dyna, rollouts, expert iteration) and is similar to a number of existing algorithms in the literature. This is not in principle a problem, if the paper can demonstrate that this particular combination works better or is more well-motivated than existing approaches. However, the paper does not compare to any other related approaches (e.g. MuZero [1], EfficientZero [2], Policy Gradient Search [3], SAVE [4], etc.)---the comparisons are mostly to ablations rather than true baselines. I appreciate the authors' attempt to clarify some of the differences between their approach and others in their rebuttal, but believe this needs to be done empirically, not just via argument. On the evaluation side, in addition to the lack of appropriate baselines, the reviewers felt that the domains were too simple and that they were not representative of the real-world problems presented in the introduction of the paper, making it difficult to evaluate how well the proposed method would actually fare in difficult real-world settings.

In light of the concerns raised by the reviewers, I do not feel this paper is ready for publication at ICLR and recommend rejection. I encourage the authors to revise the paper to clarify the novelty of their contributions with respect to related approaches (by including them as baselines) and to perform experiments on domains that are more representative for the real-world iterated batch setting.

1. Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., ... & Silver, D. (2020). Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839), 604-609.
2. Ye, W., Liu, S., Kurutach, T., Abbeel, P., & Gao, Y. (2021). Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34, 25476-25488.
3. Anthony, T., Nishihara, R., Moritz, P., Salimans, T., & Schulman, J. (2019). Policy gradient search: Online planning and expert iteration without search trees
4. Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Pfaff, T., Weber, T., Buesing, L., & Battaglia, P. W. (2019). Combining q-learning and search with amortized value estimates.
justification_for_why_not_higher_score: While all reviewers felt the paper tackled an important problem and that the approach was well-motivated, they had concerns about the novelty of the approach and the sufficiency of the evaluation. On the novelty side, the method is a new combination of existing parts (Dyna, rollouts, expert iteration) and is similar to a number of existing algorithms in the literature. This is not in principle a problem, if the paper can demonstrate that this particular combination works better or is more well-motivated than existing approaches. However, the paper does not compare to any other related approaches---the comparisons are mostly to ablations rather than true baselines. I appreciate the authors' attempt to clarify some of the differences between their approach and others in their rebuttal, but believe this needs to be done empirically, not just via argument. On the evaluation side, in addition to the lack of appropriate baselines, the reviewers felt that the domains were too simple and that they were not representative of the real-world problems presented in the introduction of the paper, making it difficult to evaluate how well the proposed method would actually fare in difficult real-world settings.
Decision: Reject