Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an MBRL framework named Plan to Predict (P2P), which treats the model rollout process as a sequential decision-making problem. The model in P2P can minimize the multi-step accumulative errors on the induced trajectories and thus quickly adapt to the state distribution of the current policy, the authors give a theoretical guarantee for P2P. Empirical results on several MuJoCo benchmark tasks verify the effectiveness of P2P.
strengths_and_weaknesses: Strengths
1. The article is overall easy to follow except Section 3.3 (see Questions below);
2. The motivation to fix the mismatch between model learning and model usage is reasonable and valuable. 
3. The experiment is well-designed to demonstrate the mechanism of P2P.

Weaknesses


The article is sound and has no major drawbacks to me. But there are several questions for the authors to further clarify (see below).
questions: 
1. The description in Section 3.3 is a bit confusing. For P2P-MPC, Line 191 mentions "Besides, the reward R^m, which cannot be directly computed by definition, is approximated here using a neural network and trained on the environment dataset". How do we estimate R^m using a neural network, do you mean directly predicting the model error (sounds like a difficult job)? In P2P-RL, the authors mention (Line 205), "simply approximated by re-predicting the dynamics in the sampled transitions and computing the prediction errors for the current model-policy". It seems that R^m is obtained based on the data P^*, calculated by the \hat P prediction error. But if we use the model for multi-step rollout, in the successor time steps, we cannot get the ground-truthP^*(s’|\hat s,a), where \hat s is predicted by \hat P, how to calculate R^m? Also, for P2P-RL, a lot of technical details are mentioned here, such as: “updating the next state in each sampled transition by applying the current policy-dynamics“; ”adopting SAC with behavior cloning as the underlying learning algorithm“; ”DualDICE is also applied to correct the estimation of the state distribution“. I suggest that the authors can add pseudocode or a more complete description in the appendix to improve the readability of this part.


2. The author's explanation for the poor performance of the P2P-RL method (in Line 230) is that "we find that P2P-RL sometimes struggles to balance the loss of behavior cloning and RL, leading to the difficulty in hyperparameter tuning and the instability of learning network parameters”. But we can find that in the Hopper environment, the cumulative error of P2P-RL and P2P-MPC algorithms is similar (Figure 3), while the policy performance of P2P-RL is twice as bad. This seems to be contrary to the explanation proposed by the authors.

3. I think some related work is missing in this article. In recent years, a series of practical work and theoretical analyses have attempted to learn the model using the "treats the model rollout process as a sequential decision-making problem" framework. Its solution is similar but not exactly the same as P2P-RL. I recommend the author to read the following articles and add to the related work, which may inspire the author's follow-up research and improvement work [1,2,3].


[1] Wenjie Shang et al. Environment Reconstruction with Hidden Confounders for Reinforcement Learning based Recommendation.  2019.

[2] Tian Xu et al. Error Bounds of Imitating Policies and Environments. 2020.

[3] Xiong-Hui Chen et al. Adversarial Counterfactual Environment Model Learning. 2022.





limitations: 
Overall, the article is sound and has no major drawbacks to me. If the authors clarify the proposed questions and improve the writing of Section 3.3, I will consider increasing the score of the article.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a model-based reinforcement learning algorithm that alternates between model-learning with the policy fixed, and policy improvement with the model fixed. The key insight is that the model-learning phase is treated as a Markov decision process (MDP) with the estimated model as the "policy" to be learned (with the actual policy fixed), where the reward is the negative mismatch between the estimated model and the actual system transition dynamics. The paper claims that encapsulating model-learning with a fixed policy as a "reversed" MDP encourages model-learning to more quickly adapt to policy updates and produce a model that is more accurate over multiple transition steps.
strengths_and_weaknesses: *** ORIGINALITY ***

The key contribution of the paper is the treatment of the unknown transition model as the "decision-maker" when paired with a fixed policy during model learning. The novelty of this idea is clearly established by the authors in their comparisons to prior work. The paper proposes the "meta-algorithm" Algorithm 1, which requires the user to choose from existing algorithms to specify the model and policy learning phases. Overall, the proposed novelty can be viewed as a specification on the objective for the model learning phase. However, the empirical improvements offered by this key insight are clearly demonstrated in Section 4, so there is strong value in it as a contribution.

*** QUALITY ***

The paper is technically sound for the most part, and the methods used seem appropriate. However, it is unclear if the authors learn the reward function in Algorithm 1 at all -- there does not seem to be a description anywhere in the main paper if R(s,a) is known or must be learned. In addition, in Figure 4 of Section 4, the results cannot be interpreted because the authors have not provided a clear description of what these plots show. Indeed, the caption only vaguely states that these results compare "performances induced by two ways of computing the multi-step prediction loss". Since the authors repeatedly tout the benefit of their approach over prior approaches using multi-step objectives, this part of Section 4.3 should be expanded and improved.

*** CLARITY ***

The paper is clearly written for the most part. Some notation in Theorem 1 is undefined in the main body of the paper and in the appendix (e.g., R_max, D_TV). The use of the term "value" throughout the paper should be clarified (e.g., does this refer to a value function?). The use of the terms "policy-dynamics" for the policy pi and "model-policy" for the dynamics model P_hat beginning on line 180 is confusing. It would probably be clearer to just refer to them as the policy and model, respectively, and ensure it is clear to the reader when either the policy-learning or the model-learning phase is being discussed.

*** SIGNIFICANCE ***

The key idea in treating dynamics learning with a fixed policy as an MDP with the transition model as the "policy" is interesting and could be useful to other researchers. The empirical results, particularly in Figures 3 and 7, seem to show that P2P can potentially reduce model error over multi-step rollouts.

The primary concern I have with this paper is its treatment of regions in state-action space with high model uncertainty. The authors seem to consider such regions as ones that should be avoided during reinforcement learning. On line 272, the authors state that their method instead "tends to generate trajectories heading to the regions with low uncertainty", while the baseline method MOPO "fails to prevent the trajectories from being absorbed into the highly uncertain region". This line of reasoning runs counter to the exploration-exploitation principle in reinforcement learning. Indeed, while the scenario in Figures 1 and 6 seems to have the goal in a region of low model uncertainty, it is certainly possible to encounter scenarios where the goal is instead in a region of high model uncertainty. Thus, exploration into regions with high model uncertainty in the online setting perhaps should be encouraged rather than avoided (subject to, e.g., safety constraints). This is a potentially critical limitation of the proposed work that is not discussed, and perhaps should be investigated with an additional experiment where the goal is located in a region of high model uncertainty.
questions: Based on the discussion above in "Strengths and Weaknesses", I have the following questions and requests for clarification:

- Is the reward function R(s,a) assumed to be known? If not, where and how do you learn R(s,a) in Algorithm 1?

- For P2P-MPC, on lines 190-191 the authors state "the reward Rm, which cannot be directly computed by definition, is approximated here using a neural network and trained on the environment dataset." For P2P-RL, on the lines 205-207 the authors state "since the samples come from the true environment, the reward Rm can be simply approximated by re-predicting the dynamics in the sampled transitions and computing the prediction errors for the current model-policy". Both statements lack significant detail and yet seem to be critical to model learning since they quantify the difference between the model P_hat and the true transition dynamics P. Please expand on these points.

- Please expand on your evaluations of the model error induced by P2P and multi-step loss function baselines, particularly in Figure 4. As it stands, the lack of details severely hampers reproducibility.

- The authors promote that their proposed method P2P learns by "actively interacting with the current policy" (line 66) and has the "capability to avoid model error" (line 84). Distinctly, the authors' ant-in-a-maze example scenario has the goal in a region of low model uncertainty. However, it is certainly possibly to encounter a scenario where the goal is in a region of high model uncertainty, thereby requiring further exploration in the online setting. Could P2P underperform or fail in such scenarios? Further experiments would be welcome here.

limitations: There is no substantial discussion of limitations in this work. There is some superficial comparison of P2P-MPC and P2P-RL at the end of Section 4.1 and in Appendix C, but little insight or discussion is provided. I have raised an issue regarding exploration-exploitation in the review sections above that may be useful for the authors to consider.

The authors have adequately addressed the societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an improved method for learning models for model-based reinforcement learning, by recasting the supervised regression approach commonly used as a control problem.
strengths_and_weaknesses: The following will be very brief and most of my comments will be under "Questions", because I have serious doubts that I actually understood the method the authors are proposing. This is not due to lack of familiarity with the field, I have done research and published in model-based reinforcement learning. I will point out my problems in the next section.

Given the very nice results and the overall introduction, I am very favorable to recommend acceptance. This is not currently reflected in the score however, due to the questions I have about the paper. Therefore I would ask the authors to go through them and hopefully improve the presentation (or convince me that I overlooked crucial details of course :) ), and to not be discouraged by the low score! I think the ideas warrant a presentation at NeurIPS, but I think that the paper can be heavily improved to make it accessible and give it a chance to shine in the community.
questions: - Figure 1: while nicely designed, I do not think I fully understand what the figure represents. It has a lot of details and an unfamiliar setup. Please simplify and clarify.
- Line 42: Since the term "objective mismatch" is used, I would encourage the authors to cite previous work by Lambert et al. and others.
- Justification: It is unclear to me why exactly the model should prevent the policy from going into high uncertainty regions. First, in most model based methods, several trajectories would be computed (explicitly with random shooting, or implicitly, by repeatedly querying a probabilistic model). These should to some extend counteract the overestimation, as the model would be less controllable in this region. Furthermore, executing the next policy on the real environment will hopefully provide counterfactual data to the previously erroneous model.
- How does predicting $R^m$ actually differ between this approach and MBPO? How is the expectation with regard to the current policy realized?
- Line 162: I do not fully understand what is meant by this sentence? Is the difference in the data used for the update?
- Line 190: The $R^m$ is trained as a neural network? How exactly? This seems crucial for making the method work, but is treated as a side comment.

I think a lot of my confusion stems from the last question: how is the reward/loss for the mode actually computed. I think a minimal change that would alleviate a lot of my concerns would be to add a more clear, step by step explanation of the algorithm to the top of Section 3, instead of relegating implementation details to later? Since the setup is fairly different from other model learning approaches, a more "tutorial style" of writing would immensely benefit my (and hopefully others) understanding.
limitations: The method discusses an optimization method and therefore does not clearly require a discussion of societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Model based reinforcement learning promises to reduce sample complexity in reinforcement learning. MBRL is responsible for some of the major breakthroughs in AI, zuch as AlphaGo Zero, and AlphaFold. MBRL may suffer from model inaccuracy, especially when the learning of the model, which is typically doen with 1 step lookahead, is separate from model usage, which is typically done with many step lookahead. The paper suggests a solution to this problem, using a meta-approach, to approach the model learning as an MDP problem itself. 
The paper offers both a theoretical analysis, and convincing experimental evidence of strongly better performance than some of the best model based, and some of the best model free approaches. This a convincing contribution.
strengths_and_weaknesses: Strengths
- clear writing, clear problem statement
- clear contribution
- good theoretical analysis
- convincing experimental evidence
- important new idea, that performs substantially better. important contribution

Weaknesses
- few
questions: - can you compare to more other MBRL approaches
- can you run on larger problems?
limitations: few limitations are mentioned in the paper. The authors are invited to provide more
There are no ethical issues to address
review_title: null

Meta-review: metareview: All the reviewers agree that this is a good paper. The idea is original and the paper has good empirical results. There were some confusions, which were resolved during the discussions and the revised paper. I recommend this paper to be accepted, possibly as a spotlight presentation.

I enlist a few concerns below, so that the authors can improve their paper. Some of them are by the reviewers, and some of them are by myself.

- Be more clear about how R^m is estimated. This is discussed in Appendix B.3 of the revised version, but given its importance to the algorithm, the authors may want to consider discussing it in the main body.

- Reviewer q2Vv mentioned their concerns about preventing the agent to go to the uncertain regions, which may prevent the exploration. The authors answered that "the exploration-exploitation tradeoff in RL mainly works on real environments instead of the approximate models".
This is not entirely accurate. Methods based on the optimism in face of uncertainty, such as UCRL, actually try to exploit the uncertainty of the model. If the model's promise of large return turns out to be false, due to its large uncertainty, we have gathered useful information and decreased our uncertainty.

- I feel that there is a gap between the theoretical results and the algorithm. It does not seem that the optimizer of the the model MDP (Definition 1), which is optimized on L5 of Algorithm 1, is the same as the optimizer of the upper bound on Theorem 1, used to justify the algorithm.
For example, the $e^m_t$ term is based on maximum over actions of the TV error between the model and the true environment, weighted according to state distribution induced by the model. On the other hand, $R^m$ in the model MDP (after taking the expectation over $s_{t+1}$ coming from distribution $P^m$), seems to be the chi-squared divergence, which is also weighted by the policy $\pi$. They are not the same.
It is OK if they are slightly different for practical purposes, as long as one can show their relation and be clear about it.

- The inequality at the beginning of Section 3.2 (Theoretical Results) requires $|J^\hat{P}(\pi) - J(\pi)|$ be smaller than C for both the new policy pi and the old policy $\pi_D$.
Disregarding the previous issue (or assuming that it can be resolved), solving the optimization problem defined on L5 of Algorithm 1 only guarantees $|J^\hat{P}(\pi) - J(\pi)|$ to be small for the current policy $\pi_D$, and not the optimized one. As such, the inequality is not satisfied, even if the algorithm works well.
Am I missing something? Please clarify it in the revised paper.

- It is claimed on L154 that monotonic policy improvement can be achieved by solving the update rule (2). I don't think it is correct. We need the value of the maximizer to be larger than $J(\pi_D)$, which may not always be the case.
confidence: Certain
Decision: Accept