Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper first demonstrates that the model shifts---the difference between the updated model and the model before updating---hinder the monotonic improvement of model-based RL. To tackle this problem, the paper proposes CMLO, which introduces an event-triggered mechanism to determine when to alleviate the model shifts. Experiments show the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:

1. The authors propose the theoretical analysis to show that the model shifts hinder the monotonic improvement of model-based RL, which provides a useful perspective in model-based RL.

2. Experiments demonstrate that the proposed method improves the performance and generalization of existing methods.


Weaknesses:
1. The authors propose a constraint estimation in their method to estimate the model shifts in Proposition 4.7. However, in Lines 245-247, the authors use the data from the current ensemble models and real environment to compute the constraint estimation, which is not consistent with Proposition 4.7.

2. The authors propose the event-triggered mechanism to determine when to update the model instead of the frequent model updating in MBPO [1]. However, the motivation to alleviate the model shifts by the event-triggered mechanism is unclear. The authors may want to explain why they introduce the event-triggered mechanism based on their proposed theoretical analysis.

[1] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019.
questions: My suggestions are as follows.

1. The advantages of the proposed event-triggered mechanism are unclear. The authors may want to explain why the event-triggered mechanism is more effective than previous methods [1, 2].

2. The authors claim that the proposed constraint estimation is “the practical overestimation for the model shifts” in Lines 247-248. The authors may want to provide the theoretical analysis of the overestimation.

3. Several definitions are missing, such as $\sigma_{M_1, M_2}$ in Theorem 4.6 and $\Delta \mathcal{D}$ in Line 245.

4. The formulation for computing the proposed volume $vol(\mathcal{S}_\mathcal{D})$ in Lines 239-244 is missing.

5. The authors may want to provide the detailed motivation of Equation 5.1.

6. The experiment settings in Section 6.2 are missing, such as the settings of all methods in Figure 2 and that of the visualization in Figure 3. The authors may want to provide the detailed settings of all ablation studies.

[1] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019.

[2] Hang Lai, Jian Shen, Weinan Zhang, Yimin Huang, Xing Zhang, Ruiming Tang, Yong Yu, and Zhenguo Li. On effective scheduling of model-based reinforcement learning. In Advances in Neural Information Processing Systems, 2021.

limitations: Yes, the authors adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies how to ensure optimization monotonicity of learning an accurate dynamics model for MBRL. They derive a lower bound for the derived policy performance improvement that depends on the one-step dynamics prediction error of the current model, constraint by the model shift. 

Inspired by the theory, they propose an algorithm to dynamically alternate between policy exploration and model learning, with the aim to improve optimization monotonicity. They evaluate the proposed algorithm on a series of MoJoCo control tasks and compared the results against a few model-free baselines, and show that their model-based method is able to reach the same SOTA asymptotic performance while being more stable and sample efficient. 
strengths_and_weaknesses: Strength:

The problem studied in the paper is important and relevant to practice. The paper is well-written and easy to follow. The proposed method is theoretically motivated.

Weakness:

My main concerns are around the experiments and potential limitations. Please see the section below for detailed comments.
questions: 1. In Figure 1, the performances on different tasks are capped at different timesteps. In several cases the learning curves have not stabilized yet, e.g. Walker2d and Swimmer. Could you please report the final performance where the learning curves stabilize or at 5M steps (as used to determine the “asymptotic performance” of the SAC and MBPO baselines).
2. What is the y-axis in Figure 3(b)? Could you please include the policy coverage and prediction error results for some of the baselines for a comparison?
3. Figure 4 looks a bit confusing to me. Do the MBRL baselines start with an initial 30k steps exploration followed by model learning (inferred from Figure 4)? Why only showing 4k steps per stage instead of the whole 60k? What is the y-axis?
4. Figure 4(b) shows that the model shifts estimation jumps drastically from stage 1 to stage 2, and that it doesn’t hit the threshold at all in stage 1 (first 4k steps). Could the authors please explain this unexpected observation?
5. The absolute performance of TRPO on HalfCheetah and Ant tasks shown in Figure 5(a) are much lower than the other baseline results shown in Figure 1. The variance also seems quite high. Is it expected?
6. Why choosing these specific training step caps in Figure 5(b)?
7. Could you briefly discuss the costs induced by estimating the volume of the convex closure, especially when applying to high-dimensional state space?
limitations: The paper mentioned briefly in the conclusion that finding the optimal threshold for each specific environment may be time consuming. Could you please elaborate on that? For example, how different are the thresholds for different environments and the amount of steps it takes for tuning.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper focuses on the monotonic improvement for model-based reinforcement learning which is an extremely important problem due to the inherently entangled nature of the multi-level optimization problem - policy optimization and model learning. Earlier research has not specifically considered the model shift which is considered in this research while proving the monotonic improvement. The primary objective is to show that $||V^{M1|\pi_1} - V^{M2|\pi_2}|| \geq C$ which can guarantee monotonic improvements under the updating dynamics. The primary reason for the model bias is a mismatch between the samples in the model learning stage and the policy optimization stage. To tackle the same, they formulate a constrained bi-level optimization framework for the MBRL problem and design an event-triggered strategy to decide when to update the model to guarantee monotonic improvement under changing dynamics. Empirical results show some improvements in sample efficiency from prior stable model-based RL methods and the ablation study supports the main hypothesis of the paper. 

strengths_and_weaknesses: The primary strength of the paper lies in the formulation of the bi-level constrained optimization objective from the lower bound objective with an event-triggered mechanism under a generative model assumption to update the model which is quite novel according to me. Earlier research has addressed the bias in the model learning and has also tried to guarantee monotonic improvement, but this paper explicitly talks about the monotonic improvement of the policy under shifted model dynamics and shows guarantees of monotonic improvements (with certain assumptions) which is interesting and novel.  The biggest challenge in guaranteeing a monotonic improvement in the policy is due to the changing dynamics which might occur due to a potential mismatch between the true trajectories and model-generated trajectories and is indeed an important challenge to overcome. The paper breaks down the source of error into 2 components with the assumption that we can always get an epsilon optimal policy under a given model which is a bit optimistic but a very common and frequently used assumption in RL. The components being 1. inconsistency gap between the model and the environment : $E_{s,a \sim d_{\pi}} TV(P(\cdot|s,a) || P_M(\cdot|s,a))$ and 2. Optimal returns under the true model and they hypothesize that the performance difference $||V^{M1|\pi_1} - V^{M2|\pi_2}||$ is lower bounded by the above two aspects. Finally, with the above notions, the authors formulate a constrained lower bound optimization problem as stated in Theorem 4.6 which is quite novel as the formulation of model-based RL as a bi-level optimization problem is quite natural as done in this paper. The strength of the paper lies in designing an event-triggered mechanism and providing a high probability bound for the value of $k$ for guaranteeing monotonic improvement under the generative model assumption. In Corollary 4.8, the authors derive a relation between the model training interval $k$, model bias, and state-space coverage which is quite unique to my knowledge and given by $k \propto \frac{2 \times vol(S)}{\epsilon^2}$ (just approximated and ignore other terms). In other words, when the model bias is less one needs to increase $k$ to increase state action coverage which is quite intuitive. Overall the ablation study seems interesting and the experimental results show some improvements over the past SOTA model-based RL methods.

The primary weakness lies in various assumptions that have been made to derive some of the primary results that are not very general. To begin with, the main result in Corollary 4.8 is shown with a generative model assumption which is quite restrictive. The primary novelty of the research from a solution perspective lies in designing the event-triggered mechanism and the theory is shown with a generative model assumption also with linear
quadratic regulator (which I still believe can be relaxed) and not for a general scenario. Secondly, it is not very clear how minimization of the constrained objective function in Proposition 4.7 turns out to be simple negative likelihood minimization. In Proposition 4.7, there is $P(s'|s,a)$ involved in the expression which can't be ignored since the $(s,a) ~ d^{\pi_2}$ and optimization variable involve $\pi_2$ as well. So, it's not clear how this is handled as we will never know $P(s'|s, a)$, and how this boils down to simple NLL minimization is not explicitly described. Also, I see in the derivations on lines 90, 97, 128, etc. the proofs involve summation over s,a which are relevant for Tabular or Discrete settings but the experiments are for continuous state-action space which causes a mismatch and some of the proofs might break when everything is expressed for continuous state-action spaces. The ablation study is interesting and the experimental results show improvement but not significant improvements. However, I feel the work addresses an important question and it might be helpful to understand this aspect for model-based RL.
questions: 1. Can you please illustrate how exactly is the volume of the of convex closure computed from the replay buffer? 

2. Can you please point me to the equation where you derive an estimation for the model shift as a product of volume $\times$ avg prediction error?

3. Instead of going by the complicated event-triggered mechanism, if I simply try to obey the constraint in the optimization problem with a TRPO type regularized update in the model parameter space that should also do right? Given that we know that under the gaussian dynamics assumption, the total variation distance simply can be upper-bounded by the difference b/w the means and variances as well and hence it should be easy to be in the feasibility region.

4. Can you please show the derivation on how you obtain line 47 in Appendix where you mention that with Lipschitzness of $V^{\pi}_{M_1} V^{\pi}_{M_2}$ you derive the bounds for $|G^{\pi}_{M_1, M_2} (s,a)| \leq L |P_{M_2}(\cdot|s,a) - P_{M_1}(\cdot|s,a)|$. 

5. Additionally in line 48 in Appendix, you derive an upper bound for value function difference in-terms of the $|P_{M_2}(\cdot|s,a) - P_{M_1}(\cdot|s,a)|$. However, it has been shown in [1] that there is a dependence of the maximum reward $R$ (and horizon $H$ which will be replaced by some factor of $\gamma$ here) intermingled/ associated with the Lipschitz constant. Basically, they prove that in the Lipschitz assumption of $||V1-V2|| \leq L ||P1- P2||$, $L$ has a dependence on $CHR_{max}$. I think that should be applicable here as well, then how does the analysis gets affected? As now there is an additional dependence on the max reward that will be added. Can you please share your thoughts on the same?

References : 
[1]. Ying Fan and Yifei Ming. Model-based reinforcement learning for continuous control with posterior sampling. In Marina Melia and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 3078–3087. PMLR, 18–24 Jul 2021.
limitations: The author mentions some points regarding the current applicability to certain environments and wants to scale with improved optimization methods which are sensible.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the relationship between the shift brought by model updates and policy performance. The authors proposed a model shift constraint and the CMLO algorithm for a monotonic improvement guarantee.
strengths_and_weaknesses: pros: 1. The authors consider an important problem in RL, the inconsistency of policy updates under shifted models. The main results and proofs are largely correct in my view.

2. The ablations verify the proposed algorithm.

cons: 1. The monotonic results are not novel in MBRL, for example, the authors cite DPI [1], which established similar monotonic results. But the authors claim that previous works "characterize the monotonicity in terms of a fixed model of interest". I don't see the reason for such a claim and the exact advantage of CMLO.

2. From the introduction and related work, the partial reasons for cons 1 might include that the authors derive the theorems from a global optimal view. But I can't judge the necessity of doing this. E.g., in theorem 4.5, why do we care about the *optimal* value under a model? Notably, the monotonic property rather than global optimality is of interest. Therefore, according to the performance difference lemma, shouldn't we focus more on the value-under-model evaluated using the *current policy*?

3. Another confusion I have is the notation of state coverage. As it is introduced from a global point of view, I can't judge its necessity before understanding the necessity of the global view. Besides, it is better to include the exact definition of state coverage in the context of RL, so that we can see why it is estimated with the replay buffer. 

4. The intuition behind the event-triggered equation 5.1 is also not very clear. What will the fraction of state coverage give?

Minor: how much additional time does it cost for estimating model shifts?

[1] Dual Policy Iteration, Wen Sun et al.
questions: 1. It would be better to have more comparisons between CMLO and previous works (e.g. DPI). Although they have different proof structures (from the local and global view?), the monotonic improvement results are similar in my view. 

2. Why do we care about the *optimal* value under a model and what is the advantage compared to the local view? I am assuming that it is the primary reason that leads to different algorithms between CMLO and previous works.

3. The definition of state coverage and the event-triggered equation is not so clear.

My concerns are closely related. So it might be the case that I missed something important. I would like to change my score depending on the authors' rebuttal.
limitations: No.
review_title: null

Meta-review: metareview: This paper studies the relation between model shift and the performance of model-based reinforcement learning. The paper proposes a new algorithm that leads to empirical improvement over certain data sets. All the reviewers agree that the paper provides useful theoretical insights into model-based reinforcement learning, and the experiments are also consistent with the theory. 
confidence: Certain
Decision: Accept