Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to solve the over-exploration issue of posterior sampling (PSRL) in model-based reinforcement learning. They propose Conservative Dual Policy Optimization (CDPO) which involves a Referential Update and a Conservative Update. They also proves CDPO can achieves the same regret as PSRL, and experimental results show CDPO can achieve better performance than PSRL.
strengths_and_weaknesses: Strengths:
1. The paper is well-written and easy to follow.
2. They provide a detailed theoretical analysis for CDPO.
3. CDPO outperforms PSRL on three MuJoCo environments.

Weaknesses:
1. In line 7 the paper claims that ``The sampled model that current policy is greedily optimized upon will thus be unsettled, resulting in aggressive policy updates and over-exploration”. However, in the main sections, the paper doesn’t provide any theoretical analysis or empirical evidence to support this claim. They only cite two papers in line 31. This leads to the lack of reasonable motivation for the proposed method. I hope the authors can provide more detailed analysis for the over-exploration issue.
2. In Eq.4.2, CDPO uses KL divergence $\mu$ to trade off exploitation and exploration, so this is a very important hyperparameter. In section 6.3, the results show that a time-decay $\mu$ performs worse than a fixed $\mu$. This is quite strange because intuitively, as the environment is gradually explored, we should make the current policy closer to the exploitation policy. I think the ablation study for $\mu$ should be more complete and have a more specific explanation. For example, how should the time decay $\mu$ be designed, and how much does $\mu$ affect the performance of the policy.
3. Since CDPO solves two model-based solvers (line 2 and line 4 in Algorithm1), the time complexity of the algorithm should be twice that of PSRL. Many MBRL algorithms take a very long time to run to convergence (for example, one of the state-of-the-art methods MBPO [1] needs almost 48 hours on HalfCheetah-v2).  If combined with CDPO, the running time of the MBRL algorithm is likely to be twice as long. Is it reasonable to take such a long time to obtain an insignificant performance improvement?
4. The benchmark environment for experiments is too simple. Only HalfCheetah is a slightly more complex environment. The authors did not conduct experiments on the complex environments commonly used in MuJoCo such as Ant, Walker2d and Humanoid.
5. The baseline model-based methods (DPI, SLBO) that the author chose to compare in section 6.2 are too old, and these methods are not the current SOTA algorithms. My suggestion is to compare with MBPO [1] and VaGraM [2].

[1] Janner, Michael, et al. "When to trust your model: Model-based policy optimization." Advances in Neural Information Processing Systems 32 (2019).
[2] Voelcker, Claas A., et al. "Value Gradient weighted Model-Based Reinforcement Learning." International Conference on Learning Representations. 2021.
questions: See  Weaknesses above.
limitations: Lack of motivation, limited evaluation
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work addresses the problem of model-based planning in RL, focused on nonlinear models and high-dimensional environments. As the model in these cases tends to be erroneous, planning greedily or optimistically wrt the model can lead to poor performance. The authors propose a conservative update algorithm that relies on greedy updates of a reference policy wrt the least-squares estimate of the model, and another update of the behaviour policy that is constrained to remain close to the reference policy but also to optimize the expected performance wrt. the posterior over models. Theoretically, they show that greedy optimization wrt the least-squares model mimics the behaviour of PSRL in terms of Expected Regret. They also bound the policy improvement at each iteration, and provide global convergence results for classes of models with finite complexity measures. Empirical evaluations show that their proposed algorithm (CDPO) has improved performance over some common baselines on a few continuous control domains.

strengths_and_weaknesses: This paper is very well-written and easy to follow. The authors do a good job of placing it within the context of existing literature. 

Strengths:
- I think the idea behind CDPO is a good one, and addresses an important problem. I also feel that overall the paper does a good job of analyzing this algorithm from different perspectives (including the results in the appendix) 

Weaknesses:
- Experimental results are good but could include more domains. 
- Results depend on convergence of (4.1) and (4.2) (see Questions)
- I think more in-depth description and implications of the result in (5.6) would have been helpful

Regarding writing:
- L217 should be argmax
- L218-219: instead of “PSRL proceeds by …” I would suggest writing out the reasoning and the lemma that is used to obtain this result (e.g. Lemma 1 of [1]). 
- L299-300: I would say “discussed” rather than “analyzed”

[1] Osband, Ian, Daniel Russo, and Benjamin Van Roy. "(More) efficient reinforcement learning via posterior sampling." Advances in Neural Information Processing Systems 26 (2013).

questions: - Each optimization in (4.1) and (4.2) needs to be done to completion. In the deep learning regime, these optimizations may never converge to the optimum point or even good local optima. How does this impact the results? Has this been an issue in the experiments and how was it addressed?

Section 6.1:
- L309-310: “incompatibility brought by the model”: Does this refer to the model class not being sufficient to learn a good model of the environment? If so, how do we know this network is not able to learn a good model of the environment for HalfCheetah? It would help to see the model prediction loss for example. Otherwise, I don’t think we can necessarily make this conclusion. It could be that the policy is harder to learn in HalfCheetah. 
- Related to the above point — L310-311 : “This phenomenon is most obvious in the half-cheetah task, where learning a globally accurate model is hard.” and L312-313: “When the model uncertainty is large, the aggressive policy updates and uninformative exploration lead to slow convergence and lower asymptotic value within finite iterations.” This is more of a hypothesis that’s not really being shown in the results. I don’t think it can be concluded from the present results that this is why the performance is lower.
- How do we know these results are due to better exploration and not, for example, better optimization? 
- Potentially a better argument for better exploration may be better shown in a finite domain where the exploration of the algorithms (i.e. what states are visited) can be visualized.

limitations: I did not find that the authors sufficiently discussed the limitations of their work. See Questions.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel algorithm: Conservative Dual Policy Optimization (CDPO). CDPO involves a referential update optimized optimized for a reference model, which is similar to PSRL. It also involves a conservative update, which provides a conservative range of randomness by maximizing the expectation of model value. CDPO provides same regret bound as of PSRL and can simultaneously offer monotonic policy improvement and global optimality. The paper presents this CDPO framework along with theoretical and empirical analysis.
strengths_and_weaknesses: The paper presents a novel algorithm that can better handle exploration and learning in the face of uncertainty. The contribution is solid given the simplicity, theoretical soundness and empirical superiority of the proposed method.
questions: The paper is well written and well organized, answering most of the questions up front. Thanks to the authors.

Section 2 of the paper builds up the technical ground for model based RL and presents the basic for regret analysis. Section 3 explains OFU and PSRL framework along with general complexity and performance bound analysis for them.

The algorithm presented in section 4 is concise and clear. Theoretical analysis presented in section 5 is solid. 

Empirical evaluation is performed on multiple domains and compared against different SOTA models. The CDPO algorithm shows clear evidence of performing better, specially for domain with higher dimensional state space involving more uncertainties.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes Conservative Dual Policy Optimization (CDPO) that involves a Referential Update and a Conservative Update. It is proven that CDPO achieves the same regret as PSRL and ensures the monotone improvement of the policy. Experimental results validates its effectiveness.


strengths_and_weaknesses: Strengths:
1.The proposed method is simple and effective. Theoretical results are sufficient.
2. The literature review is sufficient.

Weakness:
1. The motivation of the proposed method is not clear and the paper is not easy to follow.
2. The paper only test their algorithm on 3 environments of MuJoCo. I am curious about the performance on larger env such as Ant and Humanoid.
3. I am also interested at the improvement ratio of sample efficiency over baselines.
questions: See the weakness.
limitations: N/A
review_title: null

Meta-review: metareview: Reviewers all appreciated the authors effort on adding additional experiments and revising the draft during the rebuttal phase, and the reviewers are in general satisfied by the authors response. The reviewers agree that the paper has a solid contribution to MBRL.  
confidence: Certain
Decision: Accept