Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a decentralised version of the cross-entropy method for model-based reinforcement learning. The convergence of the proposed algorithm is analysed under several assumptions (Hu et al. [2011]) and evaluated in several continuous control environments compared with PETS and POPLIN. 
strengths_and_weaknesses: **Strengths**
1.  Decentralization of CEM in MBRL can deviate from local optima and improve sample efficiency over its centralisation counterpart.
2. A relatively complete theoretical and empirical analysis and in-depth ablation studies.
3. Evaluating planning tasks in different continuous control environments are nicely done.

**Weakness: **
The readability is impossible without referring appendix with many cross-reference links broken.

questions: 1. Where is the definition of the initial sampling distribution $g_{\phi}(\tau)$? The first time it appears is in line 91. It is difficult for readers unfamiliar with planning (MPC) to proceed.

2. What is $d_a$ in line 93?

3. Theorem 3.1 is entirely broken. Where are all the assumptions? It is not pleasant to just say, “The detailed proof is left to Appendix H”.

limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Cross-Entropy Method (CEM) is commonly used for planning in model-based reinforcement learning (MBRL) where the *centralized* approach is typically utilized to update the sampling distribution based on only the top-*k*  operations' results on samples. In this paper, the authors show that such a *centralized* approach makes CEM vulnerable to local optima, thus impairing its sample efficiency. To tackle this issue,  the authors propose **Decent**ralized **CEM (DecentCEM)**, a simple but effective improvement over classical CEM, by using an ensemble of CEM instances running independently from one another, and each performing a local improvement of its own sampling distribution. The theoretical and empirical analysis are provided to demonstrate the effectiveness of this simple *decentralized* approach. It is shown that compared to the classical centralized approach using either a single or even a mixture of Gaussian distributions, their DecentCEM finds the global optimum much more consistently thus improving the sample efficiency. Furthermore, they plug in their DecentCEM in the planning problem of MBRL and evaluate their approach in several continuous control environments, with comparison to the state-of-art CEM-based MBRL approaches (PETS and POPLIN). Results show sample efficiency improvement by simply replacing the classical CEM module with their DecentCEM module, while only sacrificing a reasonable amount of computational cost, which can be alleviated by using the advantage of the parallelism of their ensemble method. Lastly, they conduct ablation studies for more in-depth analysis.
strengths_and_weaknesses: **Strengths**
1. The proposed is simple but effective. This work can be utilized in any CEM-based model-based reinforcement learning algorithm.
2. The paper is well organized and the writing is good. 
3. The theoretical and empirical analysis are persuasive to demonstrate the effectiveness of the proposed method.


**Weaknesses**
1.  The assumptions of the proposed algorithm are a little strong, especially the second point. The authors assume that the reward function is known and deterministic which is not common in practice actually.

2. The visualization part in the experiment still needs to be improved.
questions: 1. Could you please add **DescentPETS** as the baseline to make the experiment more comprehensive? **DescentPETS** means replacing the CEM module in the previous PETS algorithm with the DescentCEM proposed in this paper.

2. Could you please investigate the effectiveness of the **DecentCEM** both theoretically and empirically when assumption 2 is alleviated?

3. Could you please visualize the planning trajectories with **CEM** and *DecentCEM** and compare them if possible?
limitations: 1. We can notice that the theoretical convergence result does not match the empirical observation well. Please investigate this deeply and explore more effective methods to improve the algorithm's practical convergence result.

2. Could you please investigate the theoretical convergence result of your proposed algorithm under the *constant sample size* constraint as what you have used in your experiment?

3. Could you please explore how to promote your method to other non-CEM-based MBRL algorithms?
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper propose a multi-instance CEM method, a strict generalization of CEM, which is able to remain spread-out in the search space. The authors test on continuous-action model-based RL.

strengths_and_weaknesses: The method uses common optimization philosophy and have shown good performance; and the analysis is sound. The paper is overall well written and have included an example for intuition. 

The method design principled and given theoretical attention.

However, the empirical evaluation hardly distinguish the method from the baselines, that is, the performance gain appears to be minimal.

The novelty of the proposed method is not very significant; however, it can be considered ground work for many future works.

questions: Footnote 2 seems incomplete.

How is BiggerPolicyNet tuned in the ablation study?

How can DecenCEM be used in settings other than continuous-action MBRL?

How does the performance compare to gradient based methods?

limitations: + The authors are upfront about their setting (continuous action, requires a model, etc.).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes DecentCEM, where two or more instances of cross-entropy method (CEM) are run independently with different sets of samples until convergence. For model-based decision making, the results from the ensemble are gathered and combined by choosing the best sampling distribution that maximizes the expected return of trajectories. Theoretically, they show that each instance of CEM converges under the same set of assumptions for convergence of vanilla CEM. They combine DecentCEM with CEM planning and empirically show that DecentCEM outperforms baseline methods including PETS and POPLIN in multiple environments for MBRL.
strengths_and_weaknesses: Originality

- In terms of the originality, the idea of running multiple instances of the same algorithm and combining the results is not very novel, as it is one of the common patterns in problem solving.

Quality

- The method is sound, and I agree that it could bring improvements in multiple situations.
- I think Theorem 3.1 (Convergence of DecentCEM) needs to be more specific about the "almost-sure" convergence of the solution of DecentCEM. Even if they maintain the expression "almost surely", at least they should state that it holds for large-enough sample sizes.

Clarity

- Overall, the manuscript is easy to follow and well-organized. Especially, the early introduction of the motivating examples can help better understanding of the paper.
- Some parts of the paper could mislead readers and possibly oversell the results of the paper. For instance, in the abstract, the authors mention that they "provide both theoretical and empirical analysis" and the next sentence states that they "show" that "DecentCEM finds the global optimum much more consistently". This could easily make readers think that such statement is backed by theoretical results as well, which is not true according to my understanding.
- Regarding Theorem 3.1 (Convergence of DecentCEM), please see my comment above on quality.

Significance

- Even though the proposed decentralized CEM is not fully novel algorithmically, suggesting that such approach yields valid results with potential improvements can still be somewhat meaningful.
questions: - I would like to check out the details and a more thorough version of Table A.3. How many instances or mixture components were employed for the table?
- Also, please see "Strengths And Weaknesses" for additional relevant comments.
limitations: I think one main limitation of the method is fairly covered, regarding the discrepancy between the theoretical result and practice.
review_title: null

Meta-review: metareview: This paper proposes a parallelized version of the classic cross-entropy optimization method, by using an ensemble of CEM instances running independently from one another, and each performing a local improvement of its own sampling distribution. Both a theoretical and empirical analysis are provided to demonstrate the effectiveness of this simple decentralized approach. The reviewers find the paper to be overall well-presented, and appreciate the fact that the proposed is simple but effective. Consequently, this work can be utilized in any CEM-based model-based reinforcement learning algorithm.
confidence: Certain
Decision: Accept