A Simple Decentralized Cross-Entropy Method
Authors: Zichen Zhang
Jun Jin
Martin Jagersand
Jun Luo
Dale Schuurmans
Venue: NeurIPS 2022 Accept (2022)
Keywords: planning
cross-entropy method
reinforcement learning

Abstract: Cross-Entropy Method (CEM) is commonly used for planning in model-based reinforcement learning (MBRL) where a centralized approach is typically utilized to update the sampling distribution based on only the top-$k$ operation's results on samples. In this paper, we show that such a centralized approach makes CEM vulnerable to local optima, thus impairing its sample efficiency. To tackle this issue, we propose Decentralized CEM (DecentCEM), a simple but effective improvement over classical CEM, by using an ensemble of CEM instances running independently from one another, and each performing a local improvement of its own sampling distribution. We provide both theoretical and empirical analysis to demonstrate the effectiveness of this simple decentralized approach. We empirically show that, compared to the classical centralized approach using either a single or even a mixture of Gaussian distributions, our DecentCEM finds the global optimum much more consistently thus improves the sample efficiency. Furthermore, we plug in our DecentCEM in the planning problem of MBRL, and evaluate our approach in several continuous control environments, with comparison to the state-of-art CEM based MBRL approaches (PETS and POPLIN). Results show sample efficiency improvement by simply replacing the classical CEM module with our DecentCEM module, while only sacrificing a reasonable amount of computational cost. Lastly, we conduct ablation studies for more in-depth analysis. Code is available at https://github.com/vincentzhang/decentCEM.

A Simple Decentralized Cross-Entropy Method
Zichen Zhang
Jun Jin
Martin Jagersand'
Jun Luo Dale Schuurmans i.f
Huawei Noah's Ark Lab Tequal advising (zichen,,m)] Guaeertaabbbrr.cccmmm ca jinn, fini,jum jun. luol] Shuawei. com
University of Alberta
Abstract
Cross- Cross-Entropy Method (CEM) is commonly used for planning in model-based reinforcement learning (MBRL) where centralized approach is typically utilized to update the sampling distribution based on only the top-k operations" results on samples. this paper, we show that such a centralized approach makes CEM vulnerable to local optima, thus impairing its sample efficiency. To tackle this issue, we propose Decentralized CEM (DecentCEM)), simple but effective improvement over classical CEM, by using ensemble of CEM instances running independently from one another, and each performing local improvement its own sampling distribution. We provide both theoretical and empirical analysis demonstrate the effectiveness of this simple decentralized approach. We empirically show that, compared to the classical centralized approach using either a single even a mixture Gaussian distributions, our DecentCEM finds the global optimum much more consistently thus improves the sample efficiency. Furthermore, we plug in our DecentCEM in the planning problem of MBRL,, and evaluate our approach in several continuous control environments, with comparison to the state- of-art CEM based MBRL approaches (PETS and POPLIN). Results show sample efficiency improvement by simply replacing the classical CEM module with our DeC module, while only sacrificing a reasonable amount of computational cost. Lastly, we conduct ablation studies for more in-depth analysis. Code is available at https: // //gi thub. wwon///nnntttaangg.... (etttzzhaannddeentEEI.. decentCEM.
1 Introduction
Model-based reinforcement learning (MBRL) uses model as a proxy of the environment for planning actions in multiple steps. This paper studies planning in MBRL with specific focus the Cross- Entropy Method (CEM) [De Boer et al., 2005, Mannor et al., 2003), which is popular in MBRL due to its ease of use and strong empirical performance [Chua et al., 2018, Hafner et al., 2019, Wang and Ba, 2020, Zhang et al., 2021, Yang et al., 2020]. CEM is a stochastic, derivative optimization method. It uses sampling distribution interactions with the model. These trajectories are then ranked based on their returns computed from the rewards given by the model. The sampling distribution is updated to increase the likelihood of producing the top-k trajectories with higher returns. These steps are iterated and eventually yield an improved distribution over the action sequences to guide the action execution in the real environment.
generate imaginary trajectories of environment-agenn
Despite the strong empirical performance of CEM for planning, it is prone to two problems: (1) lower sample efficiency the dimensionality of solution space increases, and (2) the Gaussian distribution that commonly used for sampling may cause the optimization to get stuck in local optima of multi-modal solution spaces commonly seen in real-world problems. Previous works addressing these problems either add gradient-based updates of the samples to optimize the parameters of CEM,
Work partially done during Zichen's internship at Huawei Noah's Ark Lab.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

or adopt more expressive sampling distributions, such as using Gaussian Mixture Model [Okada and Taniguchi, 2020] or masked auto-regressive neural network [Hakhamaneshi al., 2020]. Nevertheless, all CEM implementations to date are limited to centralized formulation where the ranking step involves all samples. As analyzed below and in Section 3, such a centralized design makes CEM vulnerable to local optima and impairs its sample efficiency.
We propose Decentralized CEM (DecentCEM)) simple but effective improvement over classical CEM, to address the above problems. Rather than ranking all samples, as in the centralized design, our method distribute the sampling budget across an ensemble of CEM instances. These instances run independently from one another, and each performs a lo- cal improvement of its own sampling distribution based on the ranking of its generated samples. The best action is then aggregated by taking an arg max among the solution of the instances. It recovers the conventional CEM when the number of instance one. We hypothesize that by shifting to this decentralized design, CEM can be less susceptible to premature convergence caused by the centralized ranking step. As illustrated in Fig. I, the centralized sampling distribution exhibits bias toward the sub-optimal solutions near top right, due to the global top-k rank- ing. This bias would occur regardless of the family of distributions used. In comparison, a decentralized approach could maintain enough diversity thanks to its local top-k ranking in each sampling instance.
Through a one -ieddmmeniooal multi-modal optimiza- tion problem in Section 3, we show that DecentCEM empirically finds the global optimum more consis- tently than centralized CEM approaches that use either a single or mixture of Gaussian distribu- tions. Also we show that DecentCEM is theoreti- cally sound that it converges almost surely to local optimum. We further apply it to sequential decision making problems and use neural networks to param- eterize the sampling distributions. Empirical results in several continuous control environments suggest that DecentCEM offers an effective mechanism to improve the sample efficiency over the baseline CEM under the same sample budget for planning.
(a) Centralized CEM
(b) Decentralized CEM
Figure 1: Illustration of CEM approaches in opti- mization. Shades of red indicate relative value of the 2D optimization landscape: redder is better. and optimal solutions are near bottom left corner of the solution space. Blue dots are top-k samples, and black dots are other samples. Open dots O repre- sent sampling distributions whose sizes indicate the number of generated samples.
2 Preliminaries
We consider a Markov Decision Process (MDP) specified by (S.A.R...,.doo..)) S Rd. is the state space, A C Id is the action space. d, d, are scalars denoting the dimensionality, R ::C S x A + R is the reward function that maps state and action pair to real-valued reward. P(s'ls, a) 5x A S + R is the transition probability from state and action pair s, the next state s'. E [o, 1] is the discount factor. dp denotes the distribution of the initial state At time step f, the agent receives state s, 2 and takes an action a, according to policy x-Iss) that maps the state to a probability distribution over the action space. The environment transitions to the next state 8 ~ P(-[st, ax) and gives reward r R(s,, at) to the agent. Following the settings from CEM-based MBRL papers (Sec. 6.1,, we assume that the reward function is deterministic (a mild assumption [Agarwal et al., 2019]) and known. Note that they are not fundamental limitations of CEM and are adopted here so as to be consistent with the literature. The return G, = Ei-n y're4i. the sum of discounted reward within an episode length of T. The agent aims to find policy i that maximizes the expected return. We denote the learned model in MBRL as fu (-1s,, a). which is parameterized by and approximates P(-ss,a).
Planning with the Cross Entropy Method Planning MBRL is about leveraging the model to find the best action in terms of its return. Pwweieriieeeeoonnieee (MPC) performs decision temms
We assume full observability, iee agent has access to the state.

---

planning at each time step up to a horizon to find the optimal action sequence:
TMPC (t)) = arg max 18 [ y (r(())))() + * (sH))
(1)
where H. is the planning horizon, 0 denotes the action sequence from time step / L H and V(SH) ) is the terminal value function at the end of the horizon. The first action in this sequence executed and the rest are discarded. The agent then re-plans the next time step.
distribution parameters
Sampling Distribution
Samples
Model
Top-k Samples
action
(1) m B/, B1
time
Rg-1)
de 0g
CEM Internal Loop
CEM
Figure 2: Cross Entropy Method (CEM) for Planning MBRL
The Cross-Entropy Method (CEM) is a gradient-free optimization method that can be used for solving Eq. (1). The workflow is shown in Fig. 2. CEM planning starts by generating N samples {Tz} = {(ā3,0 iy.1, ag.H-1)) from an initial sampling distribution (x(T) parameterized by o, where each sample тj is action sequence from the current time step up the planning horizon H. The domain (T) has a dimension of d, H.
Using model f. CEM generates imaginary rollouts based on the action sequence (Ty} (in the case of stochastic model) and estimate the associated value v((T) E|E y (r((s,,, dj,s]l where $5,0 is the current state and 8,,++1 f- Gj.s). The terminal value 7 (S,HH) is omitted here following the convention in the CEM planning literature but the MPC performance can be further improved if paired with accurate value predictor [Bertsekas, 2005, Lowrey et al., 2019]. The sampling distribution is then updated by fitting to the current top-k samples in terms of their value estimates v(TT), ), using the Maximum Likelihood Estimation (MLE) which solves:
N
Ø = arg max Σ
1(b(T))) 2 un) log go5)))
where Uth is the threshold equal to the value of the k-th best sample and I(-) is the indicator function. In practice, the update to the distribution parameters are smoothed by ø aσd + (1 - a' where [0, 1] is smoothing parameter that balances between the solution Eq. (2) and the parameter at the current internal iteration 1. CEM repeats this process of sampling and distribution update in an inner-loop, until it reaches the stopping condition. In practice, it stopped when either a maximum number of iterations has been reached or the parameters have not changed for few iterations. The output of CEM is an action sequence, typically set as the expectation of the most recent sampling distribution for uni-modal distributions such Gaussians E[ge) (ão, ai, .á=-1).
Choices of Sampling Distributions in CEM: A common choice is multivariate Gaussian distribution under which E2.(2) has an analytical solution. But the uni-modal nature of Gaussian makes it inadequate in solving multi-modal optimization that often occurs MBRL. To increase the capacity of the distribution, Gaussian Mixture Model (GMM) can be used [Okada and Taniguchi, 2020]. We denote such an approach as CEM-GMM. Going forward, we use CEM to refer to the vanilla version that employs Gaussian distribution. Computationally, the major difference between CEM and CEM-GMIN is that the distribution update in CEM-GMM is more computation-intent since it solves for more parameters. Detailed steps can be found in Okada and Taniguchi [2020].
3 Decentralized CEM
In this section, we first introduce the formulation of the proposed decentralized approach called the Decentralized CEM (DecentCEM). Then we illustrate the intuition behind the proposed approach using a one-dimensional synthetic multi-modal optimization example where we show the issues of the existing CEM methods and how they can be addressed by DecentCEM.
Other options are discussed in Appendix A.2
3

---

Formulation of DecentCEM DecentCEM is composed of an ensemble of M CEM instances indexed by i, each having its own sampling distributions Iø.- They can be described by set of distribution parameters & [ Each instance manages its own sampling and distribution update by the steps described in Section 2, independently from other instances.
Note that N samples and & elites are evenly split among the M instances. The top- T sample sets are decentralized and managed by each instance independently whereas the centralized approach only keeps one set of top-k samples regardless the distribution family used.
After the stopping condition is reached for all instances, the final sampling distribution is taken as the best distribution the set according to (the arg max uses a deterministic tie- breakink)
DecemiCEM = arg max Eg, [[(x]]] Rs arg max
(TT5))
(3)
where Es, [0(x]] denotes the expectation with respect the distribution go., approximated by the sample mean of samples. When M recovers the conventional CEM.
2.0
11.00
local optimum f(x)= 1.22
1.5
-1772
-1.750
1.0
-1,80
GEM
0.0
CEM GMM
DecentCEM
-1.0
-1.90
global optimum -00.mm1xxx5....... 5416
-2.0
-1.728@xx-22290
1.990%==5:148
-1,950
-7.5
-5.0
-2.5
0.0
2.5
5.0
Population Size
Figure 3: Left: The objective function in ID optimization task. Right: Comparison of the proposed DecenICEM method to CEM and CEM-GMM, wherein the line the shaded region denote mean and min/max cost from 10 independent runs. f:: solution of each method.
Motivational Example Consider one dimensional multi-modal optimization problem shown in Fig. 3 (Left): arg min, sin(x) + sin(10x; 7.5 S x < 7.5. There are eight local optima, including one global optimum f(x*) 1.9 where x² = 5.146. This objective function mimics the RL value landscape that has many local optima, as shown by Wang and Ba [2020]. This optimization problem is "easy" in the sense that grid search over the domain can get solution close to the global optimum. However, only our proposed DecentCEM method successfully converges to the global optimum consistently under varying population size (i.e., number of samples) and random runs, as shown in Fig. (Right). For a fair comparison, hyperparameter search has been conducted on all methods for each population size (Appendix A).
CEM iter=0
CEM iter=4
CEM-GMM iter=0
CEM-GMM iterz4
DecentCEM iter=0 DecentCEM iter=4
100
100
4
00414
Figure 4: How the sampling distributions evolve the ID optimization task, after the specified iteration.
Symbols include samples elites e, local optima A, global optimum 2nd row in each figure shows the weighted p.d.f of individual distribution. Population size: 200.
Both GMM-GMM and the proposed DecensCEM are equipped with multiple sampling distributions. The fact that CEM-GMM is outperformed by DecenrCEM may appear surprising. To gain some insights, we illustrate in Fig. how the sampling distribution evolves during the iterative update (more details in Fig. 10 in Appendix). CEM updated the unimodal distribution toward local optimum despite seeing the global optimum. CEM-GMM appears to have a similar issue. During MLE on the top-k samples, moved most distribution components towards the same local optimum which quickly

---

led to mode collapse. On the contrary, DecentCEM successfully escaped the local optima thanks its independent distribution update over decentralized top-k samples and was able maintain a decent diversity among the distributions.
GMM suits density estimation problems like distribution- clustering where the samples drawn from fixed true distribution that can be represented by multi-modal Gaussians. However, in CEM for optimization, exploration is coupled with density estimation; the sampling distribution in CEM is not fixed but rather gets updated iteratively toward the top-k samples. And the "true" distribution in optimization puts uniform non-zero densities to the global optima and zero densities everywhere else. When there is a unique global optimum, it degenerates into a Dirac measure that assigns the entire density to the optimum. Density estimation of such a distribution only needs one Gaussian but the exploration is challenging. In other words, the conditions for GMM to work well are not necessarily met when used as the sampling distribution in CEM. CEM-GMM is subject to mode collapse during the iterative top-k greedification, causing premature convergence, as observed in Fig 4. comparison, our proposed decentralized approach takes care of the exploration aspect by running multiple CEM instances independently, each performing its own local improvement, This is shown to be effective from this optimization example and the benchmark results in Section 6. CEM-GMM only consistently converge to the global optimum when we increase the population size to the maximum 1,000 which causes expensive computations. Our proposed DecentCEM runs more than 100 times faster than FEM-GMM at this population size, shown in Table A.3 in Appendix.
Convergence of DecentCEM The convergence of DecentCEM requires the following assumption:
Assumption 1. Let M be the number of instances in DecentCEM and each instance has sample size N:
of M where N& the total number of samples that satisfies N₂ = (kk")) where 3 >> max(0, 1- 22].. M and X are some positive constant and 0 < M < Ng V k.
Here the assumption of the total sample size follows the same one as in the CEM literature. Other such standard assumptions are summarized in Appendix H.
Theorem 3.1 (Convergence of DecentCEM), If a CEM instance described in Algorithm 3 converges, and we decentralize it by evenly dividing its sample size N₂ into M CEM instances which satisfies the Assumption 1, then the resulting DecenICEM converges almost surely to the best solution of the individual instances.
Proof. (sketch) We show that the previous convergence result of CEM [Hu et al., 2011] extends to DecentCEM under the same sample budget. The key observation is that the convergence property of each CEM instance still holds since the number of samples in each instance is only changed by constant factor, i.c.. number of instances. Each CEM instance converges to local optimum. The convergence of DecentCEM to the best solution comes from the arg max operator and applying the strong law of large numbers. The detailed proof is left to Appendix H.
4 DecentCEM for Planning
In this section, we develop two instantiations of DecentCEM for planning in MBRL where the sampling distributions are parameterized by policy networks. For the dynamics model learning, adopt the ensemble of probabilistic neural networks proposed in [Chua et al., 2018), 8]. Each network predicts the mean and diagonal covariance of a Gaussian distribution and trained by minimizing the negative log likelihood. Our work focuses on the planning aspect and refer to Chua et al. [2018] for further details on the model.
CEM Planning with a Policy Network In MBRL, CEM is applied to every state separately to solve the optimization problem stated in Eq. (1). The sampling distribution is typically initialized to fixed distribution at the beginning of every episode [Okada and Taniguchi, 2020, Pinneri et al., 2020], or more frequently at every time step [Hafner et al., 2019]. Such initialization schemes are sample inefficient since there is no mechanism that allows the information of the high-value region in the value space of one state to generalize to nearby states. Also, the information is discarded after the initialization. It is hence difficult to scale the approach to higher dimensional solution spaces, present in many continuous control environments. Wang and Ba [2020] proposed to use a policy network in CEM planning that helped to mitigate the issues above.
5

---

They developed two methods: POPLIN-A that plans in the action space, and POPLIN- P that plans in the parameter space of the policy network. In POPLIN-A, the policy network used learn to output the mean of Gaussian sampling distribution of actions. In POPLIN-P, the policy network parameters serve as the initialization of the mean of the sampling distribution of parameters. The improved policy network can then be used to generate an action. They show that when compared to the vanilla method of using a fixed sampling distribution in the action space, both modes of CEM planning with such learned distribution perform better. The same principle of combining a policy network with CEM can be applied to the DecentCEM approach as well, which we describe next.
Policy Network
CEM Instance
P1
Policy Network 2
CEM Instance
action
arg mast
Pollcy Network M-1
CEM Instance M-1
Var-1
Piu-1
Policy Network M
CEM Instance M
V3r
sbas
Training
Inference
Figure 5: The architecture of DeCentCEM planning with M CEM instances. d4 Φ; for planning in action space and 4/: 0, for planning policy network parameter space with the instance index i ,1,..., M).
DecentCEM Planning with an Ensemble of Policy Networks For better sample efficiency in MBRL setting, extend DecentCEM use an ensemble of policy networks to learn the sampling distributions in the CEM instances. Similar to the POPLIN variations, we develop two instantiations of DecentCEM, namely DecenrCEM-A and DecentCEM-P. The architecture of the proposed algorithm illustrated in Fig. 5. DecentCEMMA plans in the action space. It consists of an ensemble of policy networks followed by CEM instances. Each policy network takes the current state 8, as input, outputs the parameters di of the sampling distribution for CEM instance i in the action space. There is no fundamental difference from the DecentCEM formulation in Section 3 except that the initialization of sampling distributions is learned by the policy networks rather than a fixed distribution.
The second instantiation DecentCEM-P plans in the parameter space of the policy network. The output of each policy network is the network parameter 0z. The initial sampling distribution of CEM instance i is Gaussian distribution over the policy parameter space with the mean 0, In the arg max operation in Eq. (3), the sample Ti,j denotes the j-th parameter sample from the distribution for CEM instance i. Its value is approximated by the model modee--redicted value of the action sequence generated from the policy network with the parameters Ti.j-
The ensemble of policy networks in both instantiations Decen(CEMMA and Decen(CEMMPP are initial- ized with random weights, which is empirically found to be adequate to ensure that the output of the networks do not collapse into the same distribution (Sec. 6.2 and Appendix F).
Training the Policy Networks in DecentCEM When planning in action space, the policy networks are trained by behavior cloning, similar to the scheme in POPLIN [Wang and Ba, 2020]. Denote the first action in the planned action sequence at time step by the i-th CEM instance âr,i. the i-th policy network is trained to mimic a,, and the training objective is mino, Es, I|ae, (st) a where D; denotes the replay buffer with the state and action pairs (st, ât,i). ag, (sx) is the action prediction state 8; from the policy network parameterized by 0,-
While the above training scheme can be applied to both planning in action space and parameter space, we follow the setting parameter average (AVG) [Wang and Ba, 2020] training scheme when planning in parameter space. The parameter is updated as 0 = 0, + [D. En.D,, ò, where D; = { is dataset of policy network parameter updates planned from the i-th CEM instance previously. It is more effective than behavior cloning based on the experimental result reported by Wang and Ba [2020] and our own preliminary experiments.

---

Note that each policy network in the ensemble is trained independently from the data observed by its corresponding CEM instance rather than from the aggregated result after taking the arg max. This allows for enough diversity among the instances. More importantly, it increases the size of the training dataset for the policy networks compared to the approach taken in POPLIN. For example, with an ensemble of M instances, there would be M training data samples available from one real environment interaction, compared to the one data sample in POPLIN-A/P. As a result, DecenICEM is able to train larger policy networks than is otherwise possible, shown in Sec. 6.2 and Appendix F.
5 Related Work
We limit the scope of related works to CEM planning methods, which is one of the broad class of planning algorithms in MBRL. For review of different families of planning algorithms, the readers are referred to Wang et al. [2019]. Vanilla CEM planning in action space with a single Gaussian distribution has been adopted the planning method for both simulated and real-world robot control [Chua et al., 2018, Finn and Levine, 2017, Ebert et al., 2018, Hafner et al., 2019, Yang et al., 2020, Zhang al., 2021]. Previous attempts to improving the performance of CEM-based planning can be grouped into two types of approaches. The first type includes CEM in a hybrid of CEM+X where "X" some other component or algorithm. POPLIN [Wang and Ba, 2020] is a prominent example where "X" is a policy network that learns a state conditioned distribution that initializes the subsequent CEM process. Another common choice of "X" is gradient-based adjustment of the samples drawn in CEM. GradCEM [Bharadhwaj et al., 2020] adjusts the samples in each iteration of CEM by taking gradient ascent of the return estimate w.r.t the actions. CEM-RL [Pourchot and Sigaud, 2019] also combines CEM with gradient based updates from RL algorithms but the samples are in the parameter space of the actor network. To improve computational efficiency, Lee et al. [2020] proposes an asynchronous version of CEM-RL where each CEM instance updates the sampling distribution asynchronously.
The second type of approach aims improving CEM itself. Amos and Yarats [2020] proposes fully-differentiable version of CEM called DCEM. The key is to make the top-k selection in CEM differentiable such that the entire CEM module be trained in an end-to-end fashion. Despite cutting down the number of samples needed in CEM, this method does not beat the vanilla CEM in benchmark test. GACEM [Hakhamaneshi al., 2020] increase the capacity of the sampling distribution by replacing the Gaussian distribution with an auto-regressive neural network. This change allows CEM to perform search in multi-modal solution space but is only verified in toy examples and its computation seems too high to be scaled to MBRL tasks. Another method that increases the capacity of the sampling distribution is PaETS [Okada and Taniguchi, 2020] that uses a GMM with CEM. It is the approach that we followed for our CEM-GMA implementation. The running time results in the optimization task in Sec.3 shows that it computationally heavier than the CEM and DecenICEM methods, limiting its use in complex environments. Macua et al. [2015] proposed "distributed" CEM that similar in spirit to our method in that they used multiple sampling distributions and applied the top-k selection locally to samples from each instance. However, their instances are cooperative as opposed to being independent as in our work. They applied "collaborative smoothed projection steps" to update each sampling distribution instances including itself. The updating procedure is more complicated than our proposed method and proper network topology of the instances is needed: naive approach of updating according to all instances will lead to mode collapse since the resulting sampling distributions will be identical. The method was tested in toy optimization examples only. Overall, this second type of approach did not outperform vanilla CEM, a phenomenon that motivated our move to a decentralized formulation.
average of its neighboring
6 Experiments
We evaluate the proposed DecentCEM methods in simulated environments with continuous action space. The experimental evaluation is mainly set up to understand if DecentCEM improves the performance and sample efficiency over conventional CEM approaches.
6.1 Benchmark Setup
Environments We run the benchmark in a set of OpenAl Gym [Brockman et al., 2016] and Mu- JoCo [Todorov et al., 2012] environments commonly used in the MBRL literature: Pendulum,
7

---

InvertedPendulum
Acrobet
Reacher
11
Average Return
-12
-120
-25
3033
1873
8030
1833
1303
Steps
Steps
Steps
Walker2D
PETS Pusher
Ant
Average Return
-10
-208
-10
-30
-34
1040
Steps
Steps
Steps
DecentPETS
POPLIN-A
DecentCEM-A
DecentCEN-P
Figure 6: Learning curves of the proposed DeCentCEM methods and the baselines continuous control environments. The line the shaded region shows mean and standard error evaluation results from training runs with different random seeds.
InvertedPendullum, Cartpole, Acrobot. FixedSwimmer', Reacher, Hopper, Walker2D, HalfChee- tah, REIS--Reahhe3D, HelehhahlCh,,,thh
Pushhr, Ant. The three environments prefixed by "PETS" are proposed by Chua et al. [2018]. Note that MBRL algorithms often make different assumptions about the dynamics model or the reward function. Their benchmark environments are often modified from the original OpenAl gym environments such that the respective algorithm is runnable. Whenever possible, we inherit the same environment setup from that of the respective baseline methods. This is that the comparison against the baselines fair. More details on the environments and their reward functions are in Appendix B.
Algorithms The baseline algorithms are PETS [Chua et al., 2018] and POPLIN [Wang and Ba, 2020]. PETS uses CEM with a single Gaussian distribution for planning. The POPLIN algorithm combines single policy network with CEM. As described in Sec. 4, POPLIN comes with two modes: POPLIN-A and POPLIN-P with the suffix "A" denotes planning in action space and "p" for the network parameter space. We reuse default hyperparameters for these algorithms from the original papers not mentioned specifically. For our proposed methods, we include two variations DecentCEM-A and DecentCEM-P as described in Sec. where the suffix carries the same meaning as in POPLIN-A/P. The ensemble size of DecentCEM-A// as well as detailed hyperparameters for all algorithms are listed in the Appendix D.2. We also included Decentralized PETS, denoted by DecemPETS. All MBRL algorithms studied in this benchmark use the same ensemble networks proposed by Chua et al. [2018) 8] for model learning. And model-free RL baseline SAC [Haarnoja et al., 2018] was included.
Evaluation Protocol The learning curve shows the mean and standard error of the test performance out of 5 independent training runs. The test performance is an average return of 5 episodes of the evaluation environment, evaluated every training episode. At the beginning of each training run, the evaluation environment is initialized with a fixed random seed such that the evaluation environments consistent across different methods and multiple runs make it fair comparison. All experiments were conducted using Tesla V100 Volta GPUs.
6.2 Results
Learning Curves The learning curves of the algorithms are shown in Fig. 6 for InvertedPendullu. Acrobot, Reacher, Walker2D, E-TS-Pusher and Ant, sorted by the state and action dimensionality
la modified version of the original Gym Swimmer environment where the velocity sensor on the neck is moved to the head. This fix proposed by Wang and Ba [2020]
8

---

of task. The curves are smoothed with sliding window of 10 data points. The full results for all environments are included in Appendix E. We can observe two main patterns from the results. One pattern was that in most environments, the decentralized methods DecentPETTS, DeceniCEM-A/P either matched or outperformed their counterpart that took a centralized approach. In fact, the former can be seen as generalization of the later, by an additional hyperparameter that controls the ensemble size with size one recovering the centralized approach. The optimal ensemble size depends on the task. This additional hyperparameter offers flexibility in fine-tuning CEM for individual domains. For instance in Fig. 6, for the P-mode where the planning is performed policy parameter space, ensemble size of larger than works better in most environments while an ensemble size of one works better in Ant. The other pattern was that using policy networks to learn the sampling distribution in general helped improving the performance of centralized CEM but not necessarily in decentralized formulation. This is perhaps due to that the added exploration from multiple instances makes it possible to identify good solutions in environments. Using policy net in such case may hinder the exploration due to overfitting to previous actions.
Ablation Study A natural question to ask about the DecentCEM-A/P methods is whether the increased performance from the larger number of neural network parameters. We added two variations of the POPLIN baselines where a bigger policy network was used. The number of the network parameters was equivalent to that of the ensemble of policy networks in DecentCEM-A/P. We show the comparison in Reacher(2) and PETS- Pusher(7) (action dimension in parenthesis) in Fig. 7. In both action space and parameter space planning, a bigger policy network in POPLIN either did not help or significantly impaired the performance (see the POPLIN-P results in reacher and PETS- Pusher). This is expected since unlike CEM, the training data in POPLIN do not scale with the size of the policy network, as explained at the end of Sec. 4.
Reacher
PETS-Pusher
-10
Average Return
-15
POPLIN-A
POPLIN-A- BiggeellBiggrolllccyeettt
DecentCEM-A
POPLIN-P
-25
160
POPLIN-P-BiggeePPliiyeee
-30
DeCEntCEM-P
35
200
2000
4000
6000
8000
10000
0
2500
5000
7500 10000 12500 15000 Steps
Steps
Figure 7: Ablation study on the policy network size where POPLIN-A&P have a bigger policy network equivalent in the total number of network weights to their DecenCCMM counterparts.
Next, we look into the impact of ensemble size. Fig. 8 shows the learning curves of different ensemble sizes in the Reacher environments for action space planning (left), parameter space planning (middle) and planning without policy networks (right). Since we fix the total number of samples the same across the methods, the larger the ensemble size is, the fewer samples that each instance has access to. As the ensemble size goes up, exploration by more instances. Varying this hyperparameter allows us to find the best trade-off, as can be observed from the figure where increasing the ensemble size beyond sweet spot yields diminishing or worse returns.
trading off the accuracy of the sample mean for better
Reacher
Reacher
Reacher
Average Return
-28
-32
-27
-12
DecentZEM-A DecentCEM E2
DecentCEM-
DecentPETS DecentPETS E2
-:44
DecentCEM P-E2
DeceniCEN-A.EE
DecentCEK.R-E3
DecentRETS-E3
DecenICEM-A-E5
DecentCEM-P-gS
-12
DecentPETS-ES DecentPETS 56
D6
DecentCEM P.E6
-20
Steps
Steps
Steps
Figure 8: Ablation study on the ensemble size in DecenICEM-AN and DecentPETS, e.g. E2 denotes an ensemble with 2 instances. The total number of samples during planning are the same across all variations.

---

Pendulum
Pendulum
Dairwite Distances Actions
Pendulum
CEM Instance 1
7.4
Selection Ratio
0.5
Instance
1.2
0.5
CEM Instance 3
1.0
0.4-
CEM Instance 4
Action
0.8
CEM Instance 5
-0.25
0.6
0.50
0.1
-0.75
0.2
0.0
2000
4000
6000
8000
10000
-1.
3300
9500
9600 Steps
9700
0300
9500
9600
97700
Steps
Steps
(a) Selection Ratios
(b) Action Statistics
(c) Action Distance Statistics
Figure 9: Ablation of ensemble diversity in Pendulum during training of DecenICEM-A with instances. (a) Cumulative selection ratio of each CEM instance. (h)(c) Statistics of the actions and pairwise action distances of the instances, respectively. The line and shaded region represent the mean and min/max.
Figure (()) shows the cumulative selection ratio of each CEM instance during training of DecentCEM- A with an ensemble size of 5. It suggests that the random initialization of the policy network is sufficient to avoid mode collapse. We also plot the actions and pairwise action distances of the instances in Figure 9(b) and 9(c). For visual clarity, we show a time segment toward the end of the training rather than all the 10k steps. DEcenICEM-A has maintained enough diversity in the instances even toward the end of the training. DecenI and Decen(PE7S share similar trends. The plots of their results along with other ablation results are included in Appendix F.
7 Conclusion and Future Work
In this paper, we study CEM planning in the context of continuous-action MBRL. We propose a novel decentralized formulation of CEM named DecenICEM, which generalizes CEM to run multiple independent instances and recovers the conventional CEM when the number of instances is one. We illustrate the strengths of the proposed DecentCEM approach in motivational one-dimensional optimization task and show how it fundamentally differs from the CEM approach that uses Gaussian or GMM. We also show that DecentCEM has almost sure convergence a local optimum. We extend the proposed approach to MBRL by plugging in the decentralized CEM into three previous CEM-based methods: PETS, POPLIN-A, POPLIN-P. We show their efficacy in benchmark control tasks and ablations studies.
There is a gap between the convergence result and practice where the theory assumes that the number of samples grow polynomially with the iterations whereas a constant sample size is commonly used in practice including our work. Investigating the convergence properties of CEM under a constant sample size makes an interesting direction for future work. Another interesting direction to pursue is finite-time analysis of CEM under both centralized and decentralized formulations. In addition, the implementation has room for optimization: the instances currently run serially but can be improved by a parallel implementation take advantage of the parallelism of the ensemble.
Acknowledgments and Disclosure of Funding
We would like to thank the anonymous reviewers for their time and valuable suggestions. Zichen Zhang would like to thank Jincheng Mei for helpful discussions in the convergence analysis and Richard Sutton for raising a question about the model learning. This work was partially done during Zichen Zhang's 's internship at Huawei Noah's Ark Lab. Zichen Zhang gratefully acknowledges the financial support by an NSERC CGSD scholarship and an Alberta Innovates PhD scholarship. He is thankful for the compute resources generously provided by Digital Research Alliance of Canada (and formerly Compute Canada), which is sponsored through the accounts of Martin Jagersand and Dale Schuurmans. Dale Schuurmans gratefully acknowledges funding from the Canada CIFAR AI Chairs Program, Amii and NSERC.
10

---

References
A. Agarwal, N. Jiang, and S. M. Kakade. Reinforcement learning: Theory and algorithms. 2019.
B. Amos and D. Yarats. The differentiable cross-entropy method. In International Conference on Machine Learning, pages 291-302. PMLR. 2020.
D. P. Bertsekas. Dynamic programming and optimal control 3rd edition, volume i. Belmont, MA: Athena Scientific, 2005.
H. Bharadhwaj. K. Xie, and F. Shkurti. Model-predictive control via cross-entropy and gradient-based optimization. In Learning for Dynamics and Control, pages 277-286. PMLR, 2020.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.1 01540, 2016.
K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 4779-4770, 2018.
P.-T. De Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00568 2018.
C. Finn and S. Levine. Deep visual foresight for plamning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2786-2793. IEEE, 2017.
T. Haamoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
D. Hafner, T. Lillicrap. I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555-2565. PMLR, 2019.
K. Hakhamaneshi, K. Settaluri, P. Abbeel, and V. Stojanovic. Gacem: Generalized autoregres- sive eross entropy method for multi-modal black box constraint satisfaction. arXiv preprint 2rXiv:2002.07236. 2020.
J. Hu, P. Hu, and H. S. Chang. A stochastic approximation framework for class of randomized optimization algorithms. IEEE Transactions on Automatic Control, 17(1): 2011.
K. Lee, B.-U. Lee, U. Shin, and I. S. Kweon. An efficient asynchronous method for integrating evolutionary and gradient-based policy search. Advances in Neural Information Processing Systems, 33, 2020.
K. Lowrey, A. Rajeswaran, S. Kakade, E. Todorov, and I. Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. In International Conference Learning Representations, ICLR, 2019.
S. V. Macua, S. Zazo, and J. Zazo. Distributed black-box optimization of nonconvex functions. In 2015 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 3591-3595. IEEE, 2015.
S. Mannor, R. Y. Rubinstein, and Y. Gat. The cross entropy method for fast policy search. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 512- 519, 2003.
M. Okada and T. Taniguchi. Variational inference mpe for bayesian model-based reinforcement learning. Conference on Robot Learning, pages 258 272.272.22 PMLR, 2020.
C. Pinneri, S. Sawant, Blaes, Achterhold, J. Stueckler, M. Rolinek, and G. Martius. Sample- efficient cross-entropy method for real-time planning. arXiv preprint 2008 06389, 2020.

---

A. Pourchot and O. Sigaud. CEM-RL: combining evolutionary and gradient-basee based methods for policy search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033 IEEE, 2012.
T. Wang and J. Ba. Exploring model-based planning with policy networks. In 8th International Conference on Learning Representations, ICLR, Addis Ababa, Ethiopia, April 26-30, 2020.
T. Wang, X. Bao, I. Clavera, J. Hoang, Y. Wen, E. Langlois, S. Zhang. G. Zhang, P. Abbeel, and J. Ba. Benchmarking model-based reinforcement learning. CoRR, abs/1901 000000.0057.... 2019.
Y. Yang, K. Caluwaerts, A. Iscen, T. Zhang, J. Tan, and V. Sindhwani. Data efficient reinforcement learning for legged robots. In Conference Robot Learning, pages 1-10. PMLR, 2020.
B. Zhang, R. Rajan, L. Pineda, N. Lambert, A. Biedenkapp. K. Chua, F. Hutter, and R. Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In International Conference Artificial Intelligence and Statistics, pages 4015-4022 PMLR, 2021.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you deseribe the limitations of your work? [Yes] Limitations are discussed in Sec. 7 and Appendix G.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including thcoretical results...
(a) Did you state full set of assumptions of theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Ye]] See Section Appendix A, B and D.
(c) Did you report error bars (e.g.. with respect to the random seed after running experiments multiple times)? [Yes] See Section 6 and Appendix A.
(d) Did include the total amount of compute and the type of resources used (e.g.. type of GPUs. internal cluster, cloud provider)? [Yes] See section 6.1.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] The assets (Gym, mujaco) are cited and well
known.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're us- ing/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you descrihe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants
amount spent on
participant compensation? [N/A]
12


Reviews:
[{"review_id": "x6sFyuDZ0O", "reviewer": "Reviewer_PaJC", "summary": "Cross-Entropy Method (CEM) is commonly used for planning in model-based reinforcement learning (MBRL) where the *centralized* approach is typically utilized to update the sampling distribution based on only the top-*k*  operations' results on samples. In this paper, the authors show that such a *centralized* approach makes CEM vulnerable to local optima, thus impairing its sample efficiency. To tackle this issue,  the authors propose **Decent**ralized **CEM (DecentCEM)**, a simple but effective improvement over classical CEM, by using an ensemble of CEM instances running independently from one another, and each performing a local improvement of its own sampling distribution. The theoretical and empirical analysis are provided to demonstrate the effectiveness of this simple *decentralized* approach. It is shown that compared to the classical centralized approach using either a single or even a mixture of Gaussian distributions, their DecentCEM finds the global optimum much more consistently thus improving the sample efficiency. Furthermore, they plug in their DecentCEM in the planning problem of MBRL and evaluate their approach in several continuous control environments, with comparison to the state-of-art CEM-based MBRL approaches (PETS and POPLIN). Results show sample efficiency improvement by simply replacing the classical CEM module with their DecentCEM module, while only sacrificing a reasonable amount of computational cost, which can be alleviated by using the advantage of the parallelism of their ensemble method. Lastly, they conduct ablation studies for more in-depth analysis.", "questions": "1. Could you please add **DescentPETS** as the baseline to make the experiment more comprehensive? **DescentPETS** means replacing the CEM module in the previous PETS algorithm with the DescentCEM proposed in this paper.\n\n2. Could you please investigate the effectiveness of the **DecentCEM** both theoretically and empirically when assumption 2 is alleviated?\n\n3. Could you please visualize the planning trajectories with **CEM** and *DecentCEM** and compare them if possible?", "limitations": "1. We can notice that the theoretical convergence result does not match the empirical observation well. Please investigate this deeply and explore more effective methods to improve the algorithm's practical convergence result.\n\n2. Could you please investigate the theoretical convergence result of your proposed algorithm under the *constant sample size* constraint as what you have used in your experiment?\n\n3. Could you please explore how to promote your method to other non-CEM-based MBRL algorithms?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The proposed is simple but effective. This work can be utilized in any CEM-based model-based reinforcement learning algorithm.\n2. The paper is well organized and the writing is good. \n3. The theoretical and empirical analysis are persuasive to demonstrate the effectiveness of the proposed method.\n\n\n**Weaknesses**\n1.  The assumptions of the proposed algorithm are a little strong, especially the second point. The authors assume that the reward function is known and deterministic which is not common in practice actually.\n\n2. The visualization part in the experiment still needs to be improved.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nFZLLQakyhD", "reviewer": "Reviewer_H4mJ", "summary": "This paper propose a multi-instance CEM method, a strict generalization of CEM, which is able to remain spread-out in the search space. The authors test on continuous-action model-based RL.\n", "questions": "Footnote 2 seems incomplete.\n\nHow is BiggerPolicyNet tuned in the ablation study?\n\nHow can DecenCEM be used in settings other than continuous-action MBRL?\n\nHow does the performance compare to gradient based methods?\n", "limitations": "+ The authors are upfront about their setting (continuous action, requires a model, etc.).\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method uses common optimization philosophy and have shown good performance; and the analysis is sound. The paper is overall well written and have included an example for intuition. \n\nThe method design principled and given theoretical attention.\n\nHowever, the empirical evaluation hardly distinguish the method from the baselines, that is, the performance gain appears to be minimal.\n\nThe novelty of the proposed method is not very significant; however, it can be considered ground work for many future works.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "98obJFRocNl", "reviewer": "Reviewer_cLuK", "summary": "This paper proposes DecentCEM, where two or more instances of cross-entropy method (CEM) are run independently with different sets of samples until convergence. For model-based decision making, the results from the ensemble are gathered and combined by choosing the best sampling distribution that maximizes the expected return of trajectories. Theoretically, they show that each instance of CEM converges under the same set of assumptions for convergence of vanilla CEM. They combine DecentCEM with CEM planning and empirically show that DecentCEM outperforms baseline methods including PETS and POPLIN in multiple environments for MBRL.", "questions": "- I would like to check out the details and a more thorough version of Table A.3. How many instances or mixture components were employed for the table?\n- Also, please see \"Strengths And Weaknesses\" for additional relevant comments.", "limitations": "I think one main limitation of the method is fairly covered, regarding the discrepancy between the theoretical result and practice.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality\n\n- In terms of the originality, the idea of running multiple instances of the same algorithm and combining the results is not very novel, as it is one of the common patterns in problem solving.\n\nQuality\n\n- The method is sound, and I agree that it could bring improvements in multiple situations.\n- I think Theorem 3.1 (Convergence of DecentCEM) needs to be more specific about the \"almost-sure\" convergence of the solution of DecentCEM. Even if they maintain the expression \"almost surely\", at least they should state that it holds for large-enough sample sizes.\n\nClarity\n\n- Overall, the manuscript is easy to follow and well-organized. Especially, the early introduction of the motivating examples can help better understanding of the paper.\n- Some parts of the paper could mislead readers and possibly oversell the results of the paper. For instance, in the abstract, the authors mention that they \"provide both theoretical and empirical analysis\" and the next sentence states that they \"show\" that \"DecentCEM finds the global optimum much more consistently\". This could easily make readers think that such statement is backed by theoretical results as well, which is not true according to my understanding.\n- Regarding Theorem 3.1 (Convergence of DecentCEM), please see my comment above on quality.\n\nSignificance\n\n- Even though the proposed decentralized CEM is not fully novel algorithmically, suggesting that such approach yields valid results with potential improvements can still be somewhat meaningful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-yJYA9W8kdZ", "reviewer": "Reviewer_XxHs", "summary": "This paper proposed a decentralised version of the cross-entropy method for model-based reinforcement learning. The convergence of the proposed algorithm is analysed under several assumptions (Hu et al. [2011]) and evaluated in several continuous control environments compared with PETS and POPLIN. ", "questions": "1. Where is the definition of the initial sampling distribution $g_{\\phi}(\\tau)$? The first time it appears is in line 91. It is difficult for readers unfamiliar with planning (MPC) to proceed.\n\n2. What is $d_a$ in line 93?\n\n3. Theorem 3.1 is entirely broken. Where are all the assumptions? It is not pleasant to just say, “The detailed proof is left to Appendix H”.\n", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1.  Decentralization of CEM in MBRL can deviate from local optima and improve sample efficiency over its centralisation counterpart.\n2. A relatively complete theoretical and empirical analysis and in-depth ablation studies.\n3. Evaluating planning tasks in different continuous control environments are nicely done.\n\n**Weakness: **\nThe readability is impossible without referring appendix with many cross-reference links broken.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept