Decentralized Cross-Entropy Method for Model-Based Reinforcement Learning
Authors: Zichen Zhang
Jun Jin
Martin Jagersand
Jun Luo
Dale Schuurmans
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement Learning
Cross-Entropy Method
Planning
Model-Based RL

Abstract: Cross-Entropy Method (CEM) is a popular approach to planning in model-based reinforcement learning.
It has so far always taken a \textit{centralized} approach where the sampling distribution is updated \textit{centrally} based on the result of a top-$k$ operation applied to \textit{all samples}.
We show that such a \textit{centralized} approach makes CEM vulnerable to local optima and impair its sample efficiency, even in a one-dimensional multi-modal optimization task.
In this paper, we propose \textbf{Decent}ralized \textbf{CEM (DecentCEM)} where an ensemble of CEM instances run independently from one another and each performs a local improvement of its own sampling distribution.
In the exemplar optimization task, the proposed decentralized approach DecentCEM finds the global optimum much more consistently than the existing CEM approaches that use either a single Gaussian distribution or a mixture of Gaussians.
Further, we extend the decentralized approach to sequential decision-making problems where we show in 13 continuous control benchmark environments that it matches or outperforms the state-of-the-art CEM algorithms in most cases, under the same budget of the total number of samples for planning.

Under review as a conference paper at ICLR 2022
DECENTRALIZED CROSS-ENTROPY METHOD FOR MODEL-BASED REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Cross-Entropy Method (CEM) is a popular approach to planning in model-based reinforcement learning. It has far always taken centralized approach where the sampling distribution is updated centrally based on the result of a top-k oper- ation applied all samples. We show that such centralized approach makes CEM vulnerable to local optima and impairs its sample efficiency, even in a mee-dienniional modtil optimization task. In this paper, we propose De- centralized CEM (DecentCEM) where an ensemble of CEM instances run inde- pendently from one another and each performs a local improvement of its own sampling distribution. In the exemplar optimization task, the proposed decentral- ized approach DecentCEM finds the global optimum much more consistently than the centralized CEM approaches that use either single mixture of Gaussian distributions. Also, we show that DecentCEM is theoretically sound. Further, we extend the decentralized approach to sequential decision- decision-making making problems where we show in several continuous control benchmark environments that it provides effective mechanism improve the performance of CEM algorithms, under the same sample budget for planning.
INTRODUCTION
Model-based reinforcement learning (MBRL) uses model as a proxy the environment for plan- ning actions in multiple steps. This paper studies planning in MBRL with specific focus on the Cross-Entropy Method (CEM) (De Boer al. 2005: Mannor et al.) 2003). which popular in MBRL due to its ease of use and strong empirical performance (Chua et al. 2018 Hafner et al.) 2019 Wang Ba 2020 Zhang et al.) 2021; Yang et al. 2020). CEM Tree optimization method. It uses sampling distribution to generate imaginary trajectories of environment-agem interactions with the model. These trajectories are then ranked based on their returns computed from the rewards given by the model. The sampling distribution is updated to increase the likelihood of producing the top-k trajectories with higher returns. These steps are it- erated and eventually yield an improved distribution the action sequences guide the action execution in the real environment.
stochastic, derivative-
Despite the strong empirical performance of CEM for planning, it is prone to two problems: (1) lower sample efficiency as the dimensionality of solution space increases, and (2) the Gaussian dis- tribution that is commonly used for sampling may cause the optimization get stuck in local optima of mulli-modaa solution spaces commonly seen in real-world problems. Previous works addressing these problems either add gradient- based updates of the samples to optimize the parameters of CEM, or adopt more expressive sampling distributions, such as using Gaussian Mixture Model (Okada & TTanguucii 2020) or masked auto-regressive neural network (Hakhamaneshi et al.) (020). Neverthe- less, all CEM implementations to date are limited to centralized formulation where the ranking step involves all samples. As analyzed below and in Section 3 such a centralized design makes CEM vulnerable to local optima and impairs its sample efficiency.
We propose Decentralized CEM (DecentCEM) to address the above problems. Rather than ranking all samples, in the centralized design, our method distribute the sampling budget across semble of CEM instances. These instances run independently from one another, and each performs local improvement of its sampling distribution based on the ranking of its generated samples.

---

Under review as a conference paper at ICLR 2022
The best action is then aggregated by taking an arg max among the solution of the instances. It recovers the conventional CEM when the number of instance is one.
We hypothesize that by shifting to this decentralized design, CEM can be less susceptible to premature convergence caused by the centralized ranking step. As illustrated Fig. [ the centralized sampling dis- tribution exhibits bias toward the sub-optimal so- lutions near top right, due to the global top-k rank- ing. This bias would occur regardless of the family of distributions used. In comparison, a decentralized approach could maintain enough diversity thanks to its local top-k ranking in each sampling instance.
(a) Centralized CEM (b) Decentralized CEM
Through one-dimensional multi- modal optimiza- tion problem in Section g we show that DecentCEM empirically finds the global optimum more consis- tently than centralized CEM approaches that use ei- ther a single Gaussian or a mixture of Gaussian dis- tributions. Also we show that DecentCEM is the- oretically sound that it converges almost surely to local optimum. We further apply DecentCEM sequential decision making problems and use neu- ral networks to parameterize the sampling distribu- tion in each CEM instance. Empirical results in sev- eral continuous control benchmarks suggest that De- centCEM offers an effective mechanism to improve the sample efficiency over the baseline CEM methods under the same sample budget for planning.
Figure 1:
Illustration of CEM approaches in optimization. Shades of red indicate rela- tive value of the 2D optimization landscape: brighter is better. Optimal solutions are near bottom left corner of the solution space. Blue dots are top-k samples, and black dots are other samples. Open dots O represent the sampling distributions with size of dots indi- cating number of generated samples.
2 PRELIMINARIES
We consider Markov Decision Process (MDP) specified by (S..A.R.P.y.d....) S C RR is the state space, C Rdd is the action space. R: x A + R the reward function that maps a state and action pair to real-valued reward. P(s'|s, a) S x A x S + R+ is the transition probability from a state and action pair 8, a to the next state 8'. 7 E [o, 1] is the discount factor. dg denotes the distribution of the initial state sp- At time step t, the agent receives state st aad takes an action a, according to policy (-1ss) () that maps the state to a probability distribution over the action space. The environment transitions to the next state 81+1
P(-|st, az) and gives a reward
R(se, ae)
to the agent The return G, E 7'
sum of discounted reward within an episode length of T. The agent aims to find policy π that maximizes the expected return. We denote the learned model in MBRL as [- a)), which is parameterized by w and approximates P(-[s, a).
Planning with the Cross Entropy Method Planning MBRL is about leveraging the model to find the best action in terms of its return. Model-Predictive-CC Control (MPC) performs online planning at each time step up to a horizon to find the optimal action sequence:
TMPC (8L) = arg max B[_ "arlll"'''''rssrae ('r(()45)))4)) + +>n))>>VVV(() V*H))
(1)
Qce---1
where is the planning horizon, 0=:+H-1 denotes the action sequence from time step f to to H -1, and V(HH) ) is the terminal value function at the end of the planning horizon. The first action in this sequence is executed and the rest are discarded. The agent then re-plans at the next time step.
The Cross-Entropy Method (CEM) is gradient-free optimization method that can be used for solv- ing Eg. T The workflow is shown in Fig. 2 CEM planning starts by generating N samples N [(ā3,0: ((ā..o:0..1." ... - N from initial sampling distribution go (T) parameterized
assume observability, i.c., we assume that the agent has access to the state. We assume that the agent receives the true reward. This makes the problem casier hut is unfair to MBRI. methods that do assume this. Thus, we will limit comparison methods that also make this assumption.
2

---

Under review as a conference paper at ICLR 2022
distribution
parametars
Sampling Distribution
Samples
Model
Top-k Samplas
action
(a a B
Ca. Gp. Mir-1)
4g
CEM internal Loop
CEM
Figure 2: Cross Entropy Method (CEM) for Planning in MBRI.
by o, where each sample Ty is action sequence from the current time step up the planning horizon H. The domain (T) has a dimension of d, d H.
Using model f. CEM generates imaginary rollouts based the action sequence {ry} (in the case of a stochastic model) and estimate the associated value o(7;) - (r ag,a]] where 8j.0 the current state and 8j,++1 [( (3,)), The terminal value 2 (s),tr) is omitted here following convention in the CEM planning literature but the MPC performance can be further improved if paired with an accurate value predictor (Bertsekas) [2055] Lowrey et al. 2019). The sampling distribution is then updated by fitting to the current top-A samples in terms of their value estimates (T;)), using the Maximum Likelihood Estimation (MIE) which solves:
ø' arg max Σ 1(v(T)) 2 Un) log (T())
where is the threshold equal to the value of the k-th best sample and 1.)) is the indicator function. In practice, the update to the distribution parameters are smoothed by 6 ad' + (1 aa)'' where [0, is smoothing parameter that balances between the solution Eq. (2) and the parameter at the current internal iteration 1. CEM repeats this process of sampling and distribution update in an inner-loop, until it reaches the stopping condition. In practice, it is stopped when either maximum number iterations has been reached or the parameters have not changed for a few iterations. The output of CEM is action sequence, typically set as the expectation of the most recent sampling distribution for uni-modal distributions such as Gaussians jú E(ga) (ãp, â,,- ... .i.-
Choices of Sampling Distributions in CEM A common choice of the sampling distribution is multivariate Gaussian distribution under which Eq.(2) has straight- forward analytical solution. But the uni-modal nature of Gaussian makes it inadequate in solving muulti-modll optimization that often occur MBRL. To increase the capacity of the distribution, Gaussian Mixture Model (GMM) can be used (Okada & Taniguchi] 2020). We denote such an approach as CEM-GMM. Going forward, we use CEM to refer the vanilla version that uses a Gaussian distribution. Computationally, the major difference between CEM and CEM-GMM is that the distribution update in CEM-GMM involves solving for more parameters. Detailed steps can be found in Okada & Taniguchi (2020).
3 DECENTRALIZED CEM
In this section, we first introduce the formulation of the proposed decentralized approach called the Decentralized CEM (DecentCEM). Then we illustrate the intuition behind the proposed approach using a one-dimensional synthetic multi-modal optimization example where we show the issues of the existing CEM methods and how they can be addressed by DecentCEM.
Formulation of DeCentCEM DecentCEM is composed of an ensemble of multiple CEM in- stances indexed by each having its own sampling distributions Io.- They can be described by set of distribution parameters & tribution update by the steps described in Section 2 independently from other instances. Note that the number of samples and elites are evenly split among the M instances. The top- M sample sets are decentralized and managed by each instance independently whereas the centralized approach only keeps one set of top-k samples regardless of the distribution family used, After the stopping
= Each instance i manages its own sampling and dis-
&
OOteer options are discussed in Appendix A.2
3

---

Under review as a conference paper at ICLR 2022
condition is reached for all instances, the final sampling distribution is taken the best distribution in the set & according to (the arg max uses a deterministic tie- seebreaking):
DeceniCEM = arg max Esp, [v((:)] arg max
((T())
(3)
0.E0
0 /=1
where o [u(x)] denotes the expectation with respect to the distribution Po.- approximated by the sample mean. When M recovers the conventional CEM method.
-1763
local optimum f100=1 1tt.-11.28xxx22222
-1.724
Cost (2
1.403
GEM
11402 1
CBM GVN
73377xx77
1,492
DoconcCBM
-100
40. 1.008- 223
global optimum fix)= 1.9
728 2.296
300
200
306
1000
Population Size
Figure 3: Left: The objective function in a ID optimization task. Right: Comparison of our proposed DecentCEM method CEM and CEM-GMM, wherein the line and the shaded region denote the mean and the min/max cost from 10 independent runs. è: resulting solution of each method.
Motivational Example Consider a one dimensional multi-modal optimization problem shown in
Fig 3 (Left). There are eight local optima, including one global optimum f(x*) 119 where x 5.146. This objective function mimics the RL value landscape that has many local optima, as shown by Wang & Ba (2020). This optimization problem "eaay" in the sense that grid search over the domain can get us a solution close to the global optimum. However, only our proposed DecentCEM method successfully converges to the global optimum consistently under varying population size (i.e., number of samples) and random runs, as shown in Fig 3 (Right
Both CEM-GMM and the proposed DecenICEM are equipped with multiple sampling distributions. The fact that CEM-GMM is outperformed by DecentCEM may appear surprising. To gain some insights, we illustrate in Fig. how the sampling distribution evolves during the iterative update (more details in Fig. 9 in Appendix). CEM updated the unimodal distribution toward a local op- timum despite seeing the global optimum. CEM-GMM appears to have a similar issue. During MLE the top-k samples, it moved most distribution components towards the same local optimum which quickly lead to mode collapse. On the contrary, DecentCEM successfully escaped the local optima thanks to its independent distribution update over decentralized top-k samples and was able to maintain decent diversity among the distributions.
CEM 0
CEM iter=4
CEM GMM Iter=0
CEM-GMM iter=4
DecentCEM iter=0 DecentCEM iter=
100
+ 100
Figure 4: How the sampling distributions evolve the ID optimization task, after the specified iteration. Symbols include samples clites local optima global +. 2nd row in each figure shows the weighted p.d.f of individual distribution. Population size: 200.
GMM suits density estimation problems like distribution-based clustering where the samples are drawn from a fixed true distribution that can be represented by multi-modal Gaussians. However,
"For a fair comparison, hyperparameter search has heen conducted on all three methods for each population size (Appendix A
4

---

Under review as a conference paper at ICLR 2022
in CEM for optimization, exploration is coupled with density estimation: the sampling distribution in CEM is not fixed but rather gets updated iteratively toward the top-k samples. And the "true" distribution in optimization puts uniform non-zero densities to the global optima and zero densities everywhere else. When there is a unique global optimum, it degenerates into a Dirac measure that assigns the entire density to the optimum. Density estimation of such distribution only needs one Gaussian but the exploration is challenging. In other words, the conditions for GMM to work well are not necessarily met when used as the sampling distribution in CEM. CEM-GMM is subject to mode collapse during the iterative top-k greedification, causing premature convergence, as observed in Fig 4 In comparison, our proposed decentralized approach takes care of the exploration aspect by running multiple CEM instances independently, each performing its own local improvement. This shown to be effective from this optimization example and the benchmark results in Section 6 CEM-GMM only consistently converge to the global optimum when we increase the population size to the maximum 1,000 which causes expensive computations. Our proposed DecentCMM runs more than 100 times faster than CEM-GMM at this population size, shown in Table A.3 in Appendix.
Convergence of DeCentCEM We state the convergence result of DecentCEM in Theorem 3.1 We show that the previous convergence result of CEM (Hu et al.. (011) applies to DecentCEM under the same sample budget. The key observation is that the convergence property of each CEM instance still holds since the number of samples in each instance is only changed by constant factor (the
number of instances). We leave the detailed proof to appendix H
Theorem 3.1 (Convergence of DecenCEMM), If Assumptions 1-5 hold for CEM instance described in Algorithm and we decentralize it by evenly dividing its sample size N₁. into M CEM instances in DecentCEM algorithm that satisfies Assumption 6, then the sequence of iterates {Na,s generated by each CEM instance indexed by i converges almost surely to an internally chain recurrent set (Hirsch) et al. 2001) of Equation 7 Furthermore, the solution of DecentCEM x converges almost surely to the best solution of the individual instances in terms of the expected value of [V (x)).
4 DECENTCEM FOR PLANNING IN MBRL
In this section, we develop two instantiations of DecentCEM for planning in MBRL where the sampling distributions are parameterized by policy networks.
CEM Planning with a Policy Network In MBRL, CEM is applied to every state separately to solve the optimization problem stated in Eq. ( The sampling distribution is typically initialized to fixed distribution at the beginning of every episode (Okada & Taniguchi] 2020, Pinneri et al.) 2020), or more frequently at every time step (Hafner et 2019). Such initialization schemes are sample inefficient since there is mechanism that allows the information of the high-value region in the value space of one state to generalize to nearby states. Also, the information is discarded after the initialization. It is hence difficult to scale the approach to higher dimensional solution spaces, present in many continuous control environments. Wang & Ba (2020) proposed to use a policy network in CEM planning that helped to mitigate the issues above. They developed two methods: POPLIN-A that plans in the action space, and POPLIN-P that plans in the parameter space of the policy network. In POPLIN-A, the policy network is used to learn to output the mean of Gaussian sampling distribution of actions. In POPLIN-P, the policy network parameters serve as the initialization of the mean of the sampling distribution of parameters. The improved policy network can then be used to generate an action. They show that when compared to the vanilla method of using a fixed sampling distribution in the action space, both modes of CEM planning with such a learned distribution perform better. The same principle of combining policy network with CEM can be applied to the DecentCEM approach as well, which we will describe next.
DecentCEM Planning with an Ensemble of Policy Networks For better sample efficiency in MBRL setting, we extend DecentCEM to use an ensemble of policy networks to learn the sampling distributions in the CEM instances. Similar to the POPLIN paper, develop two instantiations of DecenrCEM, namely DecentCEM-A and DecemCCEM-P. P. The architecture of the proposed algorithm is illustrated in Fig. [
DecenICEM-A plans in the action space. It consists of an ensemble of policy networks fol- lowed by CEM instances. Each policy network takes the current state St input, out-
5

---

Under review as a conference paper at ICLR 2022
puts the parameters 8, of the sampling distribution for CEM instance i. There is fun- damental difference from the DecentCEM formulation in Section tion of sampling distributions is learned by the policy networks rather than fixed distribution.
3 except that the initializa-
The second instantiation DecentCEM-P plans in the parameter space of the policy network. The initial sampling distribution is Gaussian distribution over the policy parameter space with the mean at the cur- rent parameter values. In the arg max op- eration Eq. (3), the sample Ti,j denotes the parameters of the policy network. Its
Policy Network
CEM Instance
state
Policy Network
CEM Instance 2
org mag
Policy Network M-1
CEM Inalance M-1
Pac-a
NatworkM
CEM Instance M
PM
value obtained by computing the value of the action sequence generated from the policy network with the parameters Tz.
Traihing
Inference
The ensemble of policy networks in both instantiations DecentCEM-A and DecenrCEM-P are initialized with random weights, which is empirically found to be adequate ensure that the output of the networks do not collapse into the same distribution (Sec. (Sec.6 and Appendix F.
Figure 5: DecentCEM planning architecture. 1/4 = Φ for planning in action space and d/ 0, for planning in policy network parameter space.
Training the Policy Network in DecentCEM When planning in action space, the policy networks are trained by behavior cloning, similar to the scheme in POPLIN (Wang & Ba 2020). Denote the first action the planned action sequence at time step / by the i-th CEM instance as ât,i the i-th policy network is trained to mimic ât,, and the training objective is ming, Be, ((oo ما,يll where D, denotes the replay buffer with the state and action pairs (st, ât,z). ao, (5t) is the action prediction at state st from the policy network parameterized by 0;.
While the above training scheme be applied to both planning in action space and parameter space, we follow the setting parameter average (AVG) (Wang & Ba 2020) training scheme when planning in parameter space. The parameter is updated as 0, 0,+ pb. ES,D,, 6, where D, {6.) a dataset of policy network parameter updates planned from the i-th CEM instance previously. is more effective than behavior cloning based on the experimental result reported by Waang & Ba 2020) and our own preliminary experiments.
Note that each policy network in the ensemble is trained independently from the data observed by its corresponding CEM instance rather than from the aggregated result after taking the arg max. This allows for enough diversity among the instances. More importantly. it increases the size of the training dataset for the policy networks compared to the approach taken in POPLIN. For example, with ensemble of M instances, there would be M training data samples available from one real environment interaction, compared to the one data sample in POPLIN-A/P. As a result, DecentCEM is able to use larger policy networks than is otherwise possible, shown in Sec. aadd Appendix F
RELATED WORK
We limit the scope of related works to CEM plannig methods. Vanilla CEM planning in action space with a single Gaussian distribution has been adopted as the planning method for both simulated and real-world robot control (Chua et al. 2018 Finn & Levine 2017; Ebert et al.) 2018 Hafner et al.) 2019 Yang et al. 2020 /Zhng et al. 2021p. Among previous attempts to improve the performance of CEM-based pl planning, we see two types of approaches. The first type includes CEM in hybrid of CEM+X where "X"" some other component or algorithm. POPLIN (Wang Ba 2020) is a prominent example where "X" is policy network that learns a state conditioned distribution that initializes the subsequent CEM process. This addition of the policy network allows the CEM to search in the network parameter space which is shown to have smoother landscape and better exploration. Another common choice of "X" is gradient- adjustment of the samples drawn in CEM. GradCEM (Bharadhwaj et al. 2020) adjusts the samples in each iteration of CEM by taking

---

Under review as a conference paper at ICLR 2022
gradient ascent of the return estimate w.r.t the actions. The benefit that this method brings is not significant on benchmark control tasks. CEM-RL (Pourchot & Sigaud 2019) also combines CEM with gradient based updates from RL algorithms but the samples are in the parameter space of the actor network. To improve computational efficiency, [ee et al.] (002)) proposes an asynchronous version of CEM-RL where each CEM instance updates the sampling distribution asynchronously without waiting for other instances to finish. The downside with both versions of CEM-RL methods are that they rely on model-free RL algorithms. One can imagine reversing the order of CEM and RL and using CEM to update the policies/actors of RL agents. The approach taken in (Khadka et al., 2019) was along similar lines but they used a genetic algorithm instead of CEM.
The second type of approach aims at improving CEM itself. Amos & Yarats (2020) proposes a fully-differentiable version of CEM called DCEM. The key to make the top-k selection in CEM differentiable such that the entire CEM module can be trained in an end-to-end fashion. Despite cutting down the number of samples needed in CEM, this method does not beat the vanilla CEM in benchmark test. GACEM (Hakhamaneshi et al., 2020 increase the capacity of the sampling distri- bution by replacing the Gaussian distribution with an auto-regressive neural network. This change allows CEM to perform search in multi-modal solution space but it is only verified in toy examples and its computation seems too high to be scaled to MBRL tasks. Another method that increases the capacity of the sampling distribution is PaETS (Okada & Taniguchi 2020) that uses GMM with CEM. It is the approach that we followed for our CEM-GMM implementation. It is not clear benchmark tasks since their environment setup modified to have range of actions 5 times larger than the original. Also the running time results in the optimization task in See SeeB]shon shows that it is computationally heavier than the CEM and DecenICEM methods, limiting its use in complex environments. Overall, this second type of approach did not outperform vanilla CEM, a situation that motivated our move to a decentralized formulation, |aaua al. (2015) pro- posed "distributed" CEM that is similar in spirit to our method in that they used multiple sampling distributions and applied the top-k selection locally to samples from each instance. However, their instances are cooperative as opposed to being independent in our work. They applied "collabora- tive smoothed projection steps" to update each sampling distribution as an average of its neighboring instances including itself. The updating procedure is more complicated than our proposed method and proper network topology of the instances is needed: a naive approach of updating according to all instances will lead to mode collapse since the resulting sampling distributions will be identical. The method was tested in toy optimization examples only.
how well it performs
6 EXPERIMENTS
We evaluate the proposed DecenICEM methods in simulated environments with continuous action space. The experimental evaluation is mainly setup to understand if Decen:CEM improves the per- formance and sample efficiency over conventional CEM approaches.
Benchmark Setup We benchmark the algorithms in several continuous-action control environ- ments in OpenAl Gym.
Environments We run the benchmark in a set of 13 environments commonly used in the MBRL literature: Pendulum, InvertedPendulum, Cartpole, Acrobot, FixedSwimme Reacher, Hopper, Walker2D, HalfCheetah, SETS-Reacher3D. vironments prefixed by "PETS" are proposed by Chua et al. (2018). Note that MBRL algorithms often make different assumptions about the dynamics model or the reward function. Their bench- mark environments respective algorithm is runnable. Whenever possible, we inherit the same environment setup from that of the respective baseline methods. This is so that the comparison against the baselines is fair. More details on the environments and their reward functions are in Appendix B
HalfCheahh,,,
Ant. The three en-
often modified from the original OpenAl gym environments such that the
Algorithms The baseline algorithms PETS (Chua et al., 2018) and POPLIN (Wagg & Ba 2020 PETS uses CEM with a single Gaussian distribution for planning. The POPLIN algorithm com- bines a single policy network with CEM. As described in Sec POPLIN comes with two modes: POPLIN-A and POPLIN-P with the suffix "A" denotes planning in action space and "p" for the
aa modified version of the original Gym Swimmer environment where the velocity sensor on the neck is moved to the head. This fix was proposed by Wang & Ba (02))
7

---

Under review as a conference paper at ICLR 2022
InvertedFendulum
Acrobet
Reacher
Average Return
Average Retur
-2h
Steps
Steps
206
Steps
Hopper
Avera.e Return
2879
Steps
Steps
Stegs
PETS
POPUN-A
POPLINP
DecentCEM A
DeCentCEM
SAC
SAC at convergence
Figure 6: The learning curves of the proposed DecentCEM methods and the baseline methods on continuous control environments. The line and shaded region shows the mean and standard error of evaluation results from 5 training runs using different random seeds. Each run is evaluated per train- ing episode in an environment independent from training and reports average return of 5 episodes.
network parameter space. reuse the default hyperparameters for these algorithms from the orig- inal papers if not mentioned specifically. The detailed hyperparameters are listed in the Appendix
D.2 For our proposed methods, we include two variations DecentCEM-A and Decent described in Sec. 4 where the suffix carries the same meaning in POPLIN-A/P All MBRL algo- rithms studied in this benchmark uses the same ensemble networks proposed by [Chua et al. (2018) for the dynamics model learning. We also include Model-Free RL baseline SAC (Haarnoja al.
as
2018) and show its finite-time and asymptotic result.
Evaluation Protocol The learning curve shows the mean and standard error of the test performance out of independent training runs. The test performance is an average return of 5 episodes of the evaluation environment, evaluated at every training episode. At the beginning of each training run, the evaluation environment is initialized with a fixed random seed such that the evaluation environments consistent across different methods and multiple runs to make it a fair comparison. All experiments were conducted using Tesla "100-PCIE-16GH GPUs.
The learning curves of the algorithms shown in Fig, 6 for InvertedPendullmm, Ac- robot, Reacher, Hopper, Walker2D and Ant, sorted by the difficulty task. The full results for all environments are included in Appendix B
Results
We can observe two main patterns from the results. One pattern was that most environments, the DecentCEM methods either matched or outperformed their counterpart that took centralized approach. In fact, DecentCEM can be seen as a generalization of POPLIN by adding a dimension of policy ensemble size, with size recovering POPLIN. We also included negative results shown in Hopper where neither DecentCEM modes outperformed the baselines. It can be interpreted that set- ting the policy ensemble size to one is better than 5 in this environment. By varying this additional parameter, CEM can be fine-tuned for individual domains. Also all model-based methods underper- formed the model-free method SAC, suggesting the difficulty of model learning. The other pattern was that using policy networks to learn the initial sampling distribution in general helped improv- ing the performance of CEM with both centralized and decentralized formulation. This is expected as discussed in Sec 4 since the policy network allows the sampling distribution to "resume" from high-value region seen before and to generalize similar states.
Ablation Study A natural question to ask about the DecentCEM-A/9 methods whether the in- creased performance is from the larger number of neural network parameters. We added two vara-
8

---

Under review as a conference paper at ICLR 2022
Pondutum
Reacher
PETS PESS
-200
Steps
Shape
Steps
A
POPLIN BiggerFolicynet
DeconSCEM
POPLIN
POPLINP BigerPalicyee
DecenOCEM
Figure 7: Ablation study on the policy network size where POPLIN-A&P have a bigger policy network equivalent in the total number of neural network weights to their DecentCEM counterparts. For better visual clarity, curves are smoothed with sliding window of size 10.
tions of the POPLIN baselines where bigger policy network was used. The number of the network parameters was equivalent to that of the ensemble of policy networks in DecenICEM-A/P We show the comparison using three environments in Fig. 7 Pendulum(1), Reacher(2) and PETS-Pusher(7) (action dimension in parenthesis). In both action space and parameter space planning, bigger policy network in POPLIN either did not help or significantly impaired the performance (see the POPLIN-P results in reacher and PHTS-Pusher) This is expected since unlike DecentCEM, the training data in POPLIN do not scale with the size of the policy network, as explained Sec.
Figure (Left) shows the cumulative selection ratio of each CEM instance during training of DecenCCEM-A with an ensemble size of 5. It suggests that the random initialization of the policy network is sufficient to avoid mode col- lapse. We also plot the action statistics of the instances in Figure g (Right). The line and shaded area represent the mean and max/min action the instances, respectively. For visual clarity, we show time segment toward the end of the training rather than all the 10k steps. De- centCEM has maintained enough diversity in the instances even toward the end of the train- ing. DecentCEM-P is excluded from both plots since it shows a similar trend as DECeA More ablations results are included in Appendix F
621
Pendulum
Pendulum
Selection Ratio
S. 16 Ro
CEM Instancee
Instancee
CEM Instance
CEM Instance
Instance
Steps
Steps
Figure 8: Ablation of ensemble diversity. Left: Cumulative selection ratio each CEM instance. Right: Action statistics of the instances.
7 CONCLUSION AND FUTURE WORK
In this paper, we study CEM planning the context of continuous-action MBRL. We propose novel decentralized formulation of CEM named DecentCEM, which generalizes CEM to run multi- ple independent instances and recovers the conventional CEM when the number of instances is one. We illustrate the intuition and the strengths of the proposed DecentCEM approach in motivational one-dimensional optimization task and show how fundamentally differs from the CEM approach that uses Gaussian or GMM. We also show that DecentCEM has almost sure convergence to lo- cal optimum. We extend the proposed approach MBRL by instantiating two decentralized CEM methods that combine with policy networks. We show the efficacy of the proposed methods in benchmark control tasks and ablations studies.
There is gap between the convergence result and practice that the theory assumes that the number of samples grow polynomially with the iterations whereas a constant sample size is commonly used work. Investigating the convergence properties of CEM under constant
in practice including sample size makes an interesting direction for future work. Another interesting direction to pursue finite-tine time analysis of both CEM and DecentCEM.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We have included the implementation details in Appendix D and the source code in the supplemen- lary materials.
REFERENCES
Brandon Amos and Denis Yarats. The differentiable cross-entropy method. In International Con- ference on Machine Learning, pp. 291-302. PMLR, 2020.
Dimitri P Bertsekas. Dynamic programming and optimal control 3rd edition, volume i. Belmont, MA: Athena Scientific, 2005.
Homanga Bharadhwaj, Kevin Xie, and Florian Shkurti. Model-predictive control via cross-entropy and gradient- based optimization. In Learning for Dynamics and Control, pp. 277-286. PMLR, 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 4759-47700 2018.
Pieter- Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for vision-bass robotic control. arXiv preprint arXiv:1812.00568, 2018.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA). pp. 2786-2793, IEEE, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, pp. 1861-1870. PMLR, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2565. PMLR, 2019.
Kourosh Hakhamaneshi, Keertana Settaluri, Pieter Abbeel, and Vladimir Stojanovic. Gacem: Gen- eralized autoregressive cross entropy method for multi-modal black box constraint satisfaction. arXiv preprint ar arXiv:2002.07236, 2020.
Morris W Hirsch, Hal Smith, and Xiao-Qiang Zhao. Chain transitivity, attractivity, and strong repellors for semidynamical systems. Journal of Dynamics and Differential Equations, 13(1): 107-131, 2001.
Jiaqiao Hu, Ping Hu, and Hyeong Soo Chang. A stochastic approximation framework for a class of randomized optimization algorithms. IEEE Transactions on Automatic Control, 57(1):165-778 2011.
Shauharda Khadka, Somdeb Majumdar, Tarek Nassar, 7ach Dwiel, Evren Tumer, Santiago Miret, Yinyin Liu, and Kagan Tumer. Collaborative evolutionary reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 3341-3350. PMLR, 2019.
Kyunghyun Lee, Byeong-Ur Uk Lee, Ukcheol Shin, and In So Kweon. An efficient asynchronous method for integrating evolutionary and gradient-base policy search. Advances in Neural Infor- mation Processing Systems, 33, 2020.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model model-based control. In International Conference Learning Representations, ICLRR, 2019.
10

---

Under review as a conference paper at ICLR 2022
Sergio Valcarcel Macua, Santiago Zazo, and Javier Zazo. Distributed black-box optimization of nonconvex functions. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3591-3595. IEEE, 2015.
Shie Mannor, Reuven Y Rubinstein, and Yohai Gat. The cross entropy method for fast policy search. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 512- 519, 2003.
Masashi Okada and Tadahiro Taniguchi. Variational inference mpe for bayesian model-based rein- forcement learning. In Conference Robot Learning, pp. 258-272. PMLR, 2020.
Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. arXiv preprint ar Xrv:20008.. 2008..6389, 2020.
Aloïs Pourchot and Olivier Sigaud. CEM-RL:
combining evolutionary and gradient-based methods for policy search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. In 8th ternational Conference on Learning Representations, ICLR, Addis Ababa, Ethiopia, April 26-30, 2020.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. BBenchmarking model-based reinforce- ment learning. CoRR, ab7/1907.02257, 2019.
Yuxiang Yang, Ken Caluwaerts, Atil Iscen, Tingnan Zhang, Jie Tan, and Vikas Sindhwani. Data efficient reinforcement learning for legged robots. Conference on Robot Learning, pp. 1-10. PMLR, 2020.
Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp. Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In International Conference Artificial Intelligence and Statistics, pp. 4015-4023. PMLR, 2021.
11

---

Under review as a conference paper at ICLR 2022
A DETAILS OF THE MOTIVATIONAL EXAMPLE
A. SETUP AND RUNNING TIME
For a fair comparison the three methods CEM, CEM-GMM and DecenICEM, we performed hyperparameter search for all. The list of hyperparmeters performing hyperparameters for each method under each population size are shown in Table [A.2] These hyperparameters were what the data in Fig- 3 were based Note that the top percentage of samples "Elite Ratio" (in Table A.I) was used in the implementation instead of top-k but they are equivalent. The running time included in Table A.3
summarized in Table A.1 and the best
Table A.I: Hyperparameters
Algorithm
Parameter
Value
Shared
Total Population Size
100, 200, 500, 1000 0.1
Parameters
Elite Ratio
Smoothing Ratio
0.1
6: Minimum Variance Threshold
le-3
Maximum Number of Iterations
100
CEM-GMM
M: Number Mixture Components
3,5.8.10
K: Weights of Entropy Regularizer
0.25, 0.5
r: Return Mode
ss: mean of the mixture component sampled based their weights m': mean of the component that achieves the minimum cost
DecentCEM
Number Instances in Ensemble 3,5.8.10
Table A.2: Best Parameter
Total Population Size
100
200
500
1000
CEM-GMM
M 10
M
M
M
K 0.25, 'm'
= 0.5
k == 0.25
0.5 r='s"
r 'm'
r='m'
DecentCEM 10
E
E 10
E 8
Table A.3: Total Time of 10 Runs (in seconds)
Total Population Size
100
200
500
1000
CEM
0.079
0.093
0.165
0.318
CEM-GMM
7.322
11.500
24.431
59.844
DecentCFM
0,407
0.420
0.506
0.545
A.2 OUTPUT OF CEM APPROACHES
In terms of the output of CEM approaches, there exist different options in the literature. The most common option is return the sample the domain that corresponds density in the final sampling distribution. It is the mean in the case of Gaussian and the mode with the highest probability density in the case GMM. One can also draw a sample from the final sampling distribution (Okada & Taniguchi] 2020) and return it. Another option is to return the best sample observed (Pinneri et al. (2020). The best option among the three may be application dependent. It has been observed that in many applications, the sequence of sampling distributions numerically converges to deterministic one (De Boer et al. 2005). in which case the first two options identical.
the highest probability
12

---

Under review as a conference paper at ICLR 2022
B BENCHMARK ENVIRONMENT DETAILS
In this section, we over the details of the benchmark environments used in the experiments.
Table B.I lists the environments along with their properties, including the dimensionality of the observation and action spaces and the maximum episode length. Table [.2 provides a list of the reward function for each environment. Whenever possible, reuse the original implementations from the literature as noted in I.ble so as to avoid confusion. The environments that start with "PETS" are from the PETS paper (Chua et al., 2018), which is one of the baseline methods. Most other environments are from Wang et al. (2019) where the dynamics are the same the OpenAI gym version and the reward function Table B. exposed to the agent. For more details of the environments, the readers are referred to the original paper.
Note that FixedSwimmer the velocity sensor on the neck is moved to the head. This fix was originally proposed by Wang & Ba (2020). For the Pendulum environment, we use the OpenAl Gym version. The modified version Wang et al. 2019) uses different reward function which we have found to be incorrect.
modifed version of the original Gym Swimmer environment where
The number in the bracket in the "Environment Name" column denotes the source of this env environment: [1] refers to the benchmark paper from Wang et al. 2019); ; [2] denotes PETS (Chua et al. 2018).
Table B.I: The setup of the environments.
Environment Name
Observation Dim
Action Dim
Maximum Episode Length
Pendulum
3
200
InvertedPendulum [1]
4
100
Cartpole [1]
4
1
200
Acrobot [1]
6
1
200
FixedSwimmer [1]
9
2
1000
Reacher [1]
11
2
50
Hopper [1]
11
3
1000
Walker2d [1]
17
6
1000
HalfCheetah [1]
17
6
1000
PETS-Rcacher3D [2]
17
7
150
PETS-HallCheetah [2]
18
6
1000
PETS-Pusher [2]
20
7
150
Ant [1]
27
8
1000
Table B.2: Reward Functions. d, denotes the vector between the end effector to the target position.
2, denotes the height of the robot. ||v and ||v||2 denote the 1-norm and 2-norm of vector v, respectively. In PETS-Pusher, di, is the vector between the object position and the goal and d2, denotes the vector between the object position and the end effector.
Environment Name
Reward Function
Pendulum
03 0.10, + 0.001aa -03
InvertedPendulum
Cartpole cose, - 0.01I?
Acrobot
-cos0 cos (0., . + 02,1)
FixedSwimmer
4. 0.0001 a
Reacher
-dd |
Hopper
4. 0.1 l
3(2 21 1.3)
Walker2d
f. 0.1 a,
3 - 1.3)
HalfCheetah
i - 0.1 a
PETS- DETS-Reacher3D
dd 11 0.01 |a. 11
PETS-HalfChedee
f 0.1 |a
PETS-
-1.55
0.5 0.1
Ant
f. 0.1 a
3(a 3(5-))557
13

---

Under review as a conference paper at ICLR 2022
C ALGORITHMM
In this section, we give the pseudo-code of the proposed algorithms DecentCEM-A and DecentCEM-P in Algorithm [ and respectively. We only show the training phase. The algo- rithm at inference time is simply the same process without the data saving and network update. For
the internal process of CEM, we refer the readers to De Boer et al. (2005) Wang & Ba 2020).
Algorithm 1: DccenICEM-A Training
Initialize the policy networks p. with 0,, 1, 1.2. where is ensemble size. Planning
horizon H. Initialize the dynamics network fw parameterized by w. Empty Datasets Dm and Dp
!/ Episode I, warmup phase
Rollout using random policy, fill the dataset Dm with the transition data {(t, at Sc+1)}
Update using Dm by Jean-Squared Loss
// Train the dynamics network with D.
Episode onwards
repeat
f. = 0, Dp
// Each episode, reset time and dataset
repeat
foreach policy network p: the ensemble
generate reference mean of action sequence distribution μui using pi and the model fa.
// Apply CEM to refine the action distribution.
// fis. 02 are the mean action sequence of the refined distribution and its expected value pa, (EM())
10
ār.: Ao [0]
end
arg a U
Pick best distribution // Execute the first action in the mean sequence
St+11 step(a,)
14
Append the transition (&t, de, 8641) Dm..
15
Append the data {st, de to D,
16
1
Update time step
until Either reached the maximum episode length or terminal state
18
Update the model parameter w using datasct Dm
Update the policy network weights ) using dataset Dp the behavior cloning objective
20
until bored
Algorithm 2: DecentCEM-P Training
Initialize the policy networks p. with 8,, i= 1, ... M where M is the ensemble size. Planning
horizon Initialize the dynamics network fu parameterized by w. Empty Datasets D.,. and Dp // Episode 1, warmup phase
Rollout using random policy, fill the dataset Dm with the transition data {(+, at st+1)}
Update using Dm by Mean-Squared Loss
// Train the dynamics network with D.
// Episode onwards
repeat
L -= 0, D, = 0
I/ Each episode, reset time and dataset
repeat
foreach policy network ps in the ensemble do // Apply CEM refine the distribution the neural network weight.
I1 fas. V; are the mean of refined weight distribution sequence and its expected value
p., v; CEM(O.)
6; [a]
// Keep the weight the first step and discard the rest
10
11
0. arg maxg,+5; +5; 0%
I/ Pick the best distribution of weight sequence
12
a. = po, (st)
13
Sc+1 = step(ar)
Exccute the action returned by the policy network pe:
14
Append the transition (St, at, 8141) ( to Dm
15
Append the data M to Dp
16
1= 1+1
I/ Update time step
17
until either reached the maximum episade length or terminal state
18
Update the model parameter w using dataset D.
M Update the policy network weights {0, {M using dataset Dp by AVG training objective
19
20
until bored
14

---

Under review as a conference paper at ICLR 2022
D IMPLEMENTATION DETAILS
D.1 REPRODUCIBILITY
Our implementation is fully reproducible by identifying the sources of randomness and controlling the random seeds as summarized in Table D.I The seeds are set once at the beginning of the experiments.
Table D.1: Random Seed. The set (1,2.3,4,5) refers to the seeds for five runs. Note that control the random seed for the environments since there is a random number generator in openai gym environments independent from other sources
Source of randomness
Random Seed
deep learning framework (tensorflow in case)
numpy
{1.2.3.4.5}
python random module
the training environment
1234
the evaluation environment
D.2 HYPERPARAMETERS
This section includes the details of the key hyperparameters used in the proposed DecentCEM al- gorithm (Table D.5) and the baseline algorithms PETS (Table D.3). POPLIN (Table and SACE (Table D.2). For the neural network architecture for the dynamics model, the DecemICEM methods exactly follow the original one PETS and POPLIN for fair comparison, which is ensemble of fully connected networks.
Table D.2: Hyperparameters of SAC
Parameter
Value
Actor learning rate
0.0001
Critic learning rate
0.0001
Actor network architecture
(dim(observatio)) 64, 64, 2x dim(action))
Critic network architecture
ddimmobseeratioo))dddfeettn 64, 64, 1]
Table D.3: Hyperparameters of PETS
Parameter
Value
Model learning rate
0.001
Warmup episodes
1
Planning Horizon
30
CEM population size
(except PETS-reacher3D: 400)
CEM proportion of elites
10%
CEM initial distribution variance
0.25
CEM max number of internal iterations
5
*Our SAC implementation used network architectures that are similar to the policy network in our method.
The results of our implementation either matches surpasses the ones reported in PETS, POPLJN and the benchmark Wang al. (2099)
15

---

Under review as a conference paper at ICLR 2022
Table D.4: Hyperparameters of POPLINA and POPLINP
Parameter
Value
Model learning
0.001
Warmup episodes
Planning Horizon
30
CEM population size
500 (except in GETS-reacher3D. 400)
CEM proportion of elites
10%
CEM initial distribution variance
0.25
CEM max number internal iterations
Policy network architecture (A)
(dim(obsevvation) 64, 64, dim(diiiiionn [
Policy network architecture (P)
[dimí obseervation). 32, dim(action))
Policy network learning
0.001
Policy network activation function
tanh
Table D.5: Hyperparameters of the proposed DecenICEM-A/P
Parameter
Value
Model learning rate
0.001
Warmup episodes
Planning Horizon
30
Ensemble Size
5
CEM population size each instance
100 (except in PETS- reaS-reac:::3:: 80)
CEM proportion of clites
10%
CEM initial distribution variance
0.25
CEM max number of internal iterations
5
Policy network architecture (A)
[dim(observattinn) 64, 64, dim(action))
Policy network architecture (P)
[dim (osservteratii 32, dim(action))
Policy network learning rate
0.001
Policy network activation function
tanh
16

---

Under review as a conference paper at ICLR 2022
CEM iter=0
CEM iter=2
CEM iter=4
CEM iter=
f(x)
p(x)
X
X
X
CEM-GMM iter=0
CEM-GMM iter=2
CEM-GMM iter=4
CEM-GMM iter=6
f(x)
p(x)
-75
75175
-25
75-75
-1.0
-25
25
5.0
75-73
-25
6.0
2.5
5.0
75
X
DecentCEM iter=0 DecentCEM iter=2 DecentCEM iter=4 DecentCEM iter=6
f(x))
p(x) 1.
-3.8
-25
6.0
25
54
75-75
-5.0
-25
0/0
2.5
3.0
7.5
-35
-3.0
-25
0.0
25
5.6
75-75
-8.8
-25
6.0
25
5.0
75
Figure The iterative sampling process in the 1D optimization task.
E FULL RESULTS
E.1 DETAILED VISUALIZATION OF THE ITERATIVE UPDATES THE ID OPTIMIZATION TASK
Figure is version of Figure with more iterations. It shows the iterative sampling process of CEM methods in the ID optimization task and how the sampling distribution evolve over time.
E.2
FULL LEARNING CURVES
In Figure TO we report the learning curves in all 13 environments that we listed in Appendix B
The algorithms used the benchmark are: PETS, POPLIN-A, POPLIN-P and the two modes the proposed method DecentCEM-A A and DecentCEM-P. We also included the interim and asymptotic performance of a model-free algorithm SAC as reference.
The learning curves in some environments can be noisy. We apply smoothing with 1D uniform filter to the results of the following environments for easier interpretation: Cartpole, FixedSwimmer, Hopper, Walker2d, HalfCheetah, PETS-Reacher3D PETS-Pusher The window size of the filter was 10 for all but Cartpole, where 30 was used due to its high noise level for all algorithms.
Note that the performance of the baseline methods may be different from the results reported in their original paper. Specifically, in the paper by Wang & Ba (2020). PETS, POPLIN-A and POPLIN-P have been evaluated in number of environments that results may not be consistent with theirs due to differences in the implementation and evaluation protocol. For example, results of PETS, POPLIN-A and POPLIN-P in the Acrobot environment
for the benchmark. Our benchmark
17

---

Under review as a conference paper at ICLR 2022
Pendulum
InvertedPendulum
Cartpole
200
Average Recurn
-9.9
-50
-13.
-120
784
-108
200
4800
4.04
12000
2005
2009
290
200
20000
20.00
33000
6000
2880
Steps
Steps
Steps
FixedSwimmer
Average Recum
-100
-29
-100
-400
Steps
Steps
Steps
Hopper
Walker2D
HalfCheetah
Average Return
0000
-300
Steps
Steps
PETS Reacher3D
PETS-HalfCheetah
PET S- Pusher
Average Return
-00
-39
8000
1.4
-37
-20
2250
3903
12:0
190
600
-200
2000
2,000
10000
100
200
1303 15000 1209 3000
Steps
Stops
Steps
Average Return
1000
100
1883
45000
4083
1998
19000
Steps
PETS
POPUN-A
POPUNNP
A
DecentCEM-P
SAC
SAC at convergence
Figure 10: The learning curves of the proposed DecenICEM methods and the baseline methods on continuous control environments. The line and shaded region shows the mean and standard error of evaluation results from 5 training environment independent from training and reports average return of 5 episodes at every training episode. To ensure that the evaluation environments the across different methods and multiple runs, we set a fixed random seed in the evaluation environment of each task.
using different random seeds. Each run is evaluated in an
18

---

Under review as a conference paper at ICLR 2022
all better than the results in Wang & Ba (2020) We have identified bug in the POPLIN code base that causes the evaluation results to be on a wrong timescale that is much slower than what actually is. Hence the results of our implementation look far better, reaching return of 0 at about 4k steps as opposed to 20k steps reported in Wang & Ba (200).
E.3 ANALYSIS
Let's group the environments into two categories based on how well the decentralized methods perform in them:
1. Pendulum, InvertedPendulum, Acrobot, Cartpole, Reacher, Walker2D, HalfCheetah, PETS-Pusher,
Ant,
2. FixedSwimmer, Hopper, HETS-HalfCheeth
The first category is where the best performing method is one of the proposed DecentCEM al- gorithms: DecensCEM-A or DecentCEM-P. In environments where the bascline POPLIN-A or POPLIN-P could reach near-optimal performance such as pendulum and invertedPendullm, ap- plying the ensemble method would yield similar performance as before. In Acrobot, Reacher, Walker2D and PETSSPuuher, applying the decentralized approach increases the performance in both action space planning ("A") and parameter space planning ("P"). In Pendulum, InvertedPendulum, Cartpole, HalfCheetah, SETS--Reahhr3D and Ant, ensemble helps in the action space planning but either has no impact or negative impact on the parameter space planning.
The second category is where it is better not to use decentralized approach with multiple instances (note that by using one instance in the decentralized methods, we can recover one POPLIN- A, POPLIN-P). In Hopper, the issue might lie in the MBRL approach in general since all MBRL baselines performed worse than the model-free baseline SAC. One possible issue is that the true dy- namics is difficult approximate with model learning approach. Another possibility is that it's necessary to learn the variance of the sampling distribution, which none of these MBRL approaches do. To be clear, the variance is adapted online by CEM but it is not learned. In FixedSwimmer, the ensemble approach performs worse in both action space and parameter space planning. One potential reason for the result is that the optimization landscape complex and needs more samples to estimate the expected return than what was used in each instance of the ensemble. In this case, it better not to distribute the population into several instances. An alternative population size, with downside of increasing computation. HETS-HalfCheetue is slightly different in that the ensemble does improve the performance significantly when used for action space plan- ning. However, POPLIN-P performs significantly better than all other algorithms. This suggests that the parameter space planning has been able to successfully find a high return region using a single Gaussian distribution. In this case, distributing the population size would not be able to trade the estimation accuracy for better global search.
increase the total
F MORE ABLATION
Fondulum
Predilum
Pendular
-000
9.8
0.8
POPENA A
-000
becenndi 62
-1200
DeceniCin
-1400
DecenccD 6:65
2020
0200
18383
2000
2250
Staga
(a) Action Distance
(b) Ensemble Size
(c) Policy Control
Figure 11: More Ablation (a) Pairwise distance between the actions of instances in DeC A during training (b) Ensemble size ablation: E2 denotes an ensemble size of 2 (c) Policy control performance where the policy network directly used for control without CEM policy improvement
19

---

Under review as a conference paper at ICLR 2022
Figure 11 (a) is an additional plot for the ensemble diversity ablation. It shows the statistics of the pairwise distance between the output actions by the CEM instances at each time step. The same as in Fig. 8 (b), we only show time window toward the end of training for visual clarity and the line and shaded region represent the mean and min/max distances.
Figure [T (b) shows the performance of the DecentCEM-A algorithm under different ensemble size (where we keep the population size of each instance the same). Adjusting the ensemble size has an impact on the performance of Decen/CEM-A. An ensemble size of or 4 could improve the result reported in Fig. T0 to reach a near-optimal performance. We did not show the results for the parameter space planning mode "P" since all variations achieve near-optimal performance and their curves overlap. We fixed an ensemble size of 5 throughout our benchmark but tuning the ensemble size for each environment could improve the performance further.
We also study the performance of policy control where the policy network is directly used for control without the CEM step, denoted by the extra suffix "-PC". The result is shown (c) Without the policy improvement from CEM, all algorithms perform worse than their counter-part of using CEM. POPLIN-P-PC and DecenICEM-P-PS both get stuck local optima and do not perform very well. This makes sense since the premise of planning in parameter space is that CEM can search more efficiently there. The policy network is not designed to be used directly as a policy. Interestingly, DecenICEM-A-PC achieves high performance from about 6k steps (30 episodes) of training. The ensemble of policy networks adds more robustness to control than using single one.
G OVERHEAD OF THE ENSEMBLE
The sample efficiency is not impaired when going from one policy network to the multiple policy networks used in DecentCEM-A and Decen1CEM-P. This is because that the generation of the train- ing data only involve taking imaginary rollouts with the model, rather than interacting with the real environment, discussed in Section
In terms of the population size (number of samples drawn in CEM), the Decen(CEM based methods do not impose additional cost. We show in both the motivational example (Sec. and the benchmark experiments (Appendix Eb that the proposed methods work better than CEM under the same total population size.
The additional computational cost is reasonable in DeCenICEM compared to POPLIN. Each branch of policy network and CEM instance runs independently from the others. allowing for a parallel implementation. The instances have to be synchronized (arg max) but its additional cost is minimal. One caveat with our current implementation though is that is serial, which slows down the speed. This is limitation that calls for future work of parallel implementation.
H CONVERGENCE ANALYSIS OF DECENTRALIZED CEM
This section analyzes the convergence of the proposed DecentCEM algorithm in optimization. Consider the following optimization problem:
x² arg max V(z) xEX
(4)
where X C R. is non-empty compact set and V-)) is bounded, deterministic value function to be maximized, We assume that this problem has a unique global optimal solution x" but the objective function V-) may have multiple local optimum and may not be continuous.
We will show that the existing convergence result of CEM in continuous optimization established in (Hu al., 201I) also applies CeC. It assumes that the sampling distribution (x())
in CEM is in the natural exponential families (NEFs) which subsumes Gaussian distribution (with
known covariance). We restate the definition of NEFs for completeness (definition 2.1 from (Hu
et al. 201T)):
Definition H.1 (Natural Exponential Family). A family of parameterized distributions {g)()), e
C Rd4 on X C R" is called a Natural Exponential Family (NEF) if there exists continuous map-
pings r R RRd h T - IR and K I IR. such that g g exp( T(x) K k()))((x),
20

---

Under review as a conference paper at ICLR 2022
where the parameter space O {o R |K(0)] < o0), K(Ø) In I " T(z))h(z)v(dz) and v is the Lebesgue measure of X.
The mean vector function denoted as ( = [T)]] where the expectation Eg is with respect to Is and it can be shown that m(-) is invertible. Note that the expression of the densities can be simplified when restricted to a multivariate Gaussian distribution (with known diagonal covariance) where the natural sufficient statistics (x) is the identify function.
We then present the CEM algorithm below to fix notations. follows Algorithm in (Hu et al, 2011) but is modified to align with some notations introduced in previous sections in paper.
Algorithm 3: CEM
Choose family distributions go (xr), x from NEFs defined and the initial parameter db E int((h ()) where int denotes the interior of the parameter space Specify elite ratio E (0, 1) and step size sequence {au} and {Au where & denotes the time step. Set which is the parameter in the thresholding function defined in Equation 5
k = 0. Specify
if 27 if - < <1
1(x, 7)
(5)
if SーーE
repeat
Step 1: Draw N₂ i.d samples As = {21, 22,, 2N. from the distribution 962 (z)
Step 2: Calculate the sample (1 p) quantile 5tx Vif(-=)N,, 1) where [a is the ceiling function
that gives the smallest integer greater than a and Via is oth-order statistics of the sequence
(V(x,)) | where V(-) the objective function maximized.
Step Compute the new parameter @k+1
mmm (m)+t))) where doo m(do) E.po (T(z)) and
EEEA, (v())) + (1 ax)
Mk+1 OK ΣzEr, 1(V(z), 7k)
A N
Σ
+1 +1 Xa)TT
(6)
#EAx
Step 4: l- 1 k
until a stopping condition reached 10 return dx
The convergence results will require the following assumptions from (Hu et al. 201
Assumption 1. The parameter 0k+1 computed step 3 of Algorithm satisfies Φx+1 E int(() for all k.
Assumption 2. The step size sequence {oss satisfies: Ork 30
>
0 Vk limp- Og = 0 and
kk= O% co.
Assumption 3. The sequence (1x} satisfies d = O(k->) ) for some constant 2 and the sample
size N₂ (kk)) where ( max(0, 1 2A}.
Assumption 4. The (1 -pp) 1--p)-quanttee of (V(x), (0(x)} } is unique for each & 4.
We know from Hu et al. (2011) that the sequence } from equation 6 asymptotically ap- proaches the solution set of the ODE:
dr((t) L(n)
dt
L(n) = A In E. (m) (0=m=1()
(8)
where (mm))(n) is the true (1 - quantile of V Vxx) under (n)
Assumption 5. The function L(n) defined in equation 8 has unique integral curve for given initial condition.
The above assumptions 1-5 are the assumptions required by the previous convergence result of CEM.
To show the convergence of DecentCEM, we only require additional mild condition:
21

---

Under review as a conference paper at ICLR 2022
Assumption 6. Let M be the number of instances in DecentCEM and each instance has sample size of Ne where Ng. is the total number of samples that satisfies the assumption 3 M is constant M and 0 < V.
Now restate the convergence result of DecentCEM from the main text and show the proof:
Theorem 3.1 (Convergence of DecentCEM), If Assumptions 1-5 hold for CEM instance described in Algorithml 3 and we decentralize it by evenly dividing its sample size N₃ into M CEM instances in DecentCEM algorithm that satisfies Assumption 6, then the sequence of iterates {nt,s) generated by each CEM instance indexed by converges almost surely to an internally chain recurrent set (Hirsch al. 2001) of Equation 7 Furthermore, the solution of DeceniCEM (T,, converges almost surely to the best solution of the individual instances in terms of the expected value of Em (m-1(1)) (1)) (v))).
Proof. Each individual CEM instance has a sample size of Under Assumption 3, N₁. ((k)). )
Since Assumption 6 holds, M constant and gets absorbed into the θ and we have N2
M = (1.))
Hence the conditions of Theorem 3.1 in (Hu et al. (011)) holds for each CEM instance indexed by i and can be directly applied to show the almost sure convergence of their solutions {vr,a) J to an internally chain recurrent of Equation 7 If the recurrent sets isolated equilibrium points, then } converges almost surely to a unique equilibrium point.
Due to the fact that the instances in DecentCEM run independently from each other, their solutions {n (or cquivalently & J = {m (Tr.)) D might converge to identical or different solutions denoted as {n), ) DecenCCM computes the final solution by applying an arg max over all individual solutions: Io,, arg (mn,) JMM Em- (=:)] (equivalent Equation 3
Here the expectation is approximated by the sample mean with respect the distribution Im (n)
I اE= V(zy), which converges almost surely to the true expectation according to the strong law of large numbers. Hence we have that 7a,, converges almost surely to the best solution in the set {nr 1==1 found by the individual CEM instances, in terms of the expected value of Em--(()) [v)).
M
Note that the theorem implies that the solution of CEM/ / DecentCEM assigns the maximum proba- bility to a locally optimal solution to Equation 4 It does not suggest whether this local optimum is global optimum or not. To the best of knowledge, almost sure convergence local optimum is the only convergence result that has been established about CEM in continuous optimization.
22


Reviews:
[{"review_id": "xbCvc4zwVYU", "reviewer": "Reviewer_niPd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper wants to solve the problem that CEM cannot handle multi-modal action distributions within the top-k samples. In this case, CEM averages over the multiple modes and requires more samples to converge. While this problem theoretically exists and one can describe a motivational example where this problem is relevant, the problem is not relevant for the performed model-based RL experiments on the OpenAI tasks. The algorithms have sufficient entropy to break the multiple solutions and converge to a single solution during a few replannings. The difference in reward is usually not significant. As shown by Pinneri et. al. using the argmax (which also avoids averaging over modes) instead of the mean can improve run-time and marginally increase the reward, but the differences are not essential. From my personal experience with CEM, using the argmax over the mean operation can potentially increase performance in some environments but the increase is not significant. The performed experiments of the authors do not show much of an improvement of DecentCEM over the other methods. The authors also do not provide an evaluation that evaluates whether the computation time decreases with their parallel structure of CEM & policies. Furthermore, the computational complexity during training most likely increases as one needs to train N policies. Therefore, the proposed algorithm does not improve the empirical performance of Poplin. For more complex environments, the proposed approach might be beneficial, but for the OpenAi benchmark tasks, the parallel structure is not necessary to achieve a good performance. \n\nThe paper also does not provide new theoretical insights or has a good motivation. Using n-parallel policies + CEM is a trivial extension of Poplin with uncertain benefits. If one wants to address the uni-modal Gaussian assumption of CEM, one should invest in sequential Monte Carlo methods which can handle multi-modal solutions.\n\nBesides the poor motivation of the approach and limited experimental results, the paper is well-written and easy to follow. Unfortunately, only the idea is not sufficient to be presented at ICLR. ", "summary_of_the_paper": "The paper proposes DecentCEM which uses parallel instances of CEM to learn optimal policies for problems that contain multi-modal optimal actions. Instead of using a single policy as sampling distribution and optimizing the actions in the vicinity of this action trajectory, the DecentCEM method uses parallel with multiple policies and cem optimizers. Therefore, the proposed algorithm can learn multi-modal actions and should improve sample efficiency. The algorithm is evaluated on the standard OpenAI benchmark tasks.  ", "summary_of_the_review": "The advantage of the proposed algorithm is unclear and there are no theoretical insights within the paper. The algorithm is a trivial extension of POPLIN and has little relevance. Therefore, I recommend rejecting the paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "pYOeU4XPZNR", "reviewer": "Reviewer_hpTg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong:\n* The idea is straightforward, but good, and the toy example indeed suggest that a GMM cannot handle the multimodality in the optimization landscape. \n* The paper is well written, very clear, equations are concise. The appendix is really detailed as well, with full hyperparameter settings. I am convinced another researcher could replicate these experiments. \n* I really like the 1D toy example. It gives a good visual illustration of the method and what is happening. \n\nWeak:\n* I think your MBRL results are not super strong: I see quite some noisy curves, and in some of them your method indeed comes up, but in some others it is actually below the alternatives. I would say the real improvement is mostly for DecentCEM-A on Acrobot, Reacher, Ant and Reacher, but given that is below on Hopper and Walker2D, we may doubt how much signal there really is. I do like that you show all results (including negative), but you may slightly rephrase the statement that your method ‘either matches or outperforms their counterparts’. \n* Toy experiment: I find it suprising that CEM-GMM looses the global optimum. My main issue is that you do not report on the number of mixture components in the GMM? This seems to be really crucial. Looking at the appendix results, it seems that the CEM-GMM is unlucky on iteration 0 (having few samples on the right), but with enough mixture components this should still be fine. And what is the number of instances in DecentCEM? (I hope the same as the number of mixtures) Did you tune both? In short, I lack some hyperparameter choices you make in the toy experiment.\n* End of 4.3: I have some trouble understanding why the amount of data goes up per environment interaction. Each instance has a different distribution over policies right, and I can only evaluate one of these policies per environment interaction? I might be missing something here, but I would need some clarification here.\n* I think Related Work has a major omission, since it only focuses on other MBRL work, but does not discuss any related work from CEM literature. Has this decentralized approach already been tried there, or something similar. Then, it can still be relevant to MBRL, but you should note and discuss this. \n* The paper has no Discussion and Future work, which I think should always be there. \n* The graphs are quite small and in some cases just unclear. For example, in Figure 6 the colour coding is far from optimal, I really have a hard time figuring out which line is which on my printed version (in colour), since you e.g. have red and pink. On a black and white print, if would really be completely infeasible. Figure 8 is also really small, and needs an more extensive caption. If you cannot increase the plot size, at least increase the axis labels in size. \n* Your methodology was really clear, but a few things I did not get: 1) Is the v_th in Eq. 3 a threshold per CEM instance (then I would write v_th^I), 2) Is the k of CEM-GMM the same as the k for each instance of DecentCEM (i.e., does Decent CEM get more top-k samples, and should you correct for this?).", "summary_of_the_paper": "This paper studies a novel CEM method for model-based RL. While previous approaches used a centralized method (based on a unimodal Gaussian or Gaussian mixture), they propose a decentralized CEM, where each instance independently tracks its own data and top-k estimates. They first test their new method on a 1D toy task, showing better convergence to the global optimum. They then test their method on several model-based RL tasks, outperforming previous methods in some tasks, while being on par or inferior in others. ", "summary_of_the_review": "I find it really hard to judge this paper, since it has strong and weak points. As mentioned above, I like the idea (it is straightforward and easy to grasp, yet well motivated), the paper is very clearly written, has clear notation, and gives good intuitive illustration of the idea. On the downside, I do not think the results are really convincing (although there is some signal), I lack hyperparameters on the number of mixtures in the toy experiment (and the sensitivity of results to varying it), I completely miss related work from the CEM literature, there is no Discussion and Future work, and the graphs are not easy to read. Individually, all these downsides can be overlooked, but with all of them together I get in doubt. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ObttuMcyfVK", "reviewer": "Reviewer_WTkw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Using an ensemble of optimisers is a powerful idea that can help to explore the solution space more efficiently, and should be explored further.\n- The continuous control benchmark is relevant for this task, and the baselines are sensible. In particular, having included SAC as a reference is appreciated.\n- Simulation results show similar or better performance to other CEM variants in most of the environments.\n- The paper is generally well written and easy to follow.\n\nWeaknesses:\n- Training curves are stopped before convergence for many environments. This makes difficult to evaluate the stability and final performance of the proposed approach, especially for those environments in which the final performance of SAC (black dashed line) is much higher than that achieved by the proposed approach(es).\n- Although the presented algorithm is novel, the idea of using ensembles of optimisers is not. Comparison with other ensemble-based approaches, like the decentralised CEM introduced in [1] or the ensemble method proposed in [2], would be appreciated. \n\n\nQuestions:\n- What is the practical benefit of using CEM over other model-free approaches like SAC? Is it much faster in terms of actual time (i.e., the training time per step is smaller)? \n- Why is not the covariance matrix learnt? Is it because the large dimensionality of the parameter space?\n- I understand that each step in the training curves correspond to a single state-transition, but that the points are actually represented per episode (which can be finished after a max number of state transitions), is my understanding correct?\n\nMinor comments:\nSince the state space is introduced as a subset of the real vector space, I assumed it is not countable. Hence, the transition probability distribution should be in Re^+, instead of [0, 1].\n\n[1] S. V. Macua, S. Zazo and J. Zazo, \"Distributed black-box optimization of nonconvex functions,\" 2015.\n[2] S. Khadka, S. Majumdar, T. Nassar, Z. Dwiel, E. Tumer, S Miret, Y. Liu and K. Tumer, \"Collaborative Evolutionary Reinforcement Learning\", 2019.\n", "summary_of_the_paper": "This paper proposes an extension of the Cross-Entropy Method (CEM) for optimisation that consists in an ensemble of multiple standard CEM instances, each one being optimised independently, and the solution at each step is the output of the top performer CEM at that step. Simulation results in a continuous control benchmark show that the proposed algorithm matches or outperforms previous CEM variants. The authors also use a toy problem to illustrate how using an ensemble can help to escape local minimum. \n", "summary_of_the_review": "The algorithmic idea is interesting, well executed for RL and probably useful for optimisation in general, but comparison with other ensemble approaches is missing in the related work to better understand its novelty.\nSimulation results are promising but not fully convincing as it shows similar training performance to previous methods, but it is not clear the final performance of the proposed approach for many of the most interesting environments. \nI lack a clear practical motivation for the CEM approach.\n ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a decentralized version of the CEM technique, where an ensemble of CEM instances run independently from one another and each performs a local improvement of its own sampling distribution. The paper shows that the proposed technique can alleviate the problem of centralized CEM related to converging to a local optimum. The paper includes a theoretical analysis and simulation experiments that show some benefits of the proposed technique over centralized CEM.

The key criticisms from the reviewers include the straightforward nature of the proposed idea, which limits the technical contribution of the paper, as well as the limited improvements over centralized CEM in the simulation experiments. 

In summary, this is a borderline paper. While the paper is well-written and the proposed approach is clearly explained, the lack of strong empirical results that show a pronounced improvement of decentralized CEM coupled with the incremental nature of the idea of decentralized CEM makes me lean toward a rejection.
Decision: Reject