The Differentiable Cross-Entropy Method
Authors: Brandon Amos
Denis Yarats
Venue:  (2020)
Keywords: machine learning
differentiable optimization
control
reinforcement learning

Abstract: We study the Cross-Entropy Method (CEM) for the non-convex optimization of a continuous and parameterized objective function and introduce a differentiable variant (DCEM) that enables us to differentiate the output of CEM with respect to the objective function's parameters. In the machine learning setting this brings CEM inside of the end-to-end learning pipeline in cases this has otherwise been impossible. We show applications in a synthetic energy-based structured prediction task and in non-convex continuous control. In the control setting we show on the simulated cheetah and walker tasks that we can embed their optimal action sequences with DCEM and then use policy optimization to fine-tune components of the controller as a step towards combining model-based and model-free RL.

Under review as a conference paper at ICLR 2020
THE DIFFERENTIABLE CROSS-ENTROPY METHOD
Anonymous authors Paper under double-blind review
ABSTRACT
We study the Cross-Entropy Method (CEM) for the non-convex optimization of a continuous and parameterized objective function and introduce a differentiable variant (DCEM) that enables us to differentiate the output of CEM with respect to the objective function's s parameters. In the machine learning setting this brings CEM inside of the end-to-end learning pipeline where this has otherwise been im- possible. We show applications in a synthetic energy- structured prediction task and in non-convex continuous control. In the control setting we show on the simulated cheetah and walker tasks that we can embed their optimal action se- quences with DCEM and then use policy optimization to fine-tune components the controller step towards combining model-based and model-free RL.
INTRODUCTION
Recent work in the machine learning community has shown how optimization procedures can create new muilding-blocks for the end-to-end machine learning pipeline (Gould et al., 2016; Johnson al., 2016; Amos et al., 2017; Amos & Kolter, 2017; Domke, 2012; Metz et al., 2016; Finn et al., 2017; Belanger et al., 2017; Rusu et al., 2018; Srinivas et al., 2018; Amos et al., 2018). In this paper we focus on the setting of optimizing an unconstrained, non-convex, and continuous objective function fo(x) Ra x + R as f arg min, fo (x), where f is parameterized by 9 E and has inputs E R". If it exists, some gub--dderivative Vg is useful in the machine learning setting to make the output of the optimization procedure end-to-end learnable, For example, 0 could parameterize a predictive model that is generating potential outcomes conditional on happening that you want to optimize over. End-to-end learning in these settings can be done by defining a loss function L on top of é and taking gradient steps VoL. If fo were convex this gradient is easy to analyze and compute when exists and is unique (Gould et al., 2016; Johnson et al., 2016; Amos et al., 2017; Amos & Kolter, 2017). Unfortunately analyzing and computing a "derivative" through the non-conver arg min here is not as easy and is challenging in theory and practice. No such derivative may exist in theory, might not be unique, and even if uniquely exists, the numerical solver being used to compute the solution may not find a global or even local optimum of f. One promising direction to sidestep these issues is to approximate the arg min operation with an explicit optimization procedure that is interpreted as just another compute graph and unrolled through. This is most commonly done with gradient descent as in Domke (2012); Metz et al. (2016); Finn et al. (2017); Belanger et al. (2017); Rusu et al. (2018); Srinivas et al. (2018); Foerster et al. (2018). This approximation adds significant definition and structure to an otherwise extremely ill-defined desiderata at the cost of biasing the gradients and enabling the learning procedure to over-fit to the prerrpparameters of the optimization algorithm, such as the number of gradient steps or the learning rate.
In this paper show that the alternative to unrolling gradient descent for approximating the derivative through an unconstrained, non-convex, and continuous arg min. CEM for optimization is zeroth-order optimizer and works by generating a sequence of samples from the objective function. We show a simple and compu- tationally negligible way of making CEM differentiable that we call DCEM by using the smooth top-k operation from Amos et al. (2019). This also brings CEM into the end-to-end learning process in cases where there is otherwise a disconnection between the objective that is being learned and the objective that is induced by deploying CEM on top of those models.
Method (CEM) (De Boer et al., 2005) is a reasonable

---

Under review as a conference paper at ICLR 2020
We first quickly study DCEM in simple non-convex energy-based learning setting for regression. We contrast using unrolled gradient descent and DCEM for optimizing over a SPEN (Belanger & McCallum, 2016). We show that unrolling through gradient descent in this setting over-fits to the number of gradient steps taken and that DCEM generates a more reasonable energy surface.
Our main application focus is using DCEM in the context of non-convex continuous control. This setting is especially interesting as vanilla CEM is the state-of-the- the-art method for solving the control optimization problem with neural network transition dynamics as in Chua et al. (2018); Hafner al. (2018). We show that DCEM is useful for embedding action sequences into a lower-dimensional space to make solving the control optimization process significantly less computationally and mem- ory expensive. This gives us controller that induces differentiable policy class parameterized by the model-based components. We then use PPO (Schulman et al., 2017) to fine-tune the model- based components, demonstrating that it is possible to use standard policy learning for model-based RL in addition to just doing maximum-likelihenn fitting to observed trajectories.
2 BACKGROUND AND RELATED WORK
2.1 DIFFERENTIABLE OPTIMIZATION BASED MODELING IN MACHINE LEARNING
Optimizattio--asse modeling is way of integrating specialized operations and domain knowledge into end-to-end machine learning pipelines, typically in the form of a parameterized arg min oper- ation. Convex, constrained, and continuous optimization problems, e.g. in Gould et al. (2016); Johnson et al. (2016); Amos et al. (2017); Amos & Kolter (2017), capture many standard layers as special cases and can be differentiated through by applying the implicit function theorem to a set of optimality conditions from convex optimization theory, such as the KKT conditions. Non-comvex and continious optimization problems, e.g. as in Domke (2012); Belanger & McCallum (2016); Metz et al. (2016); Finn et al. (2017); Belanger et al. (2017); Rusu et al. (2018); Srinivas et al. (2018); Foerster et al. (2018); Amos et al. (2018); Pedregosa (2016); Jenni & Favaro (2018); Ra- jeswaran et al. (2019), are more difficult to differentiate through. Differentiation is typically done by unrolling gradient descent or applying the implicit function theorem to some set of optimality conditions, sometimes forming locally convex approximation to the larger non-convex problem. Unrolling gradient descent is the most common way and approximates the arg min operation with gradient descent for the forward pass and interprets the operations as just another compute graph for the backward pass that can all be differentiated through. In contrast to these works, we show how continuous and nonconvex arg min operations can also be approximated with the cross entropy method (De Boer et al., 2005) as an alternative to unrolling gradient descent.
2.2 EMBEDDING DOMAINS FOR OPTIMIZATION PROBLEMS
Oftentimes the solution space of high-dimensioonl optimization problems may have structural prop- erties that an optimizer can exploit to find a better solution or to find the solution quicker than an otherwise naïve optimizer. This is done in the context of meta-learning in Rusu et al. (2018) where gradient-descent is unrolled over a latent space. In the context of Bayesian optimization this has been explored with random feature embeddings, hand-coded embeddings. and auto- nuto--nnoder-leameed embeddings (Antonova et al., 2019; Oh et al., 2018: 8: Calandra et al., 2016; Wang et al., 2016; Garnett et al., 2013; Ben Salem et al., 2019; Kirschner et al., 2019). We show that DCEM is another rea- sonable way of learning an embedded domain for exploiting the structure in and efficiently solving larger optimization problems, with the significant advantage of DCEM being that the latent space is directly learned to be optimized over as part of the end-to-end learning pipeline.
2.3 RL AND CONTROL
High-dimensional non-convex optimization problems that have a lot of structure in the solution space naturally arise in the control setting where the controller seeks to optimize the same objective in the same controller dynamical system from different starting states. This has been investigated in, e.g., planning (Ichter et al., 2018; Ichter & Pavone, 2019; Mukadam et al., 2018; Kurutach et al., 2018: Srinivas et al., 2018: Yu et al., 2019; Lynch et al., 2019), and policy distillation (Wang & Ba, 2019). Chandak et al. (2019) shows how to learn an action space for model-free learning and
2

---

latent Under review as a conference paper at ICLR 2020
Co-Reyes et al. (2018); Antonova et al. (2019) embed action sequences with VAE. There has also been a lot of work on learning reasonable latent state space representations (Tasfi & Capretz, 2018; Zhang et al., 2018; Gelada al., 2019; Miladinović al., 2019) that may have structure imposed to make it more controllable (Watter et al., 2015; Banijamali et al., 2017; Ghosh et al., 2018; Anand al., 2019; Levine al., 2019; Singh et al., 2019). In contrast these works, we learn how to encode action sequences directly with DCEM instead of auto-encoding the sequences. This has the advantages of 1) never requiring the expensive expert's s solution the control optimization problem, 2) potentially being able to surpass the performance of an expert controller that uses the full action space, and 3) being end-to-end learnable through the controller for the purpose of finding space of sequences that DCEM is good at searching over.
Another direction the RL. and control has been pursuing is on the combination of model-based and model-free methods (Bansal et al., 2017; Okada et al., 2017; Jonschkowski et al., 2018; 8; Pereira al., 2018; Amos et al., 2018; Okada & Taniguchi, 2019; Janner et al., 2019; Pong et al., 2018) 8). Amos et al. (2018) proposes differentiable MPC and only do imitation learning the cartpole and pendulum tasks with known or lightly-parameterized dynamics - in contrast, we are able to 1) scale our differentiable controller up the cheetah and walker tasks, 2) use neural network dynamics inside of our controller, and 3) backpropagate a policy loss through the output of our controller and into the internal components.
THE DIFFERENTIABLE CROSS-ENTROPY METHOD (DCEM)
We focus on uses of the Cross-Entropy Method (CEM) (De Boer al., 2005) for optimization in this paper. In this setting, suppose we have a non-convex, deterministic, and continuous objective function fo (z) parameterized by 0 over a domain R" and we want to solve the optimization problem
f arg min fo(
(1)
The original form of CEM is an iterative and zeroth-order algorithm to approximate the solution of eq. (1) with sequence of samples from a sequence of parametric sampling distributions gø defined
over the domain IK"", such as Gaussians.
We refer the reader to De Boer al. (2005) for more details and motivations for using CEM and briefly describe how it works here. Given a sampling distribution 9s- the hyper-parameters of CEM are the number of candidate points sampled in each iteration N. the number of elite candidates to use to fit the new sampling distribution to, and the number of iterations T. The iterates of CEM are the parameters of the sampling distribution. CEM starts with an initial sampling distribution go, (X) E R", and in each iteration t generates N samples from the domain [X,,s] ~ Io. (-), evaluates the function at those points V4,, fx.,,. and re-fits the sampling distribution to the top-k samples by solving the maximum- maximum-likelihood problem'
Φi+1 = arg max Σ n n =(ve)e) (ogg)((e.)))
(2)
+
where the indicator 1(P} is if is true and O otherwise, (X) is the likelihood of X under go and (2) sorts x E R" in ascending order so that (2); S T S.... ... S (2) We can then map from the final distribution gør back to the domain R" by taking the mean of it, i.e. â = E[ (-)l-
Proposition 1. For multivariate isotropic Gaussian sampling distributions we have that d = {u, )², and eq. (2) has a closed-form solution given by the sample mean and variance of the lop-k samples as pt+1 1/5: Eiet, Xt, and of = 1/k EiET, (Xe, -P141)² where the top-k indexing set is I, = (i: Ve,2 S ).
This is well-known in statistics and is discussed in, e.g., Friedman al. (2001).
'The Cross-Entropp Method's 's name comes from eq. (2) more generally optimizing the cross-entropy mea- sure hetween distributions.
3

---

Under review as a conference paper at ICLR 2020
Algorithm DCEM(Ja, 9e, 01;7, k,)
DCEM minimizes a parameterized objective function fo and is differentiable w.r.t. 0. Each DCEM iteration samples from the distribution Iø. starting with Φ1.
fort = to T
do [xr,1]==1 Toh (.)
Sample N points from the domain Evaluate the objective function at those points > Compute the soft top-k projection of the values with eq. (4) Update @ by solving the maximum weighted likelihood problem in eq. (5)
Ut.: - (x(s)))
I = = (*//T)
end for return Elgert+ ())
Differentiating through CEM's output with respect to the objective function's parameters with Vo is useful, e.g., bring CEM into the end-to-end learning process in cases where there is otherwise a disconnection between the objective that is being learned and the objective that induced by deploying CEM top of those models. Unfortunately in the vanilla form presented above the top-k operation in eq. (2) makes & non-differentiable with respect to 0. The function samples can usually be differentiated through with some estimator (Mohamed et al., 2019) such as the reparameterization trick (Kingma & Welling, 2013), which we use in all of our experiments.
The top-k operation can be made differentiable by replacing it with soft version as done in Martins & Kreutzer (2017); Malaviya et al. (2018); Amos et al. (2019), or by using stochastic oracle as in Here we use the Limited Multi-Laber Projection (LML) layer (Amos et al., 2019), which projects points from R" onto the LML polytope defined by
L,, {p E R 10 pSSP5 and 1 ppp= k),
(3)
which is the set of points in the unit n-hypeecuuee with coordinates that sum to k. Notationally, if n implied by the context we will leave it out and write Ls. We propose a temperature-saaler LMI. variant to project onto the interior of the LML polytope with
HL (%/+)
arg min -x y (Holy) s.t. 1 yy==
(4)
0<
where r > is the temperature parameter and Hs(y) = = Ei ys log + (1 - y/) log(1 ys is the binary entropy function. Equation (4) is convex optimization layer and can be solved in a negligible amount of time with a GPU-amenable bracketing method on the univariate dual and quickly backpropagated through with implicit differentiation. We can use the LML layer to make soft and differentiable version of eq. (2) as
@ arg max Σ Ie,
II, log go (X.,)) subject I. 1c (%//7.
(5)
This is now maximum weighted likelihood estimation problem (Markatou et al., 1997; 1998; Wang, 2001; Hu & Zidek, 2002), which still admits an analytic closed-form solution in many cases, e.g. for the natural exponential family (De Boer et al., 2005). Thus using the soft top-k operation with the reparameterization trick, e.g., on the samples from results in a differentiable variant of CEM that we call DCEM and summarize in alg. We note that we usually also normalize the values in each iteration to help separate the scaling of the values from the temperature parameter.
4

---

Under review as a conference paper at ICLR 2020
Proposition 2. The cemperature-scaleee LML layer IIg (*/+) approaches the hard top-k operation as T + 0 when all components of a are unique.
We prove this in app. A by using the KKT conditions of eq. (4). Corollary 1. DCEM captures CEM as special case as T + -
Proposition 3. With an isotropic Gaussian sampling distribution, the update in eq. (5) becomes
M++11 = 1/4 E, I1, and 0f41 = 1/2 E; Ie (Xt,3 -M441) where the soft top-k indexing set is I (1.TT)
This can be proved by differentiating eq. (5), as discussed in, e.g., (De Boer et al., 2005). As a corollary, this captures prop. 1 as 0+.
APPLICATIONS
4.1 ENERGY-BASED LEARNING
Energy-based learning for regression and classification estimate the conditional probability F(yla) of an output E y' given input E X' with parameterized energy function (o(y))) E x X' the optimization problem
such that (yl(z) x exp( (op(-Folylx)) Predictions are made by solving
0 = arg min (s(1)z).
(6)
y
Historically linear energy functions have been well-studied, egg. in Taskar et al. (2005); LeCun et al. (2006), as it makes eq. (6) easier to solve and analyze. More recently non-convex energy functions that are parameterized by neural networks are being explored a popular one being Structured Prediction Energy Networks (SPENs) (Belanger & McCallum, 2016) which propose to model Fg with neural networks. Belanger et al. (2017) suggests to do supervised learning of SPENs by approximating eq. (6) with gradient descent that is then unrolled for T steps, i.e. by starting with some you making gradient updates 3t++1 = + V y Eg (ye (Velar resulting in an output ý yr, defining loss function L top of ÿ, and doing learning with gradient updates V that go through the inner gradient steps.
In this context we can alternatively use DCEM to approximate eq. (6). One potential consideration when training deep mvergy-based models with approximations to eq. (6) is the impact and bias that the approximation going to have on the energy surface. We note that for gradient descent, egg. it may cause the energy surface to overfit to the number of gradient steps that the output of the approximate inference procedure isn't even local minimum of the energy surface. One potential advantage of DCEM is that the output is more likely to be near local minimum of the energy surface so that, e.g., more test-time iterations can used to refine the solution. We empirically illustrate the impact of the optimizer choice on synthetic example in sect. 5.1.
4.2 CONTROL AND REINFORCEMENT LEARNING
Our main application focus is in the continuous control setting where we show how to use DCEM learn a latent control space that is easier to solve than the original problem and induces a dif- ferentiable policy class that allows parts of the controller to be fine-tuned with auxiliary policy or imitation losses.
We are interested in controlling discrete-time dynamical systems with continuous state-action spaces. Let H be the horizon length the controller and 2 be the space of control sequences over this horizon length, e.g. 1l could be multi-dimensional real space or box therein and L/ could be the Cartesian product of those spaces representing the sequence of controls over timesteps. We are interested in repeatedly solving the control optimization problem2
H Σ C((xs, uz) subject to Jz-1 = ftrans (2,,, uc), F1 = Finit:
H1F = arg min
Ma ELL"
For notational convenience omit some explicit variables from the arg min operator when they he inferred by the context and used elsewhere.
5

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Learning an embedded control space with DCEM
Dynamics ftrans, per-step state- cost CT u,) (inducing Co(2; Tinit) hori- zon H, full control space 24 distribution over initial states D
Fixed Inputs:
Learned Inputs:
Decoder lø 2 + 4
while not converged do
Sample initial state Tinit ~
2 = arg minge C2 Finit)
D> Solve the embedded control problem eq. (8) Update the decoder to improve the controller's cost.
4 grad-update( (V)
end while
where we are in an initial system state miit governed by deterministic system transition dynamics farans, and wish to find the optimal sequence of actions 01. such that we find a valid trajectory (f1.H1 U1:H } that optimizes the cost Crr, u.). Typically these controllers horizon control (Mayne & Michalska, 1990) where only the first action W1 is deployed on the real system, new state is obtained from the system, and the eq. (7) is solved again from the new initial state. In this case we can say the controller induces a policy ग (Iinit) = û, that solves eq. (7) and depends on the cost and transition dynamics, and potential parameters therein. In all of the cases we consider furans is deterministic, but may be approximated by stochastic model for learning. Some reinforcement learning settings consider cases where ftrans and C are parameterized and potentially used in conjunction with another policy class.
used for receding
For sufficiently complex dynamical systems, eq. (7) is computationally expensive and numerically instable to solve and rife with sub-optimal local minima. The Cross-Entropy Method is the state-of- the-art method for solving eq. (7) with neural network transitions ftrans (Chua et al., 2018; Hafner et al., 2018). CEM in this context samples full action sequences and refines the samples towards ones that solve the control problem. Hafner et al. (2018) uses CEM with 1000 samples in each iteration for 10 iterations with horizon length of 12. This requires 1000 10 x 12 120, 000 evaluations (?) of the transition dynamics to predict the control to be taken given a system state and the transition dynamics may use a deep recurrent architecture as in Hafner et al. (2018) or an ensemble of models as in Chua et al. (2018). One comparison point here is a model-free neural network policy takes single evaluation for this prediction, albeit sometimes with a larger neural network.
The first application we show of DCEM in the continuous control setting is learn latent action space Z with parameterized decoder fgle 2 - 4d that maps back up to the space of optimal action sequences, which we illustrate in fig. 3. For simplicity starting out, assume that the dynamics and cost functions are known (and perhaps even the ground-truth) and that the only problem is to estimate the decoder in isolation, although we will show later that these assumptions can be relaxed. The motivation for having such a latent space and decoder is that the millions of times eq. (7) is being solved for the same dynamic system with the same cost, the solution space of optimal action sequences 01.H E t has extremely large amount of spatial (over U) and temporal (over time in 4 structure that is being ignored by CEM on the full space. The space of optimal action sequences only contains the knowledge of the trajectories that matter for solving the task at hand, such as different parts of an optimal gait, and not irrelevant control sequences. We argue that CEM over the full action space wastes lot of computation considering irrelevant action sequences and show that these can be ignored by learning a latent space of more reasonable candidate solutions here that we search over instead. Given a decoder, the control optimization problem in eq. (7) can then be transformed into an optimization problem over Z as
arg arrg min Co(z; = Σ Cr(we, ux) ZEZ subject to Iz-1 = ptrans (322, we); 21 Dinit "1.H (2),
(8)
-
which is still challenging non-convex optimization problem that searches over decoder's 's input space to find the optimal control sequence. We illustrate what this looks like in fig. 3, and note the impact of the decoder initialization in app. C.
For notational convenience also omit the dependency of uj on Tinis here.

---

would require storing Under review as a conference paper at ICLR 2020
Unrolled Gradient Descent
Differentiable CEM
0
-2
Figure 1: We trained energy-based model with unrolled gradient descent and DCEM for ID regression task with the target function shown in black. Each method unrolls through 10 optimizer steps. The contour surfaces show the sommalized/lo-scalleee energy surfaces, highlighting that unrolled gradient descent models can overfit to the number of gradient steps. The lighter colors show areas of lower energy.
We propose in alg. to optimize the performance of eq. (7). Every time we solve eq. (8) with DCEM and obtain an optimal latent representation & along with the induced trajectory {xz, ue). step to push down the resulting cost of that trajectory with (²C(2), which goes through the DCEM process that uses the decoder generate samples to obtain We note that the DCEM machinery behind this is not necessary if reasonable local minima is consistently found as this is an instance of diiedifferentiation (Rockafellar Wets, 2009, Theorem 10.13) but practice this breaks down in non con-onvex cases when the minimum cannot be consistently found. Antonova et al. (2019); Wang & Ba (2019) solve related problems in this space and we discuss them in sect. 2.3. also note that to learn action embedding cost functions to compute V2C(2), even if the ground -ruth ones are being used, since the latent space needs to have the knowledge of how the control cost will change as the decoder's parameters change.
DCEM approximately solve eq. (8) and then learn the decoder directly
can take a gradient
still need to differentiate through the transition dynamics and
DCEM in this setting also induces differentiable policy class (Xint)) enables a policy or imitation loss J to be defined on the policy that can fine-tune the parts of the controller (decoder, cost, and transition dynamics) gradient information from VoJ. In theory the same approach could be used with CEM on the full optimization problem in eq. (7). For realistic problems without modification this is intractable and memory-intensive and backpropagating through every sampled trajectory. although as a future direction we note that it may be possible delete some of the low-influence trajectories help overcome this.
(2lo This
5 EXPERIMENTS
We use PyTorch (Paszke et al., 2017) and will openly release our DCEM library, model-based control code, and the source, plotting, and analysis code for all our experiments.
5.1 UNROLLING OPTIMIZERS FOR REGRESSION AND STRUCTURED PREDICTION
In this section we briefly explore the impact of the inner optimizer on the energy surface of a SPEN as discussed sect. 4.1. For illustrative purposes we consider simple unidimensional regression task where the ground-rruth data is generated from f(x) = sin(x) for E 00 2r... We model Plylz) x exp(-Fao [y][x)) with a single neural network Fg and make predictions ÿ by solving the optimization problem eq. (6). Given the ground-truth output y", use the loss C(j.y" ll - 1 and take gradient steps of this loss to shape the energy landscape.
We consider approximating eq. (6) with unrolled gradient descent and DCEM with Gaussian sam- pling distributions. Both of these are trained to take 10 optimizer steps and we use an inner learning rate of 0.1 for gradient descent and with DCEM
10 iterations with 100 samples per iteration and 10 elite candidates, with a temperature of 1. For both algorithms we start the initial iterate at Vo 0. We show in app. that both of these models attain the same loss the training dataset

---

Under review as a conference paper at ICLR 2020
CEM over the full action space
Iteration Samples
Iteration 3 Samples
Iteration Samples
Iteration Samples
VVV
W
Controls
W
Timestop
Timestop
Timestap
over latent action space
Iteration Samples
Iteration 3 Samples
lteration 6 Samples
Iteration Samples
Controls
Timestap
Timestep
Timestep
Timestep
Latent Dim
Latent Dim
Latent Dim 1
Latent Dim
Latent Dim
Visualization of the samples that CEM and DCEM generate to solve the cartpole task starting from the same initial system state. The plots starting at the top-left show that CEM initially starts with temporal knowledge over the control space whereas embedded DCEM's latent space generates a more feasible distribution over control sequences to consider each iteration. Embed- ded DCEM uses order of magnitude less samples and is able generate better solution the control problem. The contours on the bottom show the controller's 's cost surface C(2) from eq. (8) for the initial state the lighter colors show regions with lower costs.
Figure 2:
but, since this is a unidimensional regression task, we can visualize the entire energy surfaces over the joint input-output space in fig. 1. This shows that gradient descent has learned adapt from the initial yo = 0 position to the final position by descending along the function's surface as we would expect, but there is reason why the energy surface should be local minimum around the last iterate ÿ = 910. The energy surface learned by CEM captures local minima around the regression target as the sequence of Gaussian iterates are able capture more global view of the function landscape and need to focus in on a minimum of it for regression. We show ablations in app. B from training for 10 inner iterations and then evaluating with a different number of iterations and show that gradient descent quickly steps away from making reasonable predictions.
Discussion and limitations. We note that other tricks could be used to force the output be at local minimum with gradient descent, such as using multiple starting points or randomizing the number of gradient descent steps taken - our intention here is highlight this behavior in the vanilla case. We also note that DCEM is susceptible overfitting to the hyper-parameters behind it similar, albeit less obvious ways.
8

---

Under review as a conference paper at ICLR 2020
cheetah.rut
walker.walk
H
Full CEM
Latent DCEM
Latent DCEM+PPO
200
400
600
800
1000
200
400
600
800
1000
Evall Reward
Eval Reward
Figure 4: We evaluated our final models by running 100 episodes each on the cheetah and walker tasks. CEM over the full action space uses 10,000 trajectories for control at each time step while embedded DCEM samples only 1000 trajectories. DCEM almost recovers the performance of CEM over the full action space and PPO fine-tuning the model-based components helps bridge the gap.
5.2 CONTROL
5.2. STARTING SIMPLE: EMBEDDING THE CARTPOLE'S ACTION SPACE
We first show that it is possible to learn embedded control space as discussed in sect. 4.2 in an isolated setting. We use the standard cartpole dynamical system from Barto et al. (1983) with a con- tinuous state-action space. We assume that the ground-truth dynamics and cost are known and use the differentiable ground-truth dynamics and cost implemented in PyTorch from Amos et al. (2018). This isolates the learning problem only learning the embedding that can study what this is doing without the additional complications that arise from exploration, estimating the dynamics, learning policy, and other non-stationarities. We show experiments with these assumptions relaxed in sect. 5.2.2.
We use DCEM and alg. learn a 2-dimensional latent space Z [0, 1J that maps back up to the full control space 4 [0, 1 where we focus on horizons of length H/ = 20. For DCEM over the embedded space we use 10 iterations with 100 samples in each iteration and 10 elite can- didates, again with a temperature of 1. We show the details in app. D that we are able to recover the performance of an expert CEM controller that uses an order-of-magnitude more samples fig. shows a visualization of what the CEM and embedded DCEM iterates look like to solve the control optimization problem from the same initial system state. CEM spends a of evaluations on se- quences in the control space that are unlikely to be optimal, such the ones the bifurcate between the boundaries of the control space at every timestep, while our embedded space is able to learn more reasonable proposals.
5.2.2 SCALING UP TO THE CHEETAI AND WALKER
Next show that and show that we can learn a latent control space on top of learned model on the cheetah. run and walker. walk tasks with frame skips of 4 and respectively, from the DeepMind control suite (Tassa et al., 2018) using the MuJoCo physics engine (Todorov et al., 2012). We then tin-ttue the policy induced by the embedded controller with PPO (Schulman al., 2017), sending the policy loss directly back into the reward and latent embedding modules underlying the controller.
relax the assumptions of having known transition dynamics and reward
We start with state-of-ther -frth-rr badel RL approach by noting that the PlaNet (Hafner al., 2018) restricted state space model (RSSM) is a reasonable architecture for proprioceptive-base control addition just pixel-based control. We show the graphical model we use in fig. which maintains deterministic hidden states h and stochastic (proprioceptive) system observations and rewards Tt. We model transitions as hu+1 f ftrans (h,, observations with 24 gndee (h4), rewards with = frew (h,,2,), and map from the latent action space to action sequences with W1.7 fdee (z). We follow the online training procedure of Hafner et al. (2018) to initialize all of the models except for the action decoder pdec, using approximately 2M timesteps. We then use variant of alg. 2 to learn fdee to embed the action space for control with DCEM, which we also do online while updating the models. We describe the full training process in app. E.

---

Under review as a conference paper at ICLR 2020
Our DCEM controller induces differentiable policy class (Tini)) where are the parameters of the models that im- pact the actions that the controller is selecting. We then PPO to define a loss on top of this policy class and fine-tune the components (the decoder and reward module) they improve the episode reward rather than the maximum- likelihood solution of observed trajectories. We chose PPO because we thought it would be able to fine-tune the policy with just a few updates because the policy starting at rea- sonable point, but this did not turn out to be the case and in the future other policy optimizers can be explored. We im- plement this by making our DCEM controller the policy in the Py Torch PPO implementation by Kostrikov (2018). We provide more details behind our training procedure in app. E.
that
u1
U2
h1
hy
h3
21
22
F3
We evaluate our controllers on 100 test episodes and the re- wards in fig. 4 show that DCEM is almost (but not exactly) able to recover the performance of doing CEM over the full less trajec-
Figure Our RSSM with action quence embeddings
action space while using an tory samples (1,000 vs 10,0000). PPO fine-tuning helps bridge the gap between the performances.
Videos of our trained models are available at:
https:/ wtt.g///..o///e///aaaaaooaaaoaaa
Discussion and limitations. DCEM in the control setting has many potential future directions to explore and help bring efficiency and policy-based fine-tuning to model-based reinforcement learning. Much more analysis and experimentation necessary to achieve this as we faced many issues getting the model-based cheetah and walker tasks work that did not arise in the ground-truth cartpole task. We discuss this more in app. E. We also did not focus on the sample complexity of our algorithms getting these proof-of-concept experiments working. We also note that other reasonable baselines on this task could involve distilling the controller into a model-free policy and then doing search on top of that policy, as done in POPLIN (Wang Ba, 2019).
CONCLUSIONS AND FUTURE DIRECTIONS
We have laid the foundations for differentiating through the cross-entropy method and have brought CEM into the end-to-end learning pipeline. Beyond further explorations in the mergy--bseed learn- ing and control contexts we showed here, DCEM can used anywhere gradient descent is unrolled. We find this especially promising for meta-learning, potentially building on LEO (Rusu et al., 201 8). Inspired by DCEM, other more powerful sampling- optimizers could be made diffcrentiable in the same way, potentially optimizers that leverage gradient-based information in the inner op- timization steps (Sekhon & Mebane, 1998; Theodorou et al., 2010; Stulp & Sigaud, 2012; Mah- eswaranathan et al., 2018) 8) or by also learning the hyper-parameters of structured optimizers (Li & Malik, 2016; Volpp et al., 2019; Chen et al., 2017).
ACKNOWLEDGMENTS
We acknowledge the scientific Python community (Van Rossum & Drake Jr, 1995; Oliphant, 2007) for developing the core set of tools that enabled this work, including PyTorch (Paszke et al., 2017), Jupyter (Kluyver et al., 2016), Matplotlib (Hunter, 2007), seaborn (Waskom et al., 2018), numpy (Oliphant, 2006; Van Der Walt et al., 201 1), pandas (McKinney, 2012), and SciPy (Jones et al., 2014).
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Brandon Amos and Zico Kolter. Optnet: Differentiable optimization as layer neural networks. In
Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 136-145. JMLR. org, 2017.
Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. In Proceedings of the 34th Interna- tional Conference Machine Learning- Keaminn-Voluume 70, pp. 146-155, JMLR. org, 2017.
Brandon Amos. Ivan Jimenez, Jacob Sacks, Byron Boots, and J Zico Kolter. Differentiable mpe for end-to-end planning and control. In Advances Neural Information Processing Systems, pp. 8289-8300, 2018.
Brandon Amos, Vladlen Koltun, and Zico Kolter. The limited multi-label projection layer. arXiv preprint arXiv:1906.00 087007 2019.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Mare-Alexandre Côté, and Devon Hjelm. Unsu- pervised state representation learning in atari. arXiv preprint (oxi::990.082220 2019.
Rika Antomova, Akshara Rai, Tianyu Li, and Danica Kragic. Bayesian optimization in variational latent spaces with dynamic compression. arXiv preprint arXiv: 190.00499, 2019.
Ershad Banijamali, Rui Shu, Mohammad Ghavamzadeh, Hung Bui, and Ali Ghodsi. Robust locally-linear controllable embedding. arXiv preprint 17X.0:171..05373, 2017.
S. Bansal, Roherto Calandra, T. Xiao, S. Levine, and C. J. Tomlin. Goal-driven dynamics learning via Bayesian optimization. In IEEE Conference Decision and Control (CDC), pp. 5168 $173, 2017.
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, pp. 834-844 846, 1983.
David Belanger and Andrew McCallum. Structured prediction energy networks. In International Conference on Machine Learning, pp. 983-992, 2016.
David Belanger, Bishan Yang, and Andrew McCallum. End-to-end learning for structured prediction energy networks. In Proceedings of the 34th International Conference on Muchine Learning Searning-Volume Volume 70, pp. 429- 439. JMLR. org, 2017.
Malek Ben Salem, François Bachoc, Olivier Roustant, Fabrice Gamboa, and Lionel Tomaso. Sequential dimen- sion reduction for learning features of expensive black-box. functions. working paper or preprint, February 2019. URL https: /hal. i ves ouvertes. fr/hal 1 01 688329.
Roberto Calandra, Jan Peters, Carl Edward Rasmussen, and Mare Peter Deisenroth. Manifold gaussian pro- cesses for regression. In 2016 International Joint Conference on Neural Networks (IJCNN) pp. IEEE, 2016.
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip S Thomas. Learning action representations for reinforcement learning. arXiv preprint arXiv:1902.00183 2019.
Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap, Malt Botvinick, and Nando de Freitas. Learning learn without gradient descent by gradient descent. In Pro- ceedings of the 34th International Conference on Machine Learning Volume 70, pp. 748 7568756.777 JMLR. org. 2017.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcchre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406.107.. 2014.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabiliitic dynamics models. In Advances Neural Information Processing Systems, pp. 4765, 2018.
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (clus). arXiv preprint 1:X1:::51110722 2015.
John D Co- Co-Reyes, YuXuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self- consistent trajectory autnencoder: Hierarchical reinforcement learning with trajectory emheddings. arXiv preprint 1806.00806.0288 2018.
11

---

Under review as a conference paper at ICLR 2020
Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, (34(())19-8)) 67, 2005.
Justin Domke. Generic methods optimization saaaaiiaattt---ddddeddeeee modeling. In Artificial Intelligence and Statistics, pp. 318-326, 2012.
Chelsea Finn, Pieter Abbecl, and Sergoy Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126- 1135. JMLR. org, 2017.
Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In Proceedings the 17th International Conference Au- tonomous Agents and MultiAgent Systems, pp. 122-130. International Foundation for Autonomous Agents and Multiagent Systems, 2018.
Jerome Friedman, Trevor Hastic, and Robert Tibshirani. The elements of statistical learning, volume 1. Springer series in statistics New York, 2001.
Roman Garnett, Michael A Oshorne, and Philipp Hennig. Active learning of linear embeddings for gaussian processes, arXiv preprint arXiv: 131.6674, 2013.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare. Deepmdp: Learning continuous latent space models for representation learning. arXiv preprint arXiv:1906.02366, 2019.
Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with condiiondt policies. arXiv preprint 1cX.vv:811.008111 2018.
Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. On diffcrentiating parameterized argmin and argmax problems with application to bi-level optimization. arXiv preprint arXiv:1607.05444 2016.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lec, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: /8/ 1.04551, 2018.
Feifang Hu and James V Zidek. The weighted likelihood. Canadian Journal of Statistics, 30(3)347-333 371, 2002.
John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & eenginering, 93):900 2007.
Brian Ichter and Marco Pavone. Robot motion planning in learned latent spaces. IEEE Robotics and Automation Letters, 403):400722144, 2019.
Brian Ichter, James Harrison, and Marco Pavone. Learning sampling distributions for rohot motion planning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7087-7094, IEEE, 2018.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. arXiv preprint arXiv:1966.0825: 08253, 2019.
Simon Jenni and Paolo Favaro. Deep bilevel learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 618-633, 2018.
Matthew Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeop R Datta. Composing graphical models with neural networks for structured representations and fast inference. In Advances in neural information processing systems, pp. 2946-2954, 2016.
Eric Jones, Travis Oliphant, and Pearu Peterson. [SciPy]: Open source scientific tools for (Python]. 2014.
Rico Jonschkowski, Divyam Rastogi. and Oliver Brock. Differentiable particle filters: End-to-end learning with algorithmic priors. arXiv preprint 1 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Johannes Kirschner. Mojmár Mutny, Nicole Hiller, Rasmus Ischebeck, Andreas Krause. Adaptive and safe bayesian optimization in high dimensions via one-dimensional subspaces. arXiv preprint arXiv:1902.03229 2019.
Thomas Kluyver, Benjamin Kelley, Fernando Pérez, Brian E Granger, Matthias Bussonnier, Jonathan Frederie, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, et al. Jupyter notebooks-a publishing format for reproducible computational workflows. In ELPUB, pp. 87-90, 2016.
12

---

Under review as a conference paper at ICLR 2020
Πya Kostrikov. Pytorch implementations of reinforcement learning algorithms. htttps://github.com -kostrikev/pytorch-eeeeee a2c- pwcc--ppopccttt- gail, 2018.
Thanard Kurutach, Aviv Tamar, Ge Yang. Stuart J Russell, and Pieter Abheel. Learning plannable representa- tions with causal infogan. Advances Neural Information Processing Systems, pp. 5733-8744, 2018.
Yann LeCun, Sumit Chopra, Raia Hadsell, Ranzato, and F Huang. tutorial on energy-based learning. Predicting structured data, 1(), 2006.
Nir Levine, Yinlam Chow, Rui Shu. Ang Li. Mohammad Ghavamzadeh, and Hung Bui. Prediction, consistency, curvature: Representation learning for locally-linear control. arXiv preprint Xiviv 1000:09990000000 2019.
Ke Li and Jitendra Malik. Learning optimize. arXiv preprint wrXiv:1606.01885, 2016.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Ser- manet. Learning latent plans from play. arXiv preprint arXiv:1903 1903,01973 2019.
Niru Maheswaranathan, Luke Metz, George Tucker, Dami Choi, and Jascha Sohl-Dickstein Guided evolu- tionary strategies: Augmenting random search with surrogate gradients. arXiv preprint arXiv:1806. 10230, 2018.
Chaitanya Malaviya, Pedro Ferreira, and André FT Martins. Sparse and constrained attention for neural ma- chine translation. arXiv preprint arXiv: 105,00411 2018.
Marianthi Markatou, Ayanedranath Basu, and Bruce Lindsay. Weighted likelihood estimating equations: The discrete case with applications to logistic regression. Journal of Statistical Planning and Inference, 57(2): 215-232, 1997.
Marianthi Markatou, Ayanendranath Basu, and Bruce G Lindsay. Weighted likelihood equations with bootstrap root search. Journal of the American Statistical Association, 93(442):740-750, 1998.
André FT Martins and Julia Kreutzer. Learning what's easy: Fully differentiable neural easy-first taggers. In Proceedings of the 2017 Conference on Empirical Methods Natural Language Processing, pp. 349-362, 2017.
David Q Mayne and Hannah Michalska. Receding horizon control of nonlinear systems. IEEE Transactions on automatic control, 35(7):814 824, 1990.
Wes McKinney. Python for data analysis: Data wrangling with Pandas, NumPy, and IPython. "O'Reilly Media, Inc.", 2012.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. CoRR, abs//611..00000 1.02163, 2016. htt::/// wwtp:/iwwwoggo///oo//oooooooos /161 611 C21 62..33
Dorde Miladinović, Muhammad Waleed Gondal, Bernhard Schölkopf, Joachim M Buhmann, and Stefan Bauer. Disentangled state space representations. arXiv preprint arXiv: 7906, 19066.3200.0 2019.
Shakir Mohamed, Mihaela Rosca, Michael Figumov, and Andriy Mnih. Monte carlo gradient estimation in machine learning, arXiv preprint arXiv:1906. 2019.
Mustafa Mukadam, Jing Dong, Xinyan Yan, Frank Dellaert, and Byron Boots. Continuous-time gaussian process motion planning via probabilistic inference. The International Journal of Robotics Research, 37 (11):131--1340, 2018.
Vinod Nair and Geoffrey E Hinton. Rectified lincar units improve restricted boltzmann machines. In Proceed- ings of the 27th international conference on machine learning (ICML-I0), pp. 807-814, 2010.
ChangYong Oh. Efstratios Gavves, and Max Welling. Bock: Bayesian optimization with cylindrical kemels. arXiv preprint arXiv:1806. 01619. 2018.
Masashi Okada and Tadahiro Taniguchi, Variational inference mpe for bayesian model-based reinforcement learning. arXiv preprint arXiv: 04200, 2019.
Masashi Okada, Luca Rigazio, and Takenobu Aoshima. Path integral networks: End-to-end differentiable optimal control. arXiv preprint arXiv: 09577797777777 2017.
Travis Oliphant. guide to NumPy, volume 1. Trelgol Publishing USA, 2006.
Travis E Oliphant. Python for scientific computing. Computing in Science Engineering, 9(3):10-20, 2007.
13

---

Under review as a conference paper at ICLR 2020
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation pytorch. 2017.
Fabian Pedregosa. Hyperparameter optimization with approximate gradient. arXiv preprint orXi:/6602023355 2016.
Marcus Pereira, David D Fan, Gabriel Nakajima An, and Evangelos Theodorou. Mpc-inspired neural network policies for sequential decision making. arXiv preprint (80.xv:8802.00000 2018.
Vitchyr Pong, Shixiang Gu. Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free deep for model-based control. arXiv preprini arXiv: 09081, 2018.
Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. arXiv preprint arXiv: (xXtv:900,.000 (909,04630,, 2019.
C. Radhakrishna Rao. Convexity properties of entropy functions and analysis of diversity. Lecture Notes- Monograph Series, pp. 68-77, 1984.
Tyrrell Rockafellar and Roger J-B Wets. Variational analysis, volume 317. Springer Science & Business Media, 2009.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05966 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint 1707:0777770647,, 2017.
Jasject Sekhon and Walter R Mchane. Genetic optimization using derivatives. Political Analysis, 7:187-210, 1998.
Sumeet Singh, Spencer M Richards, Vikas Sindhwani, Jean-Jacques Slotine, and Marco Pavone. Learning stabilizable nonlinear dynamics with contraction-base regularization. arXiv preprint arXiv:1907.1322, 2019.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks. arXiv preprint 1804,v8804.0000 2018.
Freek Stulp and Olivier Sigaud. Path integral policy improvement with covariance matrix adaptation. arXiv preprini 120. 4621, 2012.
Norman Tasfi and Miriam Capretz. Dynamic planning networks. arXiv preprint arXiv:1812.1122 2018.
Ben Taskar, Vassil Chatalbashev, Daphne Koller, and Carlos Guestrin. Learning structured prediction models: large margin approach. In Proceedings of the 22nd international conference on Machine learning, pp. 896-903 903. ACM, 2005.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint :tti://801.009900 2018.
Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. A generalized path integral control approach to rein- forcement learning. journal of machine learning research. 1 (3Nov):31373311) 2010.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine model-based control. 2012 JEEE/RS/ International Conference on Intelligent Robots and Systems, pp. 5026 5033. IEEE, 2012.
Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux. The numpy array: a structure for efficient numerical computation. Computing Science & Engineering, 13(2):22, 2011.
Guido Van Rossum and Fred I. Drake Jr. Python reference manual. Centrum Wiskunde Informatica Amsterdam, 1995.
Michael Volpp, Lukas Fröhlich, Andreas Doem, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for bayesian optimization. arXiv preprint arXiv:1904.0266 026442, 2019.
Steven Xiaogang Wang Maximum weighted likelthood estimation. PhD thesis, University of British Columbia, 2001.
Tingwu Wang and Jimmy Ba. Exploring model-hased planning with policy networks. arXiv preprint arXiv: /9066 0066006449 2019.
14

---

Under review as a conference paper at ICLR 2020
Ziyu Wang, Frank Hutter, Masrour Zoghi, David Matheson, and Nando de Feitas. Bayesian optimization in billion dimensions random embeddings. Journal of Artificial Intelligence Research, 55:361-387, 2016.
Michael Waskom, Olga Botvinnik, Drew O'Kane. Paul Hobson, Joel Ostblom, Saulius Lukauskas, David C Gemperline, Tom Augspurger Yaroslav Halchenko, John B. Cole, Jordi Warmenhoven. Julian de Ruiter, Cameron Pye, Stephan Hoyer, Jake Vanderplas, Santi Villalba, Gero Kunter, Eric Quintero, Pete Bachant, Marcel Martin, Kylc Meyer, Alistair Miles, Yoav Ram, Thomas Brumnor, Tal Yarkomi, Mike Lee Williams, Constantine Evans, Clark Fitzgerald, Brian, and Adel Qalich, mwaskom/scabom:: v0.9.0 (july 2018), July 2018. URL https:: //oii. www//o//oooooooooooooooo 10. 5281/ zenodo... 1313201.
Manuel Watter, Jost Springenberg. Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from images. In Advances in neural information processing systems, pp. 2746-2754, 2015.
Tianhe Yu, Gleb Shevchuk, Dorsa Sadigh, and Chelsea Finn. Unsupervised visuomotor control through distri- butional planning networks, arXiv preprint 1rXiv=1902.05542, 2019.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abheel, Matthew J Johnson, and Sergey Levine. So- lar: Deep structured latent representations for model-based reinforcement learning. arXiv preprint arXiv: 2018.
15

---

Under review as a conference paper at ICLR 2020
A
PROOF OF PROP. 2
Proof. We first note that solution exists to the projection operation, and is unique, which comes from the strict convexity of the objective (Rao, 1984). The Lagrangian of the temperature-scaled LML projection in eq. (4) is
L( = -x' -x-y -Hb(v) + u(k - -)
(9)
Differentiating eq. (9) gives
y
V yy (yy = - r log
v
(10)
- y
and the first-order optimality condition VyL n²) = 0 gives v = o + v²)), where o is 0+ gives
the sigmoid function. Using lem.
(1
if r; 1²
yr
Ij < <-
(11)
1/2 otherwise.
Substituting this back into the constraint 1 yyy h gives that T()) < -v* < (x))+41 where S T(x).. Thus we have that
(2x) sorts E _ in ascending order that π (x)1 I T yr = (x, ( which when v; is the top-k components of x and otherwise, and
therefore the temperature-scale LML layer approaches the hard top-k function as - +0 O
Lemma 1.
(1
i
lim ('*/>) = +
0
if <0
(12)
1/2
otherwise,
where (0/+)) = (1 + exp(-=//3) is the lemperature-scaled sigmoid.
16

---

Under review as a conference paper at ICLR 2020
MSE Loss
MSE Loss
10
Unrolled GD DCEM
10"
109
10
10-2
10-2
10
10-4
Unrolled GD
DCEM
10-5
0 10000
20000
30000
10000
50000
60000
12
14
16
18
20
Number of Updates
Number of Inner Loop Iterations
Figure 5: Left: Convergence of DCEM and unrolled GD on the regression task.
Right:
Ablation showing what happens when DCEM and unrolled are trained for 10 inner steps and then different number of steps is used test-time. We trained three seeds for each model and the shaded regions show the 95% confidence interval.
Unrolled Gradient Descent
Differentiable CEM
2.5
y
2.5
-5.0
T
C
Number of Inmer Loup lteraiooss
10
20
XI
Figure 6: Visualization the predictions made by ablating the number inner loop iterations for unrolled GD and DCEM. The ground-truth regression target is shown in black.
B MORE DETAILS: SIMPLE REGRESSION TASK
Figure 5 (left) shows the convergence of unrolled GD and DCEM the training data, showing that they are able to obtain the same training loss despite inducing very different energy surfaces. Figure (right) and fig. 6 shows the impact of training gradient descent and DCEM take 10 inner optimization steps and then ablating the number inner steps at test-time.
17

---

Under review as a conference paper at ICLR 2020
ReLU
Init Scale 1.0
Init Scale
2.0
Init Scale 4.0
Init Scale 6.0
1.0
0.8
Output
0.2
0.0
Output Dimension
Output Dimension
Outout Dimension
Outout Dimension
ELU
Init Scale 1.0
Init Scale ==
Init Scale 4.0
Init Scale a= 6.0
1.0
Valuee
0.8
0.4
0.2
0.0
15
Output Dimension
Output Dimension
Output Dimension
Output Dimension
Hyperbolic Tangent
Init Scale 1.0
Init Scale 2.0
Init Scale 4.0
Init Scale 6.0
1.0
Output
0.4
0.2
Output Dimension
Output Dimension
Output Dimension
Outbut Dimension
Figure 7: Impact of the activation function on the initial decoder values
C MORE DETAILS: DECODER INITIALIZATIONS AND ACTIVATION FUNCTIONS
We have found the decoder to be influenced by the activation function that's used with it and have found the ELU (Clevert al., 2015) to perform the best. fig. conveys some intuition behind this choice. We randomly initialize neural network = fa(=) every layer weight W,, and then scale the weights with a0. them through fae. and plot the outputs. The ReLU (Nair & Hinton, 2010) induces extremely biased distribution which is seen more prevalently as a grows that is not as present when using the ELU hyperbolic tangent since they almost linear around zero. Despite the reasonable looking initializations for the hyperbolic tangent, we found that it does not perform as well in practice in our experiments. We found that the initial scale a of the decoder's parameters is also important for learning because of the network is not initially producing samples that cover the full output space as shown with a seems hard for it to learn how expand to the full output space.
(z) with no biases, where 0 (W:): ) for then sample ~ NO I), pass
18

---

Under review as a conference paper at ICLR 2020
D MORE DETAILS: CARTPOLE EXPERIMENT
In this section we discuss some of the ablations we considered when learning the latent action space for the cartpole task. In all settings we use DCEM unroll 10 inner iterations that samples 100 candidate points in each iteration and has an elite set of 10 candidates.
For training, randomly sample initial starting points of the cartpole and for validation we use fixed set of initial points. Figure 8 shows the convergence of models as we vary the latent space dimension and temperature parameter, and fig. shows that DCEM able to fully recover the expert performance on the cartpole. Because we are operating in the ground-truth dynamics setting we measure the performance by comparing the controller costs. We use case where we optimize over the latent space with vanilla CEM and then update the decoder with V.C(ff(())) where the gradient doesn't go back into the optimization process that produced å. This is non-convex min differentiation and is reasonable when & is near-optimal, but otherwise is susceptible to making the decoder difficult to search over.
indicate the
These results show few interesting points that come up in this setting, which of course may be dif- ferent in other settings. Firstly that with a two-dimensional latent space, all of the temperature values able find reasonable latent space some point during training. However after more updates, the power-temperature experiments start updating the decoder in ways that make it more difficult to search over and start achieving worse performance than the 7 1 case. For higher-dimensional latent spaces, the DCEM machinery is necessary to keep the decoder searchable. Furthermore we notice that just 16-dimensional latent space for this task can be difficult for learning, one reason this could be is from DCEM having too many degrees of freedom in ways it can update the decoder improve the performance of the optimizer.
19

---

Under review as a conference paper at ICLR 2020
Space Dimension:
Space Dimension:
Space
Thahinn Loss
Vallcaatoon LOGS
0.1
0.0)
Figure 8: Training and validation loss convergence for the cartpole task. The dashed horizontal line shows the loss induced by an expert controller. Larger latent spaces seem harder to learn and as DCEM becomes differentiable, the embedding more difficult to learn. The shaded regions show the 95% confidence interval around three trials.
Improvement Factor
1.0
0.5
2
8
16
Latent Space Dimension
1.0
0.1
0.0)
Figure 9: Improvement factor the ground-trutt cartpole task from embedding the action space with DCEM compared to running CEM on the full action space, showing that DCEM is able to recover the full performance. We use the DCEM model that achieves the best validation loss. The eтror lines show the 95% confidence interval around three trials.
20

---

Under review as a conference paper at ICLR 2020
PCA
DCEM -- 1.0
DCEM 0.1
DCEM y- 0.0
Latent Dim
Cast
Latent Dim
Best Cost 0001
A Best Cost 4.29
Zent Cost 2.02
Labent Dim
Best Codt 36 37
BeSt COZ 3334
Best Cast 19.34
Best Cot 35.40
Latent Dim
Bast
Cost: -4.81
Cant-10
Latent Dim
Latert Dim
Latent Dim
Latent Dim 1
Latent Dim 1
Latent Dim
Figure 10: Learned DCEM reward surfaces for the cartpole task. Each row shows different initial state of the system. We can see that as the temperature decreases, the latent representation can still capture near-optimal values, they are in much narrower regions of the latent space than when 7=1 1.
21

---

Under review as a conference paper at ICLR 2020
Algorithm 3 PlaNet (Hafner et al., 2018) variant that we use for proprioceptive control with optional DCEM embedding
> Models: a deterministic state model, a stochastic state model, a reward model, and (if using
DCEM) an action sequence decoder.
Initialize dataset 'D with S random seed episodes.
Initialize the transition model's deterministic hidden state ho and initialize the environment, obtaining the initial state estimate Zi..
CEM-Solve can use DCEM or full CEM
for t = ,,...,, do
U < (MM--oolveba-111
Add exploration noise p(c) to the action W4-
(re, [,, f- env.step((u))
Properly restarting if necessary
Add [T,, ue; de] D
h;
Uz, dz)
if (mod update-interval) then
Sample trajectories r = [rr Zr, Ur. dr dr DD from the dataset.
Obtain the hidden states the (hr #r from the model.
Compute the multi-step likelihood bound hr,
(Hafner et al., 2018, eq 6.) Optimize the likelihood bound
/ f- grad-update( VaLa (T,
if using DCEM then
8, arg min.e 2 Ca(z; f7) 0 t grad-update( V E, Co(2+)
D Solve the embedded control problem in eq. (8) Update the decoder
end if
end if
end for
E MORE DETAILS: CHEETAH AND WALKER EXPERIMENTS
For the cheetah. run and walker. walk DeepMind control suite experiments we start with modified PlaNet (Hafner et al., 2018) architecture that does not have a pixel decoder. We started with this over PETS (Chua et al., 2018) to show that this RSSM is reasonable for proprioceptive- based control and not just pixel-based control. This model graphically shown in fig. and has 1) a deterministic state model h f(,-1,, 1 we-1). 2) stochastic state model 2; p(x,, h2), and 3) a reward model: Tt ~ p(r: zt). In the proprioceptive setting, we posit that the deterministic state model is useful for multi-step training even in fully observable environments as it allows the model to "push forward" information about what is potentially going to happen in the future.
For the modeling components, follow the recommendations in Hafner et al. (2018) and use GRU (Cho et al., 2014) with 200 units as the deterministic path in the dynamics model and imple- ment all other functions Distributions over the state space are isotropic Gaussians with predicted mean and standard devia- tion. We train the model to optimize the variational bound on the multi-step likelihood as presented in (Hafner et al., 2018) on batches of size 50 with trajectory sequences of length 50. We start with seed episodes with random actions and in contrast to Hafner et al. (2018), we have found that interleaving the model updates with the environment steps instead of separating the updates slightly improves the performance, even in the pixel-based case, which we do not report results on here.
fully-connected layers, also with 200 units with ReLU activations.
For the optimizers we either use CEM over the full control space DCEM over the latent control space and use a horizon length of and 10 iterations here. For full CEM, we sample 1000 candi- dates in each iteration with 100 elite candidates. For DCEM we use 100 candidates in each iteration with 10 elite candidates.
Our training procedure has the following three phases, which we set up isolate the DCEM addi- tions. We evaluate the models output from these training runs on 100 random episodes in fig. 4 the main paper. Now that these ideas have been validated, promising directions of future work include trying to combine them all into a single training run and trying to reduce the sample complexity and number of timesteps needed to obtain the final model.
22

---

Under review as a conference paper at ICLR 2020
Phase 1: Model initialization. We start in both environments by launching single training run of fig. 11 to get initial system dynamics. fig. 11 shows that these starting points converge to near-state- of-the-art performance on these tasks. These models take slightly longer to converge than in (Hafner et al., 2018), likely due to how often we update our models. We note that at this point, it would be ideal to use the policy loss to help fine-tune the components so that policy induced by CEM on top of the models can be guided, but this is not feasible to do by backpropagating through all of the CEM samples due to memory, so we instead next move on initializing a differentiable controller that is feasible to backprop through.
Phase 2: Embedded DCEM initialization. Our goal in this phase is to obtain a differentiable controller that is feasible to backprop through.
Our first failed attempt to achieve this was to use offline training on the replay buffer, which would have been ideal as it would require no additional transitions to be collected from the environment. We tried using alg. 2, the same procedure we used in the ground-truie truth cartpole setting, to generate an embedded DCEM controller that achieves the same control cost on the replay buffer as the full CEM controller. However we found that when deploying this controller on the system, it quickly stepped off of the data manifold and failed to control it this seemed to be from the controller finding holes in the model that causes the reward to be over-predicted.
We then used online data collection process identical to the one we used for phase I to jointly learn the embedded control space while updating the models so that the embedded controller doesn't t find bad regions in them. We show where the DCEM updates fit into alg. 3. One alternative that we tried to updating the decoder to optimize the control cost on the samples from the replay buffer is that the decoder can also be immediately updated after planning at every step. This seemed nice since it didn't 't require any additional DCEM solves, but we found that the decoder became too biased during the episode as samples at consecutive timesteps have nearly identical information.
For the hyper- parameters, we kept most of the DCEM hhper-parameter parameters fixed throughout this phase to 100 samples, 10 elites, and a temperature
1. We ablated across 1) the number of DCEM iter- ations taken to be {3.5, 10), 2) deleting the replay buffer from phase I or not, and 3) re-initializing the model or not from phase 1. We report the best runs that we use as the starting point for the next phase in fig. 12, which achieve reasonable performance but don't 't match the performance of doing CEM over the full action space. These runs all use 10 DCEM iterations and both keep the replay buffer from phase 1. The Cheetah run keeps the models from phase 1 and the Walker re-initializes the models. The cheetah curve around timestep 600k shows, the stability here can be improved as sometimes the decoder finds especially bad regions in the model that induce extremely high losses.
Phase 3: Policy optimization into the controller. Finally now that we have a differentiable policy class induced by this differentiable controller we can do policy learning fine-tune parts of it. We initially chose Proximal Policy Optimization (PPO) (Schulman et al., 2017) for this phase because thought that it would be able to fine-tune the policy in few iterations without requiring good estimate of the value function, but this phase also ended up consuming many timesteps from the environment. Crucially in this phase, we do not do likelihood fitting all, as goal is show that PPO can be used as another useful signal to update the parts of a controller - we did this to isolate the improvement from PPO but in practice we envision more unified algorithms that use both signals at the same time. Using the standard PPO each PPO training step and ablate across 1) the number of passes to make through these episodes (1, 4). 2) every combination of the reward, transition, and decoder being fine-tuned or frozen, 3) using a fixed variance of 0. around the output of the controller or learning this, 4) the learning rate of the fine-tuned model-based portions (10-1, 10-53. Figure 13 shows the results of the best runs from this search.
we collect 10 episodes for
We conclude by showing the PPO-fine-tunee DCEM iterates for solving single control optimiza- tion problem from a random system state for the cheetah fig. 14. and walker fig. 15 tasks.
23

---

Under review as a conference paper at ICLR 2020
Past checlaaas
Ep ppadde Baward
Low
San Lons
12 loss
12117
2x10
20
200
120
1900 Lh0 2090
Z00 15e
320
Eplence
12-
400
1h0
430
100
J807
1307 D50
11.55
180
Train Reward
Eval Reward
Figure 11: Phase 1: The two base proprioceptive PlaNet training runs that use CEM over the full action space. The evaluation loss uses 10 episodes and we show a rolling average of training loss.
34:07
Sipttott Rewanrt
3.3775
3331
Figure 12: Phase 2: The training runs of learning an embedded DCEM controller with online
updates. The evaluation loss uses 10 episodes and we show rolling average of the training loss.
PlaNet + PCE cheetah.run
PlaNet+ DCEM+ walker walk
750
mw
MMM
Evall Reward
Evall Reward
800
250
200
550
500
1000
1500
20000
2500
3000
200
400
600
800
1000
12000
1400
PPO Steps
PPO Steps
Figure 13: Phase 3: The training run of UPO-fine-tuning into the model-basee components - we only use the PPO updates tune these components and do optimize for the likelihood in this phase. The evaluation loss uses 10 episodes.
24

---

Under review as a conference paper at ICLR 2020
Cantrol Dim
Control Dim
Control Dim
Control Dim 3
Control Dim
Control Dim
1.0
Contral Values
0.5
0.0
-1.0
1.0
Control Values
0.0
-10
Control Values
Control VartrelValuus Values
0.5
-0.5
Timestep
Timestep
Timestep
Timestep
Timestep
Timestep
Figure 14: Visualization the DCEM iterates on the cheetah to solve a single control problem starting from random initial system state The rows show iterates 1, 7, 10 from the top bottom.
25

---

Under review as a conference paper at ICLR 2020
Control Dim
Control Dim
Control Dim 2
Control Dim 3
Control Dim
Control Dim
Control Values
0.5
-0.5
Control Values
0.5
1.0
Control Valves
0.5
0.0
1.0
1.0
Control Values
0.0
-1.0
Timestep
Timestep
Timestep
Timestep
Timestep
Timestep
Visualization of the DCEM iterates the walker solve single control problem starting from random initial system state. The rows show iterates 1, 5, 7, 10 from the top to bottom.
Figure 15:
26


Reviews:
[{"review_id": "rJgo2xHotB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*\nVarious optimization methods can be wrapped to form black-box differentiable deep learning modules. This allows end-to-end learning of energy functions that can be used, for example, in continuous control. There is a whole cottage industry of designing these modules. Advancements in this field are of broad interest to the ICLR community. This paper proposes to unroll the cross entropy method, which is very different than the standard practice of unrolling gradient descent. Experiments on continuous control benchmarks demonstrate that this can be used to learn a latent space in which test-time optimization is performed. By doing this optimization in latent space, it can be performed much faster than in the raw high-dimensional space.\n\n*Overall Assessment*\nThe paper is well written and the technical contribution is explained well. Both evolutionary search methods (e.g. CEM) and unrolled gradient-based optimizers are very popular in ML these days. This paper will be of interest to many readers, since it works at the interface between these.\n\nI do not have much background in continuous control, model-based RL, etc. Therefore, it is hard for me to assess whether the experiments compare to the right baselines, etc. It appears to me that the experiments on cheetah and walker do not compare a particularly broad set of methods. They only compare within the design space of DCEM. Furthermore, the key result in these experiments is that the DCEM policy results in more efficient inner optimization (such that running the policy is faster). The overall messaging of the paper was not about reducing the costs of executing the policy but in improving performance. Such a result is not provided in these experiments.\n\nI have worked extensively with unrolled optimizers and can speak to the correctness and usefulness of the paper's methodological contribution.and the experiments in sec 5.1. However, these are more for providing insight into the method, and are not large-scale experiments.\n\nMy evaluation is a weak reject, since the paper would be greatly improved by stronger empirical results for the large-scale continuous control benchmarks with comparison to a broader set of methods.\n\n*Comments*\n\nThe empirical advantage of DCEM vs. unrolled GD is clear, but it's not clear to me what the intuition behind this is. You write \"one potential advantage of DCEM is that the output is more likely to be near a local minimum of the energy surface so that, e.g., more test-time iterations can be used to refine the solution.\" Why would GD not want the output to be be near a local minimum. Also, why is DCEM not also sensitive to the number of steps? The variance of the CEM distribution gives a natural lengthscale similar to the step size in GD. You discuss this further at the end of sec 5.1 Are there simple experiments you could do that compare the robustness of GD (such as with random restarts or unrolled Langevin dynamics) vs. DCEM?\n\nIn the standard CEM, doing weighted MLE with 0-1 weights coming from top-k is useful because the set of examples for MLE is size k, which yields computational savings. However, if you can tolerate doing weighted MLE on all available samples, then there may be better ways to set the weights than using a softened version of top-k. See, for example, the example weights in 'Design by Adaptive Sampling' (arxiv.org/abs/1810.03714). Can you comment on the suitability of other weighting schemes besides top-k? Also, will your relaxed top-k perform sensibly when there are many ties in the observed f(x) values?\n\nThe principal critique of the paper is the positive DCEM results on cheetah + walker are mostly about runtime, instead of performance. Can you speak more to why you don't think it is providing better performance as well? Perhaps the latent space is useful, for example, for transfer learning + adaptation? \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HyxhrAGRtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a differentiable variant of the Cross-Entropy method and shows its use for a continuous control task.   \n- It introduces 4 hyper-parameters and it is not clear how robust the method is to these. \n- Although the idea is interesting, I think the paper needs a more rigorous experimental comparison with previous work and other methods.\nDetailed review below:\n- The abstract should mention clearly that the proposed method allows you to differentiate through argmin operation and can be used for end to end learning. Similarly, please reframe parts of the introduction to make it more accessible to a general reader. For example, in the introduction,  \"approximation adds significant definition and structure to an otherwise...\". This statement requires more context to make it useful. Similarly, \"smooth top-k operation\" is not clear. \n- Is there a way to guarantee that the solution found by (D)CEM is a reasonable approximation to the argmin. For unrolled gradient descent, this can be done by looking at the gradient wrt x. \n- It might be more useful to explain CEM before the related work section or just moving the related work to the end. \n- Section 3: If the paper is about CEM, please give some motivation and details rather than just citing De Boer, 2005. \n- There is a notation clash between \\pi for the sort and policy later in the paper. Similarly, \"t\" is for both for the iterations of CEM and the time-stamp in the control problem. \n- I don't understand how Proposition 1 adds to the paper. This is a standard thing. Similarly for Proposition 3. \n- Isn't there an easier way to make the top-k operation soft - by sampling without replacement proportional to the probabilities? Please justify this design decision. Similarly, how is the temperature \\tau chosen in practice?\n- Please explain the paragraph: \"Equation 4 is a convex optimization layer and... GPU-amenable..\" Isn't this critical to the overall scalability of this method?\n- - How are the hyper-parameters for CEM  chosen - the function g(.), the value of k, \\tau, T chosen in practice. If the criticism of GD is that it overfits to the hyper-parameters - learning rate and the number of steps, why isn't this a problem with (D)CEM.  \n- Section 4: Since you're comparing against unrolled GD, please formally state what the method is. \n- Section 4.2: How is the structure of Z decided, that is how do you fix the space for searching for the policy in the Z space? \n- There are other methods that auto-encode the policy u_1:H to search the space. How does the proposed method compare to these methods? This is important to disentangle the effect of GD vs CEM and that of just searching in a more tractable space of policies. \n- Section 5.1: How is the number of optimizer steps (=10) decided? Also, how is the learning rate for GD picked. Is the performance of unrolled GD worse for all values of \\eta, even after a grid-search over the learning rates?\n- For Section 5.2, please compare to baselines mentioned in the paper. Also, there needs to be an ablation/robustness study for the DCEM method. \n\n\n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xaWlkCFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After reading authors' response, I am sticking to my original decision. Authors addressed most of the issues I raised and I am happy with their response; however, I still believe the paper should not be accepted since it is not adding enough value. The problem is important and impactful. However, the algorithmic idea comes from LML (Amos 2019), and the impact on the real problems has not been demonstrated. Hence, it is adding no value algorithmically, and adding a very small value from application perspective. It is basically saying LML can be trivially applied to differentiate through CEM, and it works on some simple toy problems. To me this is mostly a sanity check. Hence, I am sticking to my weak-reject decision.\n-------\nThe manuscript is proposing a method to make cross-entropy method (CEM) differentiable. CEM is a widely used zeroth-order optimization method. The main idea in the paper is applying the recently proposed limited multi-label projection (LML) layer in a straight-forward manner to the CEM since the major computational tool in CEM iteration is top-k selection. The authors apply the proposed method to synthetic energy-based learning and continuous control problems. \n\nThe proposed method is definitely impactful. Considering the fact that CEM is a powerful and widely used tool, I believe the work will lead to many interesting follow-ups. In addition to these, the work is addressing computational scalability of model-based RL which is both under-explored and important problem. \n\nThe proposed model is novel from a modelling perspective since it makes CEM part of end-to-end learnable models. Whereas, it has no algorithmic novelty since it is a straightforward application of the LML layer to the CEM problem. Lack of algorithmic novelty is not an issue but the authors should at least discuss similarities to LML (Amos 2019) in a clear manner in related work. Not including it in the related work is somewhat surprising to me.\n\nThe exposition can clearly be improved. First of all, Proposition 1 is an existing result, hence authors should give a proper citation in its definition. Second of all, Proposition 3 does not include anything about asymptotic (tau -> 0) whereas the stated one-line proof is using asymptotic arguments. Finally, there are other minor issues like Lemma1 not having a proof, proposition 1 has no statement about its proof etc.  The manuscript would significantly benefit from a thorough proof reading for mathematical completeness and correctness.\n\nOne major issue with the manuscript is the experimental study. 1) The only additional algorithmic element introduced by the manuscript is the tau and it is not experimented. Is it crucial to use the temperature parameter? If yes, what is the effect of it? Manuscript needs a collection of ablation studies discussing the tau. 2) The main claim of the paper is \"...make solving the control optimization process significantly less computationally and memory expensive.\" This might be true but not really experimented. Authors do not report any quantitative computation time and/or memory requirement study. I believe the latent DCEM is more memory and computation efficient but quantifying this is important.\n\nI am curious on the choice of CEM. There are other methods which can be utilized since this is basically a bi-level optimization problem. One can use implicit gradients or similar methods (like: https://arxiv.org/abs/1602.02355, https://arxiv.org/abs/1809.01465, https://arxiv.org/abs/1909.04630, http://proceedings.mlr.press/v22/domke12/domke12.pdf).  Can these methods also be utilized instead of back-propagation through optimization procedure? If yes, you should compare with them or explain why you did not. If no, you should explain why.\n\nIn summary, the paper is very impactful. On the other hand, the proposed empirical study significantly lacks in many aspects. I would be happy to increase my score if authors can address these issues.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a differentiable version of CEM, allowing CEM to be used as an operator within end-to-end training settings. The reviewers all like the idea -- it is simple and should be of interest to the community. Unfortunately, the reviewers also are in consensus that the experiments are not sufficiently convincing. We encourage the authors to expand the empirical analysis, based on the reviewer's specific comments, and resubmit the paper to a future venue.
Decision: Reject