Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a differentiable variant of the Cross-Entropy method and shows its use for a continuous control task.   
- It introduces 4 hyper-parameters and it is not clear how robust the method is to these. 
- Although the idea is interesting, I think the paper needs a more rigorous experimental comparison with previous work and other methods.
Detailed review below:
- The abstract should mention clearly that the proposed method allows you to differentiate through argmin operation and can be used for end to end learning. Similarly, please reframe parts of the introduction to make it more accessible to a general reader. For example, in the introduction,  "approximation adds significant definition and structure to an otherwise...". This statement requires more context to make it useful. Similarly, "smooth top-k operation" is not clear. 
- Is there a way to guarantee that the solution found by (D)CEM is a reasonable approximation to the argmin. For unrolled gradient descent, this can be done by looking at the gradient wrt x. 
- It might be more useful to explain CEM before the related work section or just moving the related work to the end. 
- Section 3: If the paper is about CEM, please give some motivation and details rather than just citing De Boer, 2005. 
- There is a notation clash between \pi for the sort and policy later in the paper. Similarly, "t" is for both for the iterations of CEM and the time-stamp in the control problem. 
- I don't understand how Proposition 1 adds to the paper. This is a standard thing. Similarly for Proposition 3. 
- Isn't there an easier way to make the top-k operation soft - by sampling without replacement proportional to the probabilities? Please justify this design decision. Similarly, how is the temperature \tau chosen in practice?
- Please explain the paragraph: "Equation 4 is a convex optimization layer and... GPU-amenable.." Isn't this critical to the overall scalability of this method?
- - How are the hyper-parameters for CEM  chosen - the function g(.), the value of k, \tau, T chosen in practice. If the criticism of GD is that it overfits to the hyper-parameters - learning rate and the number of steps, why isn't this a problem with (D)CEM.  
- Section 4: Since you're comparing against unrolled GD, please formally state what the method is. 
- Section 4.2: How is the structure of Z decided, that is how do you fix the space for searching for the policy in the Z space? 
- There are other methods that auto-encode the policy u_1:H to search the space. How does the proposed method compare to these methods? This is important to disentangle the effect of GD vs CEM and that of just searching in a more tractable space of policies. 
- Section 5.1: How is the number of optimizer steps (=10) decided? Also, how is the learning rate for GD picked. Is the performance of unrolled GD worse for all values of \eta, even after a grid-search over the learning rates?
- For Section 5.2, please compare to baselines mentioned in the paper. Also, there needs to be an ablation/robustness study for the DCEM method. 





review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: After reading authors' response, I am sticking to my original decision. Authors addressed most of the issues I raised and I am happy with their response; however, I still believe the paper should not be accepted since it is not adding enough value. The problem is important and impactful. However, the algorithmic idea comes from LML (Amos 2019), and the impact on the real problems has not been demonstrated. Hence, it is adding no value algorithmically, and adding a very small value from application perspective. It is basically saying LML can be trivially applied to differentiate through CEM, and it works on some simple toy problems. To me this is mostly a sanity check. Hence, I am sticking to my weak-reject decision.
-------
The manuscript is proposing a method to make cross-entropy method (CEM) differentiable. CEM is a widely used zeroth-order optimization method. The main idea in the paper is applying the recently proposed limited multi-label projection (LML) layer in a straight-forward manner to the CEM since the major computational tool in CEM iteration is top-k selection. The authors apply the proposed method to synthetic energy-based learning and continuous control problems. 

The proposed method is definitely impactful. Considering the fact that CEM is a powerful and widely used tool, I believe the work will lead to many interesting follow-ups. In addition to these, the work is addressing computational scalability of model-based RL which is both under-explored and important problem. 

The proposed model is novel from a modelling perspective since it makes CEM part of end-to-end learnable models. Whereas, it has no algorithmic novelty since it is a straightforward application of the LML layer to the CEM problem. Lack of algorithmic novelty is not an issue but the authors should at least discuss similarities to LML (Amos 2019) in a clear manner in related work. Not including it in the related work is somewhat surprising to me.

The exposition can clearly be improved. First of all, Proposition 1 is an existing result, hence authors should give a proper citation in its definition. Second of all, Proposition 3 does not include anything about asymptotic (tau -> 0) whereas the stated one-line proof is using asymptotic arguments. Finally, there are other minor issues like Lemma1 not having a proof, proposition 1 has no statement about its proof etc.  The manuscript would significantly benefit from a thorough proof reading for mathematical completeness and correctness.

One major issue with the manuscript is the experimental study. 1) The only additional algorithmic element introduced by the manuscript is the tau and it is not experimented. Is it crucial to use the temperature parameter? If yes, what is the effect of it? Manuscript needs a collection of ablation studies discussing the tau. 2) The main claim of the paper is "...make solving the control optimization process significantly less computationally and memory expensive." This might be true but not really experimented. Authors do not report any quantitative computation time and/or memory requirement study. I believe the latent DCEM is more memory and computation efficient but quantifying this is important.

I am curious on the choice of CEM. There are other methods which can be utilized since this is basically a bi-level optimization problem. One can use implicit gradients or similar methods (like: https://arxiv.org/abs/1602.02355, https://arxiv.org/abs/1809.01465, https://arxiv.org/abs/1909.04630, http://proceedings.mlr.press/v22/domke12/domke12.pdf).  Can these methods also be utilized instead of back-propagation through optimization procedure? If yes, you should compare with them or explain why you did not. If no, you should explain why.

In summary, the paper is very impactful. On the other hand, the proposed empirical study significantly lacks in many aspects. I would be happy to increase my score if authors can address these issues.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: *Summary*
Various optimization methods can be wrapped to form black-box differentiable deep learning modules. This allows end-to-end learning of energy functions that can be used, for example, in continuous control. There is a whole cottage industry of designing these modules. Advancements in this field are of broad interest to the ICLR community. This paper proposes to unroll the cross entropy method, which is very different than the standard practice of unrolling gradient descent. Experiments on continuous control benchmarks demonstrate that this can be used to learn a latent space in which test-time optimization is performed. By doing this optimization in latent space, it can be performed much faster than in the raw high-dimensional space.

*Overall Assessment*
The paper is well written and the technical contribution is explained well. Both evolutionary search methods (e.g. CEM) and unrolled gradient-based optimizers are very popular in ML these days. This paper will be of interest to many readers, since it works at the interface between these.

I do not have much background in continuous control, model-based RL, etc. Therefore, it is hard for me to assess whether the experiments compare to the right baselines, etc. It appears to me that the experiments on cheetah and walker do not compare a particularly broad set of methods. They only compare within the design space of DCEM. Furthermore, the key result in these experiments is that the DCEM policy results in more efficient inner optimization (such that running the policy is faster). The overall messaging of the paper was not about reducing the costs of executing the policy but in improving performance. Such a result is not provided in these experiments.

I have worked extensively with unrolled optimizers and can speak to the correctness and usefulness of the paper's methodological contribution.and the experiments in sec 5.1. However, these are more for providing insight into the method, and are not large-scale experiments.

My evaluation is a weak reject, since the paper would be greatly improved by stronger empirical results for the large-scale continuous control benchmarks with comparison to a broader set of methods.

*Comments*

The empirical advantage of DCEM vs. unrolled GD is clear, but it's not clear to me what the intuition behind this is. You write "one potential advantage of DCEM is that the output is more likely to be near a local minimum of the energy surface so that, e.g., more test-time iterations can be used to refine the solution." Why would GD not want the output to be be near a local minimum. Also, why is DCEM not also sensitive to the number of steps? The variance of the CEM distribution gives a natural lengthscale similar to the step size in GD. You discuss this further at the end of sec 5.1 Are there simple experiments you could do that compare the robustness of GD (such as with random restarts or unrolled Langevin dynamics) vs. DCEM?

In the standard CEM, doing weighted MLE with 0-1 weights coming from top-k is useful because the set of examples for MLE is size k, which yields computational savings. However, if you can tolerate doing weighted MLE on all available samples, then there may be better ways to set the weights than using a softened version of top-k. See, for example, the example weights in 'Design by Adaptive Sampling' (arxiv.org/abs/1810.03714). Can you comment on the suitability of other weighting schemes besides top-k? Also, will your relaxed top-k perform sensibly when there are many ties in the observed f(x) values?

The principal critique of the paper is the positive DCEM results on cheetah + walker are mostly about runtime, instead of performance. Can you speak more to why you don't think it is providing better performance as well? Perhaps the latent space is useful, for example, for transfer learning + adaptation? 

review_title: Official Blind Review #3

Meta-review: This paper proposes a differentiable version of CEM, allowing CEM to be used as an operator within end-to-end training settings. The reviewers all like the idea -- it is simple and should be of interest to the community. Unfortunately, the reviewers also are in consensus that the experiments are not sufficiently convincing. We encourage the authors to expand the empirical analysis, based on the reviewer's specific comments, and resubmit the paper to a future venue.
Decision: Reject