Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Note: I was asked to write a last-minute review for this paper since the overall rating of the other reviews are not consistent. Therefore, the review is rather brief and I will comment also on concerns raised by the other reviewers.

The paper introduces a new MAML algorithm based on evolutionary strategies (ES) for reinforcement learning tasks. Compared to prior MAML algorithms requiring an estimation of the Hessian, ES-MAML demonstrated to be more stable and efficient. Overall, the paper is well motivated, well written and uses a sound mathematical formulation of the solution approach. Furthermore, the results are convincing and show quite some promise.

Concerning the remarks from Reviewer #3, I believe that it is totally fair to use here a simple ES algorithm that still shows reasonable performance. Of course, we would expect that other ES algorithms might perform better, but this is clearly not the point of the paper. Furthermore, also other papers [1,2] showed that very simple ES algorithm can perform very well on weight optimization of policies. 
(Remark: since there is no page limit for refs, I would recommend to cite [1,2] in the paper)

I share some concerns from Reviewer #4 regarding the hyperparameters. By now, it is well known that hyperparameter tuning can improve the performance of RL algorithm quite a bit and is sometimes even the main factor for superior performance. The authors wrote in their reply to Reviewer #4: “In fact, we did not perform much tuning,”. I would like to reply: In fact, this is not a very useful answer.  If there was hyperparameter tuning involved, the amount has to be quantified (in the appendix) and the same amount should be applied to all approaches being compared in the paper.

Furthermore, I missed a discussion about the limitations of the approach. For example, I would expect that the approach will fail if the networks get too large (and thus  the parameter space is too large (>1Mio Parameters?)) and the task is fairly complicated such that the parameter space is not too redundant. I think there is a reason why people tried to use ES for optimizing DNNs for decades, but failed, and now nearly everyone uses GD variants. So, the authors should be more explicit about potential failure cases and limitations.

Small remark: I haven’t found a description of the architectures used in Section 4.4. Since the paper should be self-contained, I would recommend to briefly make this explicit in the appendix.

[1] Patryk Chrabaszcz, Ilya Loshchilov, Frank Hutter: Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari. IJCAI 2018: 1419-1426
[2] Lior Fuks, Noor Awad, Frank Hutter, Marius Lindauer:
An Evolution Strategy with Progressive Episode Lengths for Playing Games. IJCAI 2019: 1234-1240
review_title: Official Blind Review #5

final_score: rating: 8: Accept
review_content: The authors propose a new method for model agnostic meta learning (MAML) based on evolution strategies (ES) rather than policy gradients (PG). The proposed method has clear advantages over prior work: it is conceptually much simpler, simpler to implement and is a zero-order method (while PG-MAML requires 2nd order derivatives and differentiation through the update steps).  Also, the method natively allows to incorporate methods from evolution strategies, e.g., to improve exploration. Empirical results are convincing: ES-MAML consistently outperforms PG-MAML (or is at least not worse) on various tasks. Also, ES-MAML seems to be much more robust compared to PG-MAML, which is known to be brittle. The paper is well motivated and well written. The mathematical formalism is precise.


Comment/questions:

- PG-MAML is known to be very sensitive w.r.t. hyperparameters, is this also the case for ES-MAML? How were good hyperparameters found for ES-MAML?
- While this work focuses on RL, it would be interesint to see if ES-MAML is also advantages over vanilla MAML for common few-shot learning image classification problems.
- What’s the efficiency of ES-MAML compared to PG-MAML in terms of wall-clock time?
- (minor:) multiple times in the paper, \citep{} and \citept{} are used incorrectly.


review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method, ES-MAML, for optimizing the Model Agnostic Meta Learning (MAML) objective by using Evolution Strategies (ES) gradients instead of policy gradients (PG) as in the previous approaches in the literature. As a result, the use of ES avoids the need of second-order derivative estimation resulted from PG in computing the gradients of the MAML objective; second-order derivatives in MAML are known to be tricky for proper estimation. They also explore ES-MAML with different advanced adaptation operators to improve the ES gradient estimator. They perform empirical study to demonstrate the benefits of ES-MAML as compared with PG-MAML. In particular, they evaluate the comparable algorithms (ES-MAML and variants vs PG-MAML) in terms of exploratory behaviors in sparse-reward environments, adaptation ability, the stability of deterministic policies in unstable environments, and low-K benchmarks. The experimental results are rigorous and promising. They also discuss several potential extensions to ES-MAML in the appendix. 

Regarding the theoretical and algorithmic contributions, this paper combines existing techniques from ES and gradient estimators to make ES gradients work for MAML. Thus, I feel that the paper does not provide significantly new results on these dimensions. For example, the substitution of policy gradient for evolution strategies in Eq. (6) is straightforward, and there is no theoretical justification for the choice of algorithmic designs made in the paper.  However, given that the paper attempts to address an important problem (stably optimizing the MAML objective) with interesting perspective (using ES), that the proposed methods are well developed and extended, and that rigorous experiments to evaluate the proposed methods are provided, this paper could be an interesting contribution to the conference where it can encourage different perspective beyond the gradient policy view for MAML problems. 

Questions and comments. 

1. On page 3, with reference to the text “These issues: the difficulty of estimating the Hessian term (3), the typically high variance of ∇θJ(θ) for policy gradient algorithms in general, and the unsuitability of stochastic policies in some domains, lead us to the proposed method ES-MAML in Section 3.” I agree that the use of ES gradients avoids the need of second-order derivative estimation; however I am not very sure if we could say that ES-MAML here can address the high variance issue of PG given that ES can also suffer from high variance and that there is a rich literature in reducing variance of PG. 

2. Could you clarify which version of PG-MAML was used as the baseline in your experiments? Is this the “vanilla” version from Eq. (2) without any variance reduction techniques (e.g., Rothfuss et al. (2019), Liu et al. (2019)) or did you include one of the variance reduction techniques to the baseline PG-MAML? 

3. In section 4.2, with reference to the text “one of the main benefits of ES is due to its ability to train compact linear policies, which can outperform hidden-layer policies”, could you clarify what did this text mean? Did you mean that compact linear policies are better than hidden-layer policies for MAML, or does it mean that ES is not good at training hidden-layer policies, so it can train linear policies better than hidden-layer policies? 

Minor comments.  
1. Page 2, R has not been introduced. 
2. Page 3, Section 3.1: does F mean f?
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The paper proposes ES for the task of Model agnostic meta learning. Instead of the gradient-approximation which requires computing a hessian matrix, MC samples from a search distribution are used to estimate a search direction. The approach is validated on a number of experiments.

Unfortunatly, I am unable to accept this paper for a number of reasons. Mainly that the ES used is inferior and the constant step-size used can have a major effect on the experimental outcome. 

Almost all proper ES literature with real working ES algorithms are missing and ESGrad is more than 20 years behind SOTA in the field. Since ES are central to the paper, an algorithm that would not even be considered a baseline at any conference in that field is difficult to accept.
The reason for this is that nowadays all ES use dynamic sample-variances based on progress measures, e.g. Cumulative step-size adaptation and Two-Point-Adaptation as the SOTA. Without this, it can be very difficult to find reasonable solutions.
	
Most important missing references from the ES-field in this context:
	
1. and most importantly The original ES-based RL paper:
Heidrich-Meisner, Verena, and Christian Igel. "Neuroevolution strategies for episodic reinforcement learning." Journal of Algorithms 64.4 (2009): 152-168.
	
2. CMA-ES and NES
Hansen, N., Müller, S. D., & Koumoutsakos, P. (2003). Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary computation, 11(1), 1-18.
Krause, O., Arbonès, D. R., & Igel, C. (2016). CMA-ES with optimal covariance update and storage complexity. In Advances in Neural Information Processing Systems (pp. 370-378).
Wierstra, D., Schaul, T., Peters, J., & Schmidhuber, J. (2008, June). Natural evolution strategies. In 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence) (pp. 3381-3387). IEEE.

3. Review of SOTA in large-scale ES:
Varelas, K., Auger, A., Brockhoff, D., Hansen, N., ElHara, O. A., Semet, Y., ... & Barbaresco, F. (2018, September). A comparative study of large-scale variants of CMA-ES. In International Conference on Parallel Problem Solving from Nature (pp. 3-15). Springer, Cham.

4. Recent developments for noisy functions (also references other relevant algorithms with noise-handling)
Krause, O. (2019, July). Large-scale noise-resilient evolution-strategies. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 682-690). ACM.

Section 3.2
-Why should in (7) the same sigma be used as in (6)? Sigma, alpha etc should be learnable parameters learned by the outer ES.
-3.3.2: you are writing below (1) that rollouts come from a distribution, i.e. are stochastic. How would you implement a hill-climber in the stochastic setting? e.g. consider the case when the rewards are heavy-tailed.
- using a hill-climber goes completely against the SOTA in ES which showed repeatedly over the last 20 years that hill-climbing is inferior, especially in larger dimension search-spaces (>100).

Experiments:
- I am not an expert of MAML, but i would not consider this as different tasks, just as different environments for the same task. i.e. a circular running strategy should be optimal for all environments. but when considering different tasks, we would consider different policies to be optimal.
- The experiments use the same hyper parameters for all variants. However, i am not sure this is a fair comparison. E.g. HC has way more spread over the search-space than the other two methods for a given sigma, with following sample steps allowing for fixing the "too large" or "too small" spread.
Since the graph of the objective function is flat in a large area of the search space, the additional exploration through stocasticity alone might explain the results of Figure 1. In this case, the result would be pretty artificial, because real ES would adapt their step-size.
- Similar holds for the number of samples used by the outer ES (n, but named differently in th appendix?). The gradient-based approaches might require a lot more initial points with a smaller K , especially on the flat surfaces of the objectives.
- In Figure 3, middle image, why does the green curve appear to have decreasing performance after iteration 200?
- Figure 3/ 4.2 why do the three settings have different values for number of iterations and K? Why does L-DPP only appear in the third task?
-Section 4.3 and Figure 4: why is there no L-PG and HH-ES? the only curve which is is available for both algorithms has the same performance.
review_title: Official Blind Review #3

Meta-review: This paper introduces an evolution strategy for solving the MAML problem. Following up on some other evolutionary methods as alternatives for RL algorithms, this ES-MAML algorithm appears to be quite stable and efficient. The idea makes sense, and the experiments appear strong.

The scores of the reviews showed a lot of variance: 1,6,8. Therefore, I asked a 4th reviewer for a tie-breaking review, and he/she gave another 8. The rejecting reviewer mostly took objection to the fact that learning rates / step sizes were not tuned consistently, which can easily change the relative ranking of different ES algorithms. Here, I agree with the authors' rebuttal: the fact that even a simple ES algorithm performs well is very promising, and further tuning would only strengthen that result. Nevertheless, it would be useful to assess the algorithm's sensitivity w.r.t. its learning rate / step size.

In summary, I agree with the tie breaking review and recommend acceptance as a poster.
Decision: Accept (Poster)