Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose to use a combination of Andrychowicz et al. LSTM based approach and Metz et.al feed forward network to learn an optimizer that is useful accross any task. The authors propose to use a few thousand tasks as developed in Metz et. al. for the purpose of training the optimizer. They focus on evolutionary strategies (ES) in lieu of unrolled optimization to keep the problem tractable. 

The task is a challenging one and the authors propose a hierarchical optimizer similar to Wichrowska et. al. I wish the authors invested more thought into the architecture of the optimizer beyond assimilating features of previous work. 

My main concern with the paper is lack of generalization capability of the proposed approach. The authors could have done a more thorough job evaluating the architecture by performing ablation studies, i.e., different permutation of inputs to the LSTM and feed-forward modules and understanding its effect. Fo e.g., how useful is the second moment or parameter values to FF; tensor shape/losses to the LSTM. I am also interested in understanding the trade-off involved in using ES over unrolled optimization. There should be some experiment discussing this trade-off as it seems to be a critical improvement over prior approaches. The sub optimal performance on imagenet is concerning.The authors should also provide guidance on task selection for optimizer training. This guidance could be coupled to the sub-par performance on imagenet. 


review_title: Lot of promising experiments for learned optimizers but little insights

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary

This paper attempts to address the fundamental barriers of learned optimization. The authors identify three barriers: computational requirements, number of training tasks and lack of inductive bias. A “large-scale” evaluation and comparison of learned optimizers is then carried out using many (1024) multi-core CPUs. A simple modification of an existing learned optimizer is also proposed that involves adding more input features. Unfortunately the results don’t seem to reveal any new insights.  

Strengths
- The paper proposes a new, simple hierarchical learned optimizer that outperforms existing learned optimizers. The proposed model is very simple in theory but the implementation seems to still require quite a bit of “hand-engineering” in terms of selecting features etc.


- The experimental investigation reveals some interesting (albeit unsurprising) insights of large scale training of learned optimizers. These include things like training with more tasks improves performance, that learned optimization performs well in the hyper-parameter regime in which it was trained, that it learns some form of regularization and that it outperforms Adam when a non-optimal learning rate is used.


Concerns:
- My main concern with the paper is that some claims are over-blown. Although it is not clear at all that the current generation of learned optimizers can outperform hand-crafted optimizers, the paper makes misleading claims that can easily be taken out of context. Statements like, “We see this final accomplishment as being analogous to the first time a compiler is complete enough that it can be used to compile itself.” and “we believe learned algorithms will transform how we train models" are too strong given the current evidence of the performance of the learned optimizers. I would suggest the authors tone down these claims.


- The proposed hierarchical learned optimizer (as well as existing ones) seem to be more fragile than hand-crafted approaches such as Adam. For example, on CIFAR-10 in Figure 5 the learned optimizer fails even for batch sizes in the training regime. Is there any reason why this might be the case, especially considering that it has access to all the same information as Adam and Adam’s “hand-crafted” operations are quite simple? 


- The disadvantages of the proposed learned optimizer still seems to outweigh the benefits. For example, the “careful tuning of learning rate schedules and momentum timescales” is traded instead for the selection of design of a sufficient range of tasks on which to train the optimizer. This seems to be a far more difficult task than just tuning a few hyperparameters. In addition, although hand-crafted optimizers “do not leverage alternative sources of information beyond the gradient”, the learned optimizers do not do much better and just seem to learn very simple regularization strategies. Currently the discussion on the advantages and disadvantages is completely separate. I think these need to be contrasted and compared on the grounds of what properties a user would prefer in an optimizer.


- The contribution of the paper in terms of new insight or knowledge is not clear. The “large-scale” training on a wide range of tasks and many unrolled steps is interesting but I’m not sure what new insights can be inferred from this? Furthermore, the hierarchical optimizer seems to be a small improvement of the mode proposed in (Wichrowska et al., 2017) with some additional input information (like validation loss).

review_title: Interesting large-scale analysis of learned optimizers, but unfortunately no barriers are overcome

final_score: rating: 7: Good paper, accept
review_content: The goal of a learned optimizer is to replace a human-designed optimizer with a parametric optimizer. However, prior learned optimizers were ineffective at generalizing to a diverse set of tasks. This paper investigates how to learn a useful optimizer by increasing computational scale, building a large, diverse training dataset, and designing the learned optimizer's architecture.

Strengths: 
+ This paper thoroughly examines the challenges of how to train a learned optimizer.

Weaknesses:
+ Training the learned optimizer is fairly complex and computationally expensive, which will prevent broader adoption and makes the paper difficult to reproduce.

Questions:
1) What conditions are necessary for a learned optimizer to replace a standard optimizer like Adam?
2) Could a learned optimizer work with noisy, quantized gradients? e.g. deploying a learned optimizer in a federated learning environment

review_title: Official Blind Review #4

Meta-review: The paper aims to address several challenges in learning neural network-based optimization algorithms by increasing the #unrolled steps, increasing the #training tasks, and exploring new parameterizations for the learning optimizer. The authors demonstrated the effectiveness of applying persisted Evolution Stratergies and backdrop through over 10,000 inner-loop steps can improve the performance of the learned optimizer. Empirical experiments showcased incorporating LSTM to the previous state-of-the-art improve their training performance. 

There are a lot of interesting ideas in the paper. However, packaging them together and only glance over each idea briefly unfortunately dilutes the contribution and the novelty of the work. There are still some major concerns echoed among the reviewer:

1) The proposed hierarchical optimizer seems interesting. It is one of the major contributions of the paper. But, its architecture was only briefly mentioned in Sec 3.3. Its motivation, implementation and the corresponding engineering choices remain unclear by just reading the main text. Some of the details were discussed in the appendix but it would be of great interest if authors could give some intuition on which subset of the tasks the proposed architecture gives the most improvement / failure among the 6000 tasks.

2) Training the optimizer on a diverse set of tasks is crucial for the learned optimizer to generalize. One of the paper's contributions is to further expand the task dataset from the prior work Metz et al., (2020). The authors have conducted very thorough experiments on this new dataset, which is amazing. I would argue there are even enough results for another standalone paper. However, there is surprisingly little detail on how the newly proposed dataset differs from the prior TaskSet dataset. What are the new optimization problems? How are they different from the family of tasks in TaskSet? A TSNE plot of the tasks similar to Figure 1 from Metz et al. (2020) could provide more intuition for the reader and highlight the contribution. 

Overall, if the authors could provide more insight into their experiments and the proposed methods, it would help the readers greatly to see the novelty and the contribution of the paper. The current version of the paper will need additional development and non-trivial modifications to be broadly appreciated by the community. 


Decision: Reject