Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper aims to shed the light on the work (mechanisms) of the learned optimizers. The main contributions of this paper are the following ones:
 1. The authors found the connection of learned optimization patterns with classical optimization techniques: such as momentum/gradient clipping, etc.
 2. The authors applied the methods from dynamical systems to the analysis of learned optimizers. In particular, the authors used linearization of dynamical systems in vicinity of fixed points to analyse the properties mentioned above, as well as the trajectory of hidden states of learned optimizers during optimization of the inner problem.

Overall, the findings of this paper sound interesting but the paper requires further development. For example, the following directions can be considered:  theoretical analysis of the obtained results, practical applications of the results or investigation of resulting properties on a larger class of optimizers training methods.

To analyse different properties of the learned optimizers the authors considered a set of three simple tasks using an optimizer training method similar to [1] and tried to find the connection between the learned optimization pattern and classical optimization techniques or identify some new behaviour patterns. While the choice of such simple tasks is well-motivated by the fact that optimizers training is very computationally intensive the choice of training method from [1] leads to significant limitations of obtained results from my point of view. In particular, the main difference between [1] and newer methods is its generalization property. As authors from [1] noted the considered method has problems with generalization (for example, if the activation function of the inner task is changed, the learned optimizer is no longer able to generalize and train NN with the different activation function). Due to this it is difficult to say whether findings from this paper can be useful in the more interesting settings or with more advanced optimizer training methods.

Overall, this paper is well-written, but there are some sections of the paper that are difficult to follow. For example, section 3.2. One of the main techniques used in the paper is linearization of dynamical systems around fixed points. I would recommend adding into the paper a brief description of the method with its limitations and all necessary definitions. In particular, I would provide the rigorous definitions of fixed point/stable fixed point/global fixed point of dynamical systems.

Some other questions and issues:
1. Appendix C.2 has incorrect reference to Figure 5.
2. Figures 10/11/12. Adam optimizer has 3 parameters alpha, beta, gamma. The plots corresponding to Adam have 2 parameters each, what is the value of the third parameter?
3. The authors of [2] noticed that it is beneficial to provide additional input features that are borrowed from classical optimization methods (e.g. momentums) besides gradients.  Could you please clarify how these observations align with the findings that learned optimizers are able to learn classical techniques?
4. How will the method properties change if the hidden size of the learned optimizer will be decreased (e.g. from 256 to 40/20)?

[1] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp. 3981–3989, 2016.

[2] Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. arXiv preprint arXiv:1703.04813, 2017.

review_title: Official Blind Review #1

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
The author proposed a set of tools to analyze the properties of the learned neural network-based optimizers.

This toolset consists of (i) update function visualizer (ii) linearization of the update equation around the fixed point. The author takes an RNN-based optimizer as an example and analyzes it in the empirical section. 

The author investigates several properties including (1) momentum using linearization (2) gradient clipping using the function visualizer (3) learning rate schedule and (4) learning rate adaptation. It shows that the learned optimizer indeed possesses some useful properties. 

--------
Review:
Clarity: In general, some parts of the paper is clearly written and motivated such as the problems of the current learned optimizer.  However, the paper quality can be improved if some sections can be more clearer. For example, in section 3.2, the author only mentioned the high-level idea of the proposed tool (linearization) without a detailed mathematical introduction. It would be much more helpful the author can give a brief introduction to the linearization in the Appendix, rather than describe it in words and postpone it in the momentum analysis section.

Novelty: The tool seems to be novel but the idea of linearization is not new, which has been used in other analyses of the dynamical systems as mentioned by the author. As for the function visualizer, it seems to be a plot of the parameter update, which is standard.

Technical soundness:
I have checked some of the details, it seems to be correct.

Significance:
The proposed tools may be useful for analyzing neural network-based optimizers, and help to diagnose their behaviors. The tools are easy to implement and can be applied to a broad class of optimizers.

Weakness:
Although the paper proposed an interesting way to visualize some of the properties of learned optimizers, I think the paper in the current stage is not enough for a full conference paper. Here are some of my concerns and questions for the author:
1. The proposed tools are mainly used to visualize some of the properties of the learned optimizer. This may be helpful for visualizing some properties but it provides no analysis on why it has those behaviors and what advantages these properties when optimizing a function $f$. Or what properties of the function $f$ can induce such behavior of the learned optimizer.
2. The author only demonstrates the usage of the tools in analyzing a single/isolated property. How these properties are combined is not analyzed. 
3. The author only analyzes one NN-based optimizer. are there any other forms of the leaned optimizers? What advantages of those optimizers compared to the one used in the paper? Can your analysis confirm those advantages and provide possible reasons? This would be stronger evidence to back up the validity of the proposed method compared to the current analysis without comparisons.
4. As mentioned in the clarity, better use mathematical equations to explain certain terms rather than words. For example, in section 3.1, the slope w.r.t what, the gradient $g$?
5. In section 3.2, I am a bit confused about the difference between the fixed point and convergence point? Any examples of scenarios that it is a fixed point but not a convergence point?
6. In section 4.1 (momentum), for the first figure in the bottom row (Figure 3), is it the eigenvalue point in the regression task? What about the first one used in Figure 7 (Appendix B). Are they the same? 
7. In section 4.3, your analysis shows the optimizer has this autonomous behavior. What advantages does this behavior provide? The author also mentions that the parameters should not be updated during the autonomous behavior. How can we confirm this through the plot (Figure 5)?
8. I am not fully sure what you mean in the second paragraph (section 4.4). Do you mean that at the convergence points, you manually give some gradient perturbations, and the fixed points are moved away from the convergence point to get the S-curved shape?
review_title: Interesting investigations on the learned optimizer but not enough for a full conference paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: # Summary

This paper studies learned optimizers and tries to discover what they have learned. The authors argue that the dynamics of the learned optimizers are responsible for those behaviors. Results are mainly presented as visualization and with tools from reverse-engineering dynamical systems. 

# Strength

- This seems to be the first paper on re-discovering what the neural optimizers have learned from dynamical system angle.
- Overall, the paper is easy to follow.

# Weakness

- All three experiments are done on toy datasets. Although the authors argue in Section 4 that "These tasks were selected because they are fast to train (particularly important for meta-optimization) and covered a range of loss surfaces (convex and non-convex, low- and high-dimensional)", I completely do not agree. Even in the original neural optimizer paper by Andrychowicz et al. they considered CIFAR (subset). If we only run the optimizers for several hundreds of steps, I think it's totally affordable. It is not convincing that the observed behaviors will generalize to real-world problems.
- In the title and abstract, the authors mention "novel mechanisms". But it is not clear to me which behaviors shown in Section 4 are considered novel? I guess the authors might refer to learning rate adaption, "the effect is to decrease the learning rate of the optimizer when large gradients are encountered". But in Section 4.4, paragraph 4, the authors say ".... similar to the changes observed as the RMSProp state varies," which might suggest this is not novel to human-designed optimizers with a memory mechanism. Even this is novel, this is only observed on three toy datasets.
review_title: Toy datasets

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Hello authors,

Thank you for your submission.  I very much enjoyed reading it.  I found the writing to be clear and only found one grammatical error (detailed below).  As with any black-box system like a learned optimizer, there is naturally a lot of interest in what, actually, the optimizer itself is learning, and why it has learned in the way it has.  In this effort, the authors perform interesting experiments with intriguing results for the first of those two questions.

I found the experiments to be comprehensive and the figures to be adequately described.  It would be great if the authors could provide more details in the main paper on the precise process used to approximate the nonlinear dynamical system.  It could be easy for a reader to assume, without details, that a learned optimizer is "definitely" learning momentum, while missing the nuance of how those results were obtained.  (For the record I am not saying I disagree with the results, only that if more detail can be provided, it would be helpful.)

The experiments are only performed with one form of learned optimizer, using an RNN with 256 GRUs.  What happens if different learned optimizer architectures are used?  Is there reason to believe that they would exhibit similar behavior?  Have the authors perhaps done any experiments with other architectures to see if the results are replicated?  Adding some extra information about this question to Section 4 could improve the paper.

Overall, I think this is a good paper and should definitely be accepted.

Small bits:

 - Can we really drop the subscript notation in (1)?  Couldn't F use other elements of g?  Or are we restricting the situation for this exploration?
 - You might consider switching Figure 2 and Figure 1; the background on learned optimizers meshes well with Figure 2 and so it may be more streamlined to introduce it there.  However, your call.
 - Page 4: "The RNN is is trained" -> "The RNN is trained"
 - It would be really helpful if Figure 3 was on the same page as Section 4.1.  This also applies to Figures 4 and 5.

## Post-rebuttal comments

Thanks again to the authors for the submission.  My concerns are clarified and so I will leave my rating intact.

review_title: An intriguing exploration of learned optimizers

Meta-review: A line of work since 2016 has investigated learning NN-based optimisers, which produce optimisation updates by processing loss/gradient info with neural networks. This paper tries to understand the learned dynamics of these NN-based optimisers by linear approximation to the learned non-linear dynamics. Visualisation of these approximations are shown on 3 optimisation problems: linear regression, Rosenbrock function, and a toy neural network classification problem, with the hope of covering different types of objective landscapes. 

Reviewers agreed that the paper studies an important research question, which would interest researchers working on meta-learning learning algorithms. However there are several major concerns raised by the reviewers: (1) the example optimisation problems are toyish, and (2) the paper does not explain very well the link between the visualised behaviour and the better optimisation results, i.e. it is unclear to the reviewers why the learned dynamics lead to better optimisation results. 

While I am not too concerned about issue (1), I think issue (2) is a significant one, flagging that the clarity of the paper needs to be improved. Ultimately, the paper is motivated by the question "How is a learned optimizer able to outperform a well tuned baseline?", so a reader would expect some clear explanation towards answering this question. Also some reviewers are concerned about the fact that only the RNN-based optimiser in Andrychowicz et al. (2016) is analysed; since there exists other forms of learned optimisers, focusing on studying only one type of them might lead to early conclusions that are not so accurate.
Decision: Reject