Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work, the author got trained learned optimizers on four different optimization tasks, and analyzed their behavior. They discovered that learned optimizers learn a plethora of intuitive mechanisms: momentum, gradient clipping, schedules, forms of learning rate adaptation. While the coarse behaviors are qualitatively similar across different tasks, the mechanisms are tuned for particular tasks.
main_review: Originality:
To the best of my knowledge, this kind of work which try to figure out the working mechanism of learned optimizer is rare. Very few people try to interpret it but they want to put forward a powerful optimizer. Thus, I think this work is innovative.

Quality:
I have seen that you do experiments on different suitations. This is very good and I believe that your analysis is convincing. 

Clarity:
The description and visualizition part is good.

Significance:
I admit that your work is innovative and give new idea to the following researchers. However, after visualizition of the learned optimizer. You give very limited suggestions about how to benefit from it. Such as how to design a better optimizer? How to select the design between different tasks and so on.
limitations_and_societal_impact: This work give visualizition of the learned optimizer. But they give very limited suggestions about how to benefit from it. Such as how to design a better optimizer? How to select the design between different tasks and so on.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper deals with the analysis of the learned optimizers (LO). In particular, it tries to isolate their properties and distinctions with respect to the existing baseline optimizers.

It is true that a nascent field of learned optimizers, while enjoying good performance, currently lacks in depth analysis of why the results work. This paper doesn't answer all the questions, but it attempts to bridge the gap between a mysterious performance of LO and known well analyzed optimization techniques like momentum, gradient clipping, LR schedules and adaptation.

The paper uses 4 different tasks for the analysis (Linear regression, Rosenbrock, Two moons and MNIST) and uses a simple RNN optimizer for a LO. 

The presented empirical result is analyzed from a theoretical perspective of a general LO-state update function. The theory has a very mild assumptions on the LO for analysis (essentially only that they have to converge to the fixed point). 

The paper is mostly well written, however, some parts are hard to read, misleading or not explained well (see bellow).
main_review: First of all, I think that these kind of analysis paper are very valuable and I encourage the authors to design a toolbox that can be applied to a given LO in order to analyze and understand its performance.

My main concern is the generalization and applicability of the analysis of the paper. From what I understand the paper interlaces and switches between two modes:
- theoretical analysis of a general LO of the form from eqn. (1-2). This was used e.g. to derive an expression for Momentum in section 5.1. 
- empirical analysis of 4 tasks with limited theoretical understanding (e.g. section 5.2, 5.3 and partially 5.4).

The theoretical analysis has a very mild assumptions on the LO form and only requires that the optimizer state converges to a fixed point. The empirical analysis focuses on the RNN LO, but I'm curious what example of a general convergent update rule *are not* covered by this analysis? Is RNN just use as a general example with any other LO being applicable as well? 

It is a little bit confusing how much we can really generalize beyond 4 tasks and which conclusions to make out of this analysis? The authors showed that a given optimizer exhibit certain types of behavior that we know from known optimizers, but _is this surprising and unexpected_? What can we learn from this analysis, beyond just "it is a first step and we hope the results would be find useful down the line"? It would be great if the authors add a comparative table in the main text (or appendix) with a summary of what they learned from each of 4 tasks. Together with a loss decrease curves from fig 1 top these would be a good succinct outcome. 

Most of the other comments below are just clarifying the confusions when I read the paper. While this is not crucial, the paper would benefit greatly from a careful proofreading. 

Comments:
- I would be a little careful with the notation, e.g. l.61 defines $w$ as readout weights. I don't see why they are called like this? What exactly do they represent? They are not defined in Andrychowicz et al nor anywhere else I looked. Are $w$ are being used at all in the paper?
- l.69: state variable is a scalar, later: state var are low-dimensional.
- l.72: "dynamics are straightforward". Arguably so, e.g. Nesterov accelerated gradient.
- a lot of important information is hidden in the Appendix. It is quite hard to read the paper as is. without looking back and forth to the Appendix.
- l84: "fast to train". Not precise enough. Fast with respect to what? Low number of iterations? Fast gradient computation? All of the above?
- l.86: maybe instead of random, create 2-3 group with low/normal/hard Hessian conditioning? I don't think that fig 1(a) would be too packed if one adds few more curves.
- Fig 1: maybe use log-x and show the dynamic till convergence?
- Fig 1: different shares of red/orange are very hard to distinguish. Consider using different color palette.
- which tasks were used to train the meta-optimizer? Same as the training? The paper mention this at l.164, but it is not clear.
- l.178: misleading: learned optimizer is not *approximately linear*, the authors *linearly approximate it* using the Jacobian. These two statements mean very different things.
- Momentum is defined differently at l.135 than at line 623 and uses different lingo (velocity and mom. hyperparameter vs timescale and learning rate).
limitations_and_societal_impact: The authors did not include any section to address societal impact of their work. Since this is mostly an analysis of _existing_ algorithms, I think it is ok.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This submission focuses on analyzing the behaviors of learned optimizers. The goal is to elucidate properties and patterns learned by such learned optimizers, in order to better analyze, understand, and lay groundwork to potentially improve the behavior of learned and/or manually-designed optimizers. A novel reverse engineering approach is taken and identifies learned interpretable and fundamental behavior such as momentum, gradient clipping, learning rate schedules, and learning rate adaptations.

The submission trained learned optimizers on each of four representative tasks: Linear Regression, Rosenbrock, Two Moons, MNIST . These tasks were selected because they are fast to train, cover a range of loss surfaces (convex and non-convex, low- and high-dimensional, deterministic, and stochastic). Additionally, three baseline optimizers (momentum, RMSProp, and Adam) were tuned individually for each task. The learned optimizers outperformed the baseline optimizers on the meta-objective.

The submission uses two primary methods to analyze optimizer behavior and mechanisms: 1) visualize optimizer update function at a particular optimizer state. 2) analyze optimizer dynamics around fixed points.  The study was able to show that learned optimizers implement classical momentum and a soft form of gradient clipping. The paper visualizes and provides a careful explanation for the best learned optimizer for Rosenbrock (others in appendix) for both. The report also analyzes learned behavior for learning rate schedules and learning rate adaptation. The study isolated the behavior of tuning per layer and parameter type in the CNN for the MNIST problem.

The submission includes a detailed Appendix that illustrates the same behaviors for all the other learned optimizers not included in the main paper, along with other materials such as the meta-learning approach.

main_review: The topic of the submission, learned optimizers, is an important and timely topic. Overall, the submission is well-written, takes a systematic approach to analyzing the behaviors of learned optimizers, and leverages visualizations and detailed explanations to provide evidence of the learned behaviors.  The findings are useful in validating that behaviors of learned optimizers align with manually-designed optimizers, and equally important, valuable as a foundation for further analysis and potential advancement of learned optimizers.

The cases for momentum, gradient clipping, and learning rate adaptation were especially clearly laid out and visualized. 

The learning rate schedule discussion (Section 5.3) illustrated that a learned optimizer adjusted the learning rate over iterations. The report claims the learned optimizer is implementing a “schedule” using autonomous (not input driven) dynamics. However, it is unclear from this analysis that the learned optimizer is actually implementing a “schedule” or has only learned to adapt the learning rate. Additional analysis and evidence to support the assertion that a schedule was learned would strengthen this section.

The learning rate adaptation discussion (Section 5.4), explained well the effects of the learning rate adaptation by the learned optimizer. The illustrations breaking down the behaviors for the positive and negative gradients were well explained and very helpful for illustrating adaptation taking place by the optimizer.  However, line 236 refers to a “new mechanism” implemented by the learned optimizer. Given that manually-designed/baseline optimizers also leverage adaptive learning rates, it was unclear why this is being characterized as a “new mechanism.” 

The additional materials and discussion in the Appendix materially added to the findings, and it would have been helpful to have had some of the information from the Appendix (e.g., potential functional benefits of some of the learned behaviors) earlier, in the main text). 

It would also be interesting to understand which of the identified behaviors the learned optimizers that performed suboptimally did not exhibit and whether some connection could be made to these behaviors and the performance of the optimizer.

Nit: 193 “(gray line in Fig. 3b)” Perhaps it was an issue with the rendering, but I could not see a gray line in 3b (?)





limitations_and_societal_impact: No potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors devise tools for investigating the behaviour of learned, gradient-based, optimization algorithms in order to understand better how they achieve their empirically validated high performance. 

To this end, they visualize the update function, i.e., the change in parameters in dependence of the gradient input, which allows to identify mechanisms employed also by hand-designed optimizers, such as gradient clipping, momentum, and learning rate adaptation. The authors demonstrate that this visualization is particularly insightful when analyzing the update rule using the linearized optimizer state dynamics around fixed points. Furthermore, the authors identify that learned optimizers encode learning rate schedules through autonomous state trajectories. 

Using their tools, the authors analyze one existing type of learned optimizer (the RNN-based approach proposed by Andrychowicz et al., "Learning to Learn by Gradient Descent by Gradient Descent") on four relatively simple problems (linear regression, Rosenbrock, two moons, MNIST). They find that (depending on the problem) the learned optimizer can exhibit well-known mechanisms like gradient clipping, momentum, learning rate adaptation, and learning rate schedules. Furthermore, they provide clues that the optimizer can learn parameter-type (bias/weights) and layer-specific learning-rate schedules in neural-network based problems.
main_review: Originality: The authors propose to combine visualization tools and methods from dynamical system analysis to gain useful insights into the behaviour of learned, gradient based optimizers. While neither their analyzation methods nor the analyzed optimization algorithm are novel, the application of those methods to better understand the learned optimizers is new and the way the authors combine said methods is clever and insightful. 

Quality: The proposed analyzation approach uses well-understood methods from dynamical system analysis to visualize the behaviour of learned optimizers. The authors apply these methods in a technically sound manner and on a range of simple problems. As the approach is intuitive and concise, it yields a range of useful insights into the behaviour of learned optimizers and unveils both parallels to known optimization mechanisms as well as novel behaviours. The authors admit that their work does not yet yield a "holistic picture" of learned optimizer behaviour. I agree with that, i.p., because the authors analyze only one particular type of learned optimizer. The submission could be strengthened by including different learned optimizer architectures.

Clarity: The submission is very well written and the visualizations are of high quality. In my opinion, the exposition could be strengthened by considering the following points:

- Sec. 5.1:
    - Maybe try to merge some of the information/equations of App. C into the main text.
    - Provide some more background on why the "learned optimizer uses a single eigenmode to implement momentum" (l. 189).
- Sec. 5.3:
    - Provide some more background on the connection between autonomous trajectories and the implementation of learning rate schedules.
- Sec. 3.2:
    - How do you generate a distribution of problems for MNIST? Do you also sample the initial weights of the CNN?

Minor points (not influencing my score):

- l. 36: three disparate tasks → four disparate tasks
- Eq. (2) does not contain the readout function $r$ referred to in l. 658.

Significance: The authors compile a toolbox of analyzation methods for learned optimizers. While the used methods are not new, the authors apply them in an intuitve and clever way. Similarly, while the experiments do not yet yield a holistic picture (i.p. because the authors apply their tools to only one type learned optimizer), they still provide some interesting and important insights into the behaviour of the RNN-based optimizer. Therefore, I think the submission in its current form is significant, i.p., because future work will benefit from it. I therefore recommend acceptance.

---------------------------------------------------
After reading the other reviews, I am even more convinced that the paper is interesting and valuable and the proposed methodology is sound. In addition, the authors will improve on the clarity of their manuscript. Thus, I raise my score to 7.
limitations_and_societal_impact: No potential negative societal impact.
review_title: null

Meta-review: This paper was very actively discussed amongst the reviewers.

Points in favor were summarized by reviewer QfSG as follows:

1. The analytical tools presented in the paper are a strong foundation for characterizing learned optimizers. Such tools are lacking in the field. In addition to the tools themselves (which are useful for other L2O researchers), the systematic approach taken to the experiments and explanations could also be helpful for others seeking to contribute such tools (for and beyond L2O methods).

2. The methodology, experiments, and conclusions taken from the experiments are well-grounded and solid, especially if one takes into consideration the significant additional depth provided in the appendix.

3. The results show that learned optimizers match or mimic techniques in state-of-the-art optimizers, despite being trained on diverse types of optimization problems - which gives merit to both L2O approaches and to the tools presented. That is, even if it is not surprising, given how learned optimizers are derived, that learned optimizers learn similar techniques as known optimizers -- that finding (learning to similar methods) brings merit to L2O approaches as finding 'trusted' approaches.

4. The authors demonstrates applicability beyond just the 4 tasks, and responded to the concerns.

Reviewer sWNV replied to these 4 points as follows:

1. I agree that these tools are lacking and having a paper like this (and especially a toolbox) would lie a good foundation onto a better understanding of the learned optimizers. This is undeniable contribution and I'm happy that this paper tackles it.

2. The techniques provided in the paper are good, however, there are still some questions remain. For example, the momentum analysis (sec 5.1 and App C) relies on the linear approximation of the recurrent dynamics. It is not clear to me that this approximation is useful and actually holds in practice. It would be useful to actually run the momentum optimizer with the parameters recovered by this analysis and see if it is close to the training dynamics of the LO. Of course, this also would be not accurate, since the LO also do much more than just momentum...

3. I think that it is a bit of an overstatement to say that LO match or mimic known optimizers, they specifically being tested to match or mimic the specific property of a given optimizer. I would love to see an example when this does not happen and the LO solely memorizes the task without exhibiting the properties of the known optimizers. 

4. My main criticism is that the from the title, introduction and the discussion it seems like the analysis holds and applies to many Learned Optimizers, whereas in reality it is only 1 that is being analyzed. This by itself is not a problem and doesn't deny the technical contribution, but it is misleading to the reader (especially the cursory reader that relies of Abstract + Intro for the quick understanding of the paper). I would really encourage the authors to pivot the narrative of the paper and instead focus on the techniques and methods of general LO analysis instead of claims about Learned Optimizers in general. To this extend, I would like to re-iterate that the release of the general and easy-to-use toolbox that can generate an extensive report given a used provided LO and a task (ideally a benchmark of many tasks) should be an essential part of this paper.

After this internal discussion, the reviewers offered to change the narrative and title of the paper to alleviate the concerns in 4. above.

Overall, the paper presents a solid contribution to the field of learned optimizers, and I recommend acceptance. I encourage the authors to act on the suggestions by reviewer sWNV, especially adapting narrative + title.
Decision: Accept (Poster)