Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to find the inductive biases and properties for a good optimizer and hence improves upon the existing learnt optimizers.

strengths_and_weaknesses: Strenths
– An analysis of stability of the learnt optimizers.
– Designs new optimizer based on the properties that effect the stability.

Weakness
– Based on the definition of stability, convergence is not guaranteed. As the limit T->\inf L(\theta;T) can be finite even when the loss is oscillating.
– The results do improve on the existing optimizer, the results are shown on very limited setting.
– The results fail on high resolution imagenet and on bigger architectures.
– Due to the above reason, this is not useful in the practical settings.

questions: Can the authors present a reason why the optimizers fail to stabilize when the input is of high resolution as shown in Appendix
limitations: None
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the stability of the learning process from the lens of dynamical systems and characterizes the stability of the optimization process in terms of the eigenvalues of the training dynamics. They use the resulting insights to propose modifications to the learned optimizer’s architecture and the meta-learning procedure to improve its stability and generalization properties. They demonstrate the effectiveness of their modifications by testing the learnt optimizer on a variety of tasks. Importantly, they show that the resulting learned optimizer generalizes much better to tasks very different from the tasks it was meta-trained on.
strengths_and_weaknesses: **Strengths**
1. I liked the first principles approach of analyzing the stability of the existing learned optimizers presented by the paper. Moreover, the paper used the analysis to propose well motivated improvements to the architecture which resulted in significant performance gains. 
2. The paper demonstrates impressive results on various tasks. The improvements on tasks that the optimizer wasn’t trained on were particularly impressive!
3. The paper was also clearly written and easy to follow.

**Weaknesses**
1. Ablation/analysis of the stability of the optimizer and validate if it was indeed the stability causing the issues.
2. The experiments show that the optimizer does reasonably well when tested on tasks very different from the tasks it was trained on. However, I would also like to see how well the optimizer performs when trained on a very wide range of tasks. Does it perform better than the standard optimizers (like adam) across all possible tasks? How much better/worse does it perform compared to an optimizer trained for the specific task? etc.


questions: I think both the points in the Weaknesses sections are fixable and would appreciate the authors making amends along those directions.

**Minor Comments**
1. In line 299 of the paper, the authors say “STAR’s performance is comparable (and occasionally substantially better, 300 as in Figure 4a) to heavily hyperparameter-tuned Adam-based model”. From looking at the plots though, this sounds a bit disingenuous. STAR seems to be doing better on the CIFAR plots but worse on the other datasets (which is reasonable btw, I don’t expect it to work better, but I think it should be made clear in the text). So I would prefer if the paper talked about the differences in performance between tasks and the plausible causes.
2. Typos - line 217 “the a” -> “a”
limitations: The paper discusses certain limitations of the current approach w.r.t the assumptions regarding the convexity of the optimization problem and the need for baking in more inductive biases in order to generalize across tasks. However, I would like to see a deeper discussion on why the proposed learned optimizer still fails to outperform standard optimizers like Adam/NAdamW on a variety of tasks.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method to improve learned optimizer generalization by heavily regularizing both training and the architecture of the learned optimizer. These regularizations are derived from a theoretical analysis of the noisy quadratic problem from the perspective of dynamical systems. Using these insights, the paper presents experiments demonstrating the ability to generalize to much larger and very different problems, as well as much longer optimization depths.

strengths_and_weaknesses: Strengths
- The theoretical analysis provides good insight that leads to regularization techniques that will be useful for the area of learned optimization
- Experiments show reasonable improvements over a tuned Adam optimizer.

Weaknesses
- Evaluations are all on models much smaller than used in practice. The largest model used appears to be a “Resnet Like” ConvNet on CIFAR and a small transformer.
- The actual, exact regularizations that are constructed based on the theoretical analysis are far too obfuscated, given that this is the primary contribution of the paper. Tracking down a good summary of the actual method made takes quite a while and ends in the appendix, for example, section B.1 should probably be in the main text, since this is very important for readers who are considering applying this method.

questions: - What is the computational/memory overhead of the learned optimizer used? How does the training curve compare when plotted against training time instead of iteration/epoch?
- Is the blackbox magnitude term also an output head of the MLP, so the 3 heads are m_g, m_b, d?

limitations: - Evaluations are relatively limited in scale, with small architectures and datasets.
- The efficacy of this method in practice as an actual optimizer is questionable, though it seems positioned more as a path to eventual efficacy.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Learned optimizers suffer from being unable to extrapolate for a greater number of steps than they were trained on, and further the learning of these optimizers themselves are unstable. The authors propose that these learned optimizers can be studied under the lens of linear dynamical system, and by constraining the eigenvalues of the update map $A$ via introducing a nominal term, weight decay and preconditioning, the training of such learned optimizers become much more stable and are better able to generalize across greater time steps and even other domains.
strengths_and_weaknesses: Strengths
- The perspective of viewing learned optimizers through the lens of linear dynamical systems is refreshing and enlightening, and could potentially explain why they diverge, and the authors provide theoretical substantiation as well
- Strong empirical results in both generalizing across time (in terms of additional time steps optimized) and across different domains

Weaknesses
- The flow of the paper is rough on the edges and could use a bit more intuition in some areas, particularly on the section on applying the preconditioning to the output vs to the input gradient
- Minor typos in the paper (e.g. "Incorporating *a* nominal terms" in pg. 3 Fig 1a, "to the *a* hyperparameter" in line 217, "inner *trainin*" in pg 8)


questions: I was curious if learned optimizers did indeed diverge (or are more likely to diverge) when $\rho (A) > 1$. Do we find that this is true empirically for existing learned optimizers? 

Generalization of the learned optimizer is interesting and I would expect the preconditioner matrix P and nominal term $\alpha$ provided by the learned optimizer to be domain dependent, and would be expecting more of the divergence in Fig. 4c, STAR (WD=0.1). Do you have an explanation why your method generalizes rather than diverge?

The Hyperparam experimental setting doesn't seem to be properly described in the paper, and it would be good if you could provide some details on that.

**Post Rebuttal** The authors clarified in detail the questions above, and I have changed my rating to reflect that.
limitations: N/A
review_title: null

Meta-review: metareview: All 4 knowledgeable reviewers recommended acceptance of the paper (2x accept, 1x weak accept, 1x borderline accept), appreciating the the importance of the studied problem, the first principles approach and the obtained theoretical and empirical results. I mainly agree and recommend acceptance of the paper. Still, I ask the authors to carefully consider the reviewers' comments when preparing the final version of the paper and in particular improve the presentation in line with the suggestions. Also, some of the raised points on limitations should be included in a revised discussion.
confidence: Certain
Decision: Accept