Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes Amortized Proximal Optimization method, which parameterizes proximal point method and performs online meta-learning the parameters during training. Specifically, they emphasize the importance of considering both function space discrepancy and weight space discrepancy at the same time. They consider two meta-learning scenarios: 1) learning rate scheduling and 2) Kronecker-factored preconditioning matrix. Meta-learning is done with usual one-step lookahead method, but w.r.t. the same training minibatch. The experimental results demonstrate the efficacy of the proposed method over diverse learning scenarios, in terms of training convergence and generalization performance.
strengths_and_weaknesses: # Strengths
- paper is very clearly written and easy to follow
- Nice introduction motivating why we need both FSD and WSD at the same time in Figure 1, which seems to be the main contribution of this paper.
- Connection between the proposed method and the existing second-order optimization methods is well explained.
- Computationally efficient parameterization for the preconditioning matrix, as well as reasonable one-step lookahead meta-learning procedure
- Experiments are very extensive, and overall achieve competitive performance compared to the existing optimizers without learnable parameters.

# Weaknesses
- The biggest and significant weakness of this paper is that they did not compare with any of the existing learnable optimizers. Although the existing meta-learned optimizers mostly consider offline setting (Li et al, Lee et al, Park et al, Flennerhag et al.), Some recent work consider online setting such as Micaelli et al., and Gao et al. Especially, Micaelli et al. proposed to perform online optimization for learning rate, weight decay parameter, and momentum parameter at the same time by using forward mode differentiation, which seems very relevant to this submission. I strongly encourage the authors to compare their model to Micaelli et al. at least for the learning rate scheduling scenario. Gao et al. also seems very relevant to this work as they parameterize Bregman divergence of mirror descent. I believe there should be much more literatures that I'm not aware of, so I will defer it to the discussions with other reviewers. Nonetheless, I believe that the authors should compare with at least one or two existing learnable optimizers. I will adjust my score (either decrease or increase) based on this.


# References
- Li et al, Meta-SGD: Learning to Learn Quickly for Few-Shot Learning, 2017
- Lee et al, Gradient-based Meta-learning with Learned Layerwise Metric and Subspace, ICML 2018
- Park et al, Meta-Curvature, NeurIPS 2019
- Flennerhag et al, Meta-Learning with Warped Gradient Descent, ICLR 2020
- Micaelli et al, Gradient-based Hyperparameter Optimization Over Long Horizons, NeurIPS 2021
- Gao et al., Meta Mirror Descent: Optimiser Learning for Fast Convergence, ICLR 2022 workshop
questions: - How the hyperparameters $\lambda_{FSD}$ and $\lambda_{WSD}$ have been selected?
- Why in the outer-loop you used the same minibatch to compute the loss, which seems a little bit unusual?
- Typo: L279, "Figure 4 (right)" --> "Figure 4 (middle)"
limitations: They did not particularly mentioned the limitations of this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes Amortized Proximal Optimization (APO) which is an online meta-optimization for optimization parameters. The modern neural network update rule can be interpreted by proximal point update where it find minimization of three different function w.r.t update rule: 1) the current mini-batch loss, 2) function discrepancy, and 3) weight discrepancy. The authors also reveal that this interpretation is related to other update rule such as Gradient Descent, Hessian-Free, and Natural Gradient. Based on this interpretation, they propose a new meta-objective by sampling two different mini-batches when updating meta-parameters. This paper uses the proposed meta-optimization for optimizing learning rate adaptation and preconditioning matrix for the update rule, and the extensive experimental results shows the effectiveness of APO.
strengths_and_weaknesses: - Originality

Many other approaches that meta-learns optimization parameters focus on minimizing training (or validation) loss w.r.t to the parameters. The main originality of this paper is the proposed meta-objective which includes two different discrepancy term, and it is novel. 

---

- Quality

The intuition of this paper is well-motivated throughout the paper, by relating modern neural networks update rule to the proximal point method. Based on the claim on the relatedness, the derivation of the meta-objective seems technically sound to me. The experiments are sufficient for me to understand the effectiveness of the propose method.

---

- Clarity

This paper is well-written (especially the Figure 2 and Algorithm 1 are very nice to understand the paper).

---

- Significance

This paper is really nice to be adapted to online optimization of any continuous hyperparameters. Further, it is easy to be combined  with other hyperparameter optimization techniques, since the main modification is the meta-objective; for example, one can use implicit differentiation [1] to obtain $\nabla Q(\phi)$ instead of using automatic differentiation of 1-step lookahead.

---

- Weakness

I think this paper is really good, but one thing to improve this paper (I am not going to degrade my score if authors would not address it) is that it is worthwhile to compare it with other hyperparameter optimization methods such as Hypergradient Descent [2]. I think the comparison would provide a clear insight to understand the effectiveness of the proposed meta-objective, since the algorithm is quite similar to [2] except that the discrepancy terms.

[1] Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, 464 pages 1540–1552. PMLR, 2020.

[2] Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. arXiv preprint arXiv:1703.04782, 2017.
questions: - This paper use an efficient approximation for optimizing the meta-objective, by considering only 1-step horizon.  If one update the meta-parameter for every $K$ step, it is natural for me to unroll all the previous K steps (if it is possible). For example, either FMD (RMD is possible for preconditioning matrix) [3], or implicit differentiation [2] are feasible to unroll them. Can you elaborate the reason why one should use 1-step approximation?

[3]  Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning, pages 1165–1173. PMLR, 2017.
limitations: I did not find any potential negative societal impact of this paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces an online meta-optimization method called Amortized Proximal Optimization (APO) that can dynamically adjusts parameters like learning rate and preconditioning matrices during the training process.  Before introducing APO, background information is provided to motivate the necessity for both  a function space distance (FSD) term and a weight space distance (WSD) term for the proximal objective. Then, prior 1st and 2nd order optimization methods are interpreted as proximal point methods with various approximations for the loss, FSD, and WSD.  APO is proposed as a meta-optimizer that can be combined with existing optimization strategies to dynamically adjust meta-parameters.  Experiments show APO adaptively learns good learning rate schedules that outperform fixed learning rates and are competitive with manually defined learning rate schedules.  When applied to preconditioning matrices, APO achieves optimization performance competitive with more computationally expensive second order approaches.  
strengths_and_weaknesses: Strengths:
- Well written paper that provides useful insights into connections between proximal optimization and popular optimization methods and the role of FSD and WSD in improving performance.  
- Extensive empirical results across multiple types of problems and architectures.  
- Strong results for APO-Precond that show improved performance for poorly conditioned regression problems and FP16 training.  

Weaknesses:
- Presentation of experimental results can be improved.
- APO underperforms SGDm with manual decay schedule in nearly all tasks in table 4. 
- APO in some sense trades off selection a learning rate and decay schedule with tuning \lambda for the proximal penalty terms.  Perhaps this is not surprising since we shouldn't expect a free lunch and a more powerful optimization should require more tuning.
- The decision to use a separate batch for the FSD term of the meta-objective feels heuristic and is only supported by empirical evidence in Appendix E.  


## Originality 
I am not too familiar with work on developing new optimization methods but I found the presentation and methodology to be interesting and original.

## Quality
The overall quality of the paper is high with adequate detail provided in the supplementary material for all experiments.  I do think the experimental results need to be improved however in the following ways:
- Error bars/std need to be provided for Table 2 and are also highly recommended for all other tables.  For Table 2 in particular, the differences between KFAC and APO-P on architectures with batch-norm layers seem very small and likely insignificant.  
- Table 4 can use bolding/underlines for users to quickly identify the best optimizer in each setting.  I think this will show that manual decay schedule does the best in nearly every setting.  While this result detracts from the benefit of APO, I think it's worth investigating how to close this gap.  
- What is the computational cost for KFAC and APO-P in Section 5.1?

## Clarity
The paper is well written in most places.  I do have a few suggestion/questions:
- VGG-16 is stated to not have batch-norm in ln 286-287 but then in ln 296, only LeNet and AlexNet are considered to not have batch-norm.
- ln 32-34 can be worded better.
- ln 142-145: This part was confusing for me and I could have used a concrete example.

## Significance
I think this work can potentially have high significance if the authors are able to close the gap between manual decay and learning lr with APO and show how APO can be applied to multiple meta-parameters.  It still has impact in the current form

questions: - Is there any intuition for the empirical results in Section E of the Appendix where the same minibatch needs to be used to compute the the gradient of the base optimizer and the loss term of the meta-objective while a different minibatch should be used for the FSD term?
- How should APO be extended to adapt multiple meta-parameters? e.g. both learning rate and preconditioning matrix?  Are there meta-parameters that are not compatible with APO? can momentum be adapted with APO?  What about generalization/regularization parameters like weight-decay?
- It seems like manual learning rate schedules outperfrom APO-lr due to using smaller learning rates towards the end of training.  Why does APO not learn to decay LR even more/faster?  Should the \lambda parameters of APO be adjusted during training as well to promote faster convergence?  
- I suggest including a legend for Figure 4 left.
- See additional suggestions/questions in Quality section of the box above.

__Typos__
- Footnote 2 is missing.
- ln 289, extra "a"
limitations: The authors do not discuss the consistent underperformance of APO relative to using a manual decay schedule for many vision tasks in Table 4.  I think it is worth understanding why there's this gap and see if it's possible to bridge this gap with modifications to APO.  I also think a discussion of potential additional compute cost associated with tuning \lambda for APO would be useful.  Finally, the authors make it sound like APO is applicable to arbitrary meta-parameters but the questions above regarding optimizing multiple meta-parameters simultaneously and regularization related meta-parameters are unanswered.
review_title: null

Meta-review: metareview: This paper proposes a technique for online adaptation of optimization hyper-parameters. The key advance seems to be the simultaneous minimization of function and weight space discrepancy, in addition to the minimization of the minibatch loss.

All reviewers recommended this paper be accepted. One reviewer raised their score by two points, and went on to "moderately champion" the paper during discussion. The paper appears to have additionally meaningfully improved through the course of the review process.

Based upon the reviewer consensus, I also recommend paper acceptance.
confidence: Certain
Decision: Accept