Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to apply reverse mode auto-differentiation to compute gradients w.r.t  hyperparameters of an optimizer and update them using gradient descent, in contrast to computing them manually as done in previous works. It offers the flexibility in terms of choosing the optimizer for updating the hyperparameters. It also proposes to apply this procedure recursively for each hyperparameter of hyperparameters, resulting in a stacked hyperoptimizer. Experimentally, it tests the hyperoptimization idea on the hyperparameters of SGD and Adam, and compare its performance against the case when hyperoptimization is not used. 

strengths_and_weaknesses: - This paper is well-written and can be easily understood. The pseudocode together with the computation graph clearly demonstrate the hyperoptimization procedure. There is a minor typo in section 3.1, should be “4.08% and 4.48% test error for SGD/SGD and SGD/Adam, respectively”. The main findings are also highlighted. 
- The novelty of this paper is limited. As mentioned in the paper, [1] computed the derivative of loss w.r.t. the learning rate of SGD and performed gradient-descent type update. For optimizers with more hyperparameters and more complicated gradients such as Adam, the paper does not demonstrate the clear advantage of optimizing all the hyperparameters over only optimizing the learning rate. For example, in Table 1 (b), when only the learning rate of Adam is optimized, the resulting test error is lower than that when all the hyperparameters are optimized (3.00 % vs 3.03 %). Hence, autodifferentiation may not take full advantage over manual computation when simple expression for updating the learning rate can be derived as in the case of SGD or Adam. 
- Despite lack of novelty, some interesting results are presented. Figure 1a) shows that when hyperoptimization is used, the algorithm is less sensitive to the initialization of hyperparameters, and can make certain degree of corrections when bad initialization occurs. This is an intriguing property as a good initialization is difficult to know a priori and can be dependent on many factors such as optimizer and dataset. Another interesting observation is that the final step size can be either larger or smaller when compared against its initial value as shown in Table 1 (a) and Figure 2 (b). This shows that the hyperoptimization approach can tune the hyperparameter properly depending on its magnitude. 
- One main contribution of this paper is that with autodifferentiation, one can perform gradient -type optimization for the hyperparameter of hyperparameters. The results seem to show that test error decreases as height increases, but it is not very convincing when comparing Figure 4 b) with Figure 4 c). The claim regarding the sensitivity of different height to base-level step size does seem to be true. 
- Some main limitations of the experimental results are: given hyperparameters are properly initialized, the hyperoptimization does not demonstrate clear advantage over the case when they are fixed during training. Another limitation is that even hyperoptimizer can optimize hyperparameters of hyperparameters, but one still need to provide proper initializations for these higher-level hyperparameters. Hence, the search cost of hyperparameter may not be necessarily reduced. Moreover, as mentioned in the paper, it is difficult to provide a common initialization for different heights. This may limit the usage of the stacked hyperoptimizer. 

[1] Online Learning Rate Adaptation With Hypergradient Descent
questions: - To strengthen the paper, it would be good if the author can further test the stacked hyperoptimizer on more optimizers, such as AdaGrad, RMSProp etc, and report similar results as Figure 2 and 4. 
- It would be good to plot also the training loss of stacked and unstacked hyperoptimizers. It is interesting to see whether the hyperoptimizer converges faster than normal SGD or Adam. Also please report the computation time for different heights of the stacked hyperoptimizer. 
- I believe the experiments of this work are interesting and intriguing, and can be beneficial for the community to reduce the search cost of hyperparameter or build better optimizers. This paper is at the border line of acceptance. If all my concerns are addressed, I would raise my score. 
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper outlines a method to computer hypergradients using automatic differentiation by modifying backpropagation, allowing hypergradient descent to be extended to arbitrarily large layers of hypergradient descent without consuming arbitrarily large amounts of programming labor. The paper then uses this technique to validate the hypothesis that taller towers of hyperoptimization yield increasingly robust hyperparameters, eventually converging to the optimal as the number of layers increases even with poorly chosen initial values.
strengths_and_weaknesses: Strengths:
- Paper is extremely well written, and the best that I have reviewed recently. The writing is very easy to understand and follow.
- I believe that the experimental results validating the higher order hyperoptimization hypothesis will be immensely valuable to the optimization and meta-learning community.
- Well-documented, easily readable, and easily usable source code is provided.
- The method presented will be very useful to the community in advancing future meta-learning and optimization research.

Weaknesses:
- The contributions of this paper are primarily in presentation, and not as deep or novel as the typical NeurIPS paper.
- Evaluation networks are rather small compared to networks used in modern practice, and more comparable to those used by learning to optimize papers that are crippled by high memory and compute requirements. Since the overhead of hypergradient descent is much lower, it would be good to see a benchmark of at least a ResNet-152 or some transformer architecture, especially in a higher-order hyperoptimization experiment.

questions: - The computational overhead of hypergradient descent using AD is stated to be negligible; what is the actual value –  is it even statistically measurable?

limitations: Authors address all limitations I can think of.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work presents a way of optimizing parameters (hyperparameters) of first order optimizers as the learning rate and the momentum coefficient using another first order optimizer as SGD or ADAM, called hyperoptimizer. Furthermore, this method can be applied recursively to optimize the parameters of the hyperoptimizer and so on, creating a chain of optimizers. The gradient of the loss w.r.t. the hyperparameters is computed by backpropagating through only one step of the optimizer, so that the weights and biases of the neural network and the hyperparameters can be optimized jointly with a small increase in cost compared to standard training. 

The authors provide a simple pseudocode implementation in pythorch which exploits Automatic Differentiation (AD), thus avoiding error-prone implementation of derivatives by hand. They also validate this method on several neural architecture and datasets: MLPs on MNIST, CNNs on CIFAR-10 and RNNs on Tolstoy. Results show that this approach often outperforms training with the fixed sub-optimal hyperparameters used as a starting point for the algorithm, and is comparable to using the hyperparameters found at the end of training. Furthermore, they show that, when optimizing only the learning rate,  the longer the chain of optimizers the less sensitive the final performance is to the initial learning rate.

strengths_and_weaknesses: Strenghts:
1. The method is simple to implement, clearly explained and probably not too costly compared to standard optimizers. The authors present Pytorch code snippets and discuss the approach in great detail.
2. Convincing experimental evaluation. Experiments are done with several architectures, MLPs CNNs and RNNs, and small to medium sized dataset. The advantages brought by the method are evident and match the authors’ claims.
3. High significance. A Better optimizer for training neural networks is useful to the majority of the machine learning community. The ease of implementation could favor its adoption.

Weaknesses:
1. Comparison with related works can be improved. The authors discuss a connection with programming language theory without providing references. They also do not discuss or compare against other adaptive optimizers, as [1, 2], which are also less sensitive to the initial learning rate and are more theoretically grounded.
2. Lacks large scale experiments (e.g. on ImageNet) and does not include transformers models, which are widely used neural architectures.
3. Limited Novelty. The idea of optimizing the learning rate using a first order method with gradient computed by back propagating through one step of the optimizer was presented in [3]. The novelty of this work lies in (i) applying the idea recursively, (ii) optimizing other hyperparameters like the momentum coefficient, (iii) providing a simple Pytorch implementation exploiting automatic differentiation and (iv) an independent experimental evaluation.

[1] Orabona, Francesco, and Tatiana Tommasi. "Training deep networks without learning rates through coin betting." Advances in Neural Information Processing Systems 30 (2017).

[2] Vaswani, Sharan, et al. "Painless stochastic gradient: Interpolation, line-search, and convergence rates." Advances in neural information processing systems 32 (2019).

[3] Baydin, Atilim Gunes, et al. "Online Learning Rate Adaptation with Hypergradient Descent." International Conference on Learning Representations. 2018.

**Post author's response**
The authors addressed most of my concerns and promised to address others. Therefore I am increasing my score from 6 to 7.



questions: 1. How much higher is the time and memory cost of the method compared to standard SGD/ADAM? how does it increase as the chain of (hyper-)optimizers becomes longer? 
The computational cost is discussed only briefly in the text and all plots have the number of epochs in the x-axis. I think this aspect is worth an additional table/section or the inclusion of a plot with time in the x-axis.

2. The authors should consider to specify that the function  $f$ in $f(w_i)$ and $f(w_{i-1})$ differ, since they are computed on different minibatches. This could be done by e.g. using $f_i$ in place of $f$. I think this would make the exposition clearer.

3. Did the authors try to theoretically study the convergence of the proposed approach? 
I think this could be an interesting avenue for future work and could be discussed in the paper as a limitation of the current work.

4. How many runs have been done for each experiment? How are errors/variance in the table/plots computed? This should be discussed when presenting the result or in the appendix.

limitations: The authors properly discussed the limitations of the method in Sec. 5. Main limitations are handling high learning rate values and implementation issues related to the PyTorch implementation.
review_title: null

Meta-review: metareview: All reviewers recommend accepting the paper. Many years ago I experimented with this type of approach, for a single layer of hyper-parameters, and came to the conclusion that setting the hyper-hyper-parameters was just as difficult as setting the parameters of the optimization algorithm. I am shocked that this changes as you increase the number of layers, and I think others would be too. If this approach genuinely reduces hyper-parameter tuning in such a simple way, it should be a spotlight or oral presentation.
confidence: Certain
Decision: Accept