Gradient Descent: The Ultimate Optimizer
Authors: Kartik Chandra
Audrey Xie
Jonathan Ragan-Kelley
Erik Meijer
Venue: NeurIPS 2022 Accept (2022)
Keywords: automatic differentiation
differentiable programming
hyperparameter optimization

Abstract: Working with any gradient-based machine learning algorithm involves the tedious task of tuning the optimizer's hyperparameters, such as its step size. Recent work has shown how the step size can itself be optimized alongside the model parameters by manually deriving expressions for "hypergradients" ahead of time.

We show how to *automatically* compute hypergradients with a simple and elegant modification to backpropagation. This allows us to easily apply the method to other optimizers and hyperparameters (e.g. momentum coefficients). We can even recursively apply the method to its own *hyper*-hyperparameters, and so on ad infinitum. As these towers of optimizers grow taller, they become less sensitive to the initial choice of hyperparameters. We present experiments validating this for MLPs, CNNs, and RNNs. Finally, we provide a simple PyTorch implementation of this algorithm (see http://people.csail.mit.edu/kach/gradient-descent-the-ultimate-optimizer).

Gradient Descent: The Ultimate Optimizer
Kartik Chandra" MIT CSAIL Cambridge, MA kach@csail cilm@csail.mia mim.itt.. edu
Audrey Xie" MIT CSAIL Cambridge, MA ahx@mit. edu
Jonathan Ragan-Kelley
Erik Meijer Meta, Inc. Menlo Park, CA eri erikm@fb.com com
MIT CSAIL Cambridge, MA jrk@csail. mim. edu
Abstract
Working with any gradient-based machine learning algorithm involves the tedious task of tuning the optimizer's hyperparameters, such as its step size. Recent work has shown how the step size can itself optimized alongside the model parameters by manually deriving expressions for "hypergradients" ahead of time.
We show how to automatically compute hypergradients with simple and elegant modification to backpropagation. This allows us to easily apply the method to other optimizers and hyperparameters (e.g. momentum coefficients). We can even recursively apply the method to its own sypeer-yypepparaeeeet and so on ad in- finitum. As these towers of optimizers grow taller, they become less sensitive to the initial choice of hyperparameters. We present experiments validating this for MLPs, CNNs. and RNNs. Finally, we provide simple Py PyTorch implementation of this algorithm (see people.csail wwt.eccssi..iiiecuuuu. mmt..ed//kach///// wai..ed///acb/rradeentt -gradient-descentt deseet-eeeett--eeee pessccnttthh--use uultimate-optimiin
Introduction
When we train deep neural networks by gradient descent, we have to select a step size a for our optimizer. If fails to converge. Choosing an appropriate a is thus itself an optimization task that machine learning practitioners face every day. Why not apply gradient descent here, too? To do so, we need compute the derivative of the loss function not only with respect to the neural network's weights, but also with respect to a. Baydin et (2018), applying an insight from Almeida al. (1999), describe how to efficiently compute such "hypergradients" by manually differentiating standard optimizer update rules with respect the step size hyperparameter. This allows for on-line learning rate adaptation, which generally improves convergence, especially when the initial a is sub-optimal.
small, the optimizer runs very slowly, whereas if is too large, the optimizer
However, the above method has three limitations: (1) manually differentiating optimizer update rules tedious and error-prone, and must be re-done for each optimizer variant; (2) the method only tunes the step size hyperparameter, not other hyperparameters such as the momentum coefficient; and (3) the method introduces a new hyperparameter, the hyper-step-size, which must also be tuned.
In this paper, address all three limitations by replacing manual differentiation with automatic differentiation (AD), which (1) automatically computes correct derivatives without any additional human effort, and (2) naturally generalizes to other hyperparameters (e.g. momentum coefficient) for free. As for (3), we observe that AD can be applied to optimize not only the hyperparameters, but also the hypper-yyerpprramtter.. and the seeer-byyeer-ypppprraeeete can implement arbitrarily tall towers of recursive optimizers, which are increasingly robust to the choice of initial hyperparameter. These "hyperoptimizers" therefore reduce the burden humans responsible for tuning the hyperparameters. (Such an effect was hypothesized by Baydin et al., but not tested because manual differentiation of complex sequences of nested optimizers impractical.)
and so on. In fact,
"Equal contribution.
Work done in part at Meta, Inc, and in part at Stanford University.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Although "just apply AD" is seemingly straightforward recipe, an efficient implementation that properly allows for recursive self-application requires some care. To close the loop, we take inspiration from the study of recursion and combinators in programming language theory (and the long tradition of programming language papers named "Lambda: The Ultimate X*). We spell out the details in Section 2, and evaluate our method in Section 3. We find that across a variety of architectures (MLPs, CNNs, and RNNs) our hyperoptimizers are robust to suboptimal choices of initial hyperparameters, and that this robustness increases as we grow the stacks of optimizers taller.
Implementing hyperoptimizers
Consider some loss function that we want minimize using gradient descent, and let w; be the weights at the beginning of step i (we will omit subscripts on f. even though it varies at each step due to the stochasticity of minibatches). First, recall the standard weight update rule at step for SGD, using some fixed step size at:
af(w: Ow;
W1++
We would like to also update a at each step, so we will index it as well with the step number; that is, let ag be the step size at the beginning of step At each step, we will first update a, to a,+1 using some update rule yet to be derived, and then use the updated step size 0441 to update the weights from W/ to W1+1-
04+11 = 0;
adjustment for as
Of(ws) Ow,
W1+1 W.
a++1
What should the adjustment for a; be? By analogy to w, we want to adjust ag in the direction of the gradient of the loss function with respect to a,, scaled by some hyper-step size K. In other words, the adjustment should be x(0f(w:)/0a)) Our modified update rule is therefore:
Of(w))
0j+1 0; das
(1)
0f(w)
Wi+1 = w; - ac+l1
(2)
Ow,
All that remains is to compute (w(w)) (wx))/ax in equation (1). Below, we first review how Baydin et al. (2018) take this derivative by hand. Then, we show how obtain the same result automatically and efficiently using AD. Finally, we discuss how this automation allows us to generalize the method.
2.1 Computing the step-size update rule by hand
One option to compute (w:) (w;)/ explored by Baydin et al. (2018), is to proceed by direct manual computation of the partial derivative. Applying the chain rule (1), we have
① (w5-1 - ai
af(w:-1)
@f(w:) Jo;
af(w)
Ow;
af(w:)
Dw:
(3)
Ow:
Ow;
Sai
af(w)
a((r)-1))
Ow;
Owi-1
where (3) is obtained by substituting the update rule in (2) for wi and (4) is obtained by observing that w;-1 and (w;--1 do not depend on a, As Baydin et al. note, this expression lends itself to simple and efficient implementation: simply remember the past two gradients from backpropagation, and take their dot product obtain the hypergradient with respect the step size.
We were able to take this derivative by hand because the update rule for SGD simply multiplication by a constant, whose derivative is trivial. What about other optimizers? Consider the Adam optimizer (Kingma and Ba, 2014), which has much more sophisticated update rule involving the four hyperparameters 31, 82, (though is typically not tuned). Diffcrentiating the update rule by hand,

---

we obtain significantly more complex expressions for the hypergradients:
8f(w:-1) 0mn;
m,-11 + +1.1, ( -mm)
Ow:
my
Ow;
04
doy
(E: + Vo:)
3
(1 (1, (E2 + vol)
(- [8((i)=)) (- Bw;
Uz-1
ii22
(6- 16-1)
Ow;
agtiii
Ow;
dc;
(es + vo)
032;
20, (ei + ++vvio Uu)
This manual approach derive hypergradients simply does not scale: it tedious and error-proee, and must be repeated for every optimizer variant. However, with little bit of care, can compute hypergradients automatically and efficiently alongside the regular gradients.
2.2 Computing the step-size update rule automatically
In order to compute hypergradients automatically, let us first briefly review the mechanics of reverse- mode AD. Differentiable programming systems that provide reverse-mode AD typically build up computation graph as the function is computed forwardly. For example, when a user computes the function f(::). the system internally stores DAG whose leaves the weights wi, whose internal nodes are intermediate computations, and whose root is the final loss. It can then backpropagate through the computation graph starting at this root node, depositing gradients each internal node as it descends, until the weights w, at the leaf nodes have accumulated their gradients (ww)) Once the gradient 0 (f(w)) ((w,)/0rr /ów; computed by the backwards pass, we update the weights u1+1 Wi 0 (w:)/ /uw, and repeat the cycle for the next step of gradient descent.
An important consideration this point is for the weights be "detached" from the computation graph before the next iteration of this algorithm - that is, for the weights to be forcibly converted to leaves of the graph by removing any inbound edges. The effect of the "detach" operation is depicted in Figure la. If this step were skipped, backpropagation at the next iteration would continue beyond the current weights and into the previous iteration's is computation graph. Over time the computation graph would grow taller linearly in the number of steps taken; because backpropagation is linear in the size of the graph, the overall training would become quadratic-tine and intractable.
Let us take PyTorch as an example. In the built-in SGD optimizer (Paszke et al., 2017, optim/sgd.py commit ff94c9d), this is implemented by wrapping the update in the @torch. no grad context manager. Here, we need finer grained control over gradient flow, so will make the detaach () operations explicit. Below is pseudocode for SGD optimizer that uses detach !) have discussed. The highlighted calls to detach () correspond to detaching the weights and their gradients.
def SGD init init__(sel,, alpha):
self alpha alpha
def SGD. step d_w w.grad (etach()
w w. () self.alphe alt.falppaa.ettchh <d_ d_w
Now, in order to have backpropagation deposit the gradient with respect to a; as well as wi, we can simply refrain from detaching o, from the graph, detaching instead ily parents. This is depicted in Figure 1b. Because we want to compute 0f(w::/Oo;; the edge from o; to w, needs to remain intact. To implement this, instead of calling (etach() on alpha directly, we detach its parents when applying equation (1). This yields the following auttmeeu hyprrottimization algorithm:
def HyperSGD.. . ste)
update alpha using equation (1)
d_ d_alpha = welf.alpha..o..............m detach allh.alpha
aelf.allpa.dettaae
detach)) d_alpha
# update using equation (2)
d_ ..grad.detach() w.detach() self deta.ehh d.w
3

---

SHip
THipp
20060-11
deruh
desit
<PP
Nacipop
Teckpipp
(a) Computation graph of SGD with a single fixed hy- (b) Computation graph of SGD with a continuously- updated hyperparameter
perparameter
Figure Visualizing the computation graphs of SGD and HyperSGD.
Since only extend the computation graph by little extra amount, corresponding evaluating the optimizer, the hyperoptimizer's computational overhead is negligible (see Figure 4f).
2.3 Extending to other optimizers
As suggested by Maclaurin et al. (2015), should be possible apply gradient-based methods to tune hyperparameters of common variations on SGD such as AdaGrad (Duchi et al., 201 1), AdaDelta (Zeiler, 2012), Adam (Kingma and Ba, 2014). The above implementation of HyperSGD generalizes quite easily to these optimizers - we simply replace the last line with the new update rule.
Unlike previous work, method also allows for simultaneously optimizing all hyperparameters of these optimizers (e.g. all of a, $1, and B3 for Adam) "for free." We simply treat them just like a alpha in the implementation. Our evaluation in Section 3.2 demonstrates that this indeed advantageous to do. There are, however, two important subtleties: First, because hyperparameters like 31 and 33 must be strictly in the domain (0, 1), clamp the "raw" values to this domain using scaled sigmoid. Without this step, we might accidentally adjust these values outside their domains. Second, the Adam optimizer in particular involves the term vē,, which is continuous but not differentiable 0; Because Adam normally initializes 6n O, backpropagation fails on the first step due to division by zero. We fix this problem by initializing on rather than O.
2.4 Stacking hyperoptimizers recursively
At this point it is natural to ask whether the hyperoptimizer can itself be optimized; that is, whether the peprrahyperprrameters can be adjusted by hyper-hyyeroptinizer, The possibility of doing recursively ad infinitum obtain optimization algorithm that is highly robust to the human- chosen hyperparameter was hypothesized by Baydin et al. (2018). Computing the gradients of these higher-order hyperparameters by hand is impossible without knowing the exact sequence of stacked optimizers in advance, and, as discussed above, would be extremely tedious and error-prone.
However, the ability to compute these gradients automatically by AD makes it possible to realize this vision. To do so, let us revisit our previous implementation of HyperSGD. Notice that there is an opportunity for recursion lurking here: the adjustment to a plpha can be factored out with a call to SGD. step, where SGD's hyperparameter kappa.
def HyperSGD. (tpeerSGD.seep(w): update alpha using Equation (1)
SGD (kappa) step step(sell..alpil
# update using Equation (2)
d w.grad ..grad..deacci. detach()
w.detach() self.alph. d.w

---

Because SGD is already careful to properly detach its parameter (typically w, but in this case a), this implementation is functionally identical the one above. Indeed, any optimizer that observes this protocol would suffice, so let us abstract out the optimizer as a parameter to HyperSGD:
def HyperSGD... .......iii. i....nnit..... (self, a alpha, opt): self.alpha alpha self.optimizer = opt
def SteersGD..ttepw//
step step(self.alpha
d_w w.grad (
w. (w.eetch() self.alpin d_w
opt HyperSGD (0. 01, (pt=SGD(kappa))
Finally, after this refactoring, we can recursively feed HyperSGD itself as the optimizer, obtaining a level-2 hyperoptimizer HyperSGD 010 HyperSGD (0.01, SGD(O. 01) Similarly, imagine taller towers, or towers that mix and match multiple different kinds of optimizers, such as Adam- Siiimizzddyy-SGD-ppptiieee/yyy
natural concern whether this process actually exacerbates the hyperparameter optimization problem by introducing even more hyperparameters. Baydin et al. (2018) predicted that as the towers of hyperoptimizers grew taller, the resulting algorithms would become less sensitive to the human-chosen hyperparameters. This is indeed the case; Section 3.4 presents an empirical evaluation.
3 Experiments
In this section we evaluate the hyperoptimizers made possible by our system, exploring in particular the benefits of optimizing hyperparameters beyond just the step size, and of stacking hyperoptimizers to multiple levels. Each of these experiments was conducted on single NVIDIA TITAN Xp GPU,
3.1 Hyperoptimization for SGD
First, establish some basic properties about hyperoptimizers: (1) whether SGD hyperoptimizer performs better than a regular SGD optimizer, and (2) whether the final learned step size is better than the initial human-chosen step size. We test the latter property by running fresh regular SGD optimizer with the final learned step size of the hyperoptimizer. Following authors of prior work (Maclaurin et al., 2015; Baydin al., 2018), we conducted initial experiments on the MNIST dataset (Lecun et al., 1998) using a neural network with one fully-connected hidden layer of size 128, tanh activations, and batch size of 256. We trained all networks for 30 epochs, reporting statistics over 3 runs. As a baseline we used SGD with a = 0.01.
Table la summarizes the results of our experiments. We find that hyperoptimized SGD outperforms the baseline by a significant margin. This holds even if we use other optimizers (c.g. Adam) to adjust the step size of the SGD optimizer. Furthermore, when re-ran the regular optimizers with the new learned hyperparameters, we found that they performed better than the initial hyperparameter.
3.2 Hyperoptimization for Adam, AdaGrad and RMSProp
In Section 2.3, we described how to apply our system to the Adam optimizer, simultaneously optimizing not only the learning rate a, but also the momentum coefficients 31,2- Here, we ask three questions: (1) whether hyperoptimized Adam optimizers perform better than regular Adam optimizers, (2) whether the learned hyperparameters outperform the baseline, and (3) whether there is benefit to optimizing all the hyperparameters, opposed to only optimizing the learning rate as Baydin et al. (2018) do. Because Adam has significantly faster convergence than SGD, we only these experiments for 5 epochs to avoid overfitting.
Table 1b summarizes the results of our experiments. We find that indeed the hyperoptimized Adam optimizer outperforms the regular Adam optimizer on its "default" settings. As with SGD, the learned hyperparameters perform better than the initial hyperparameters when re-run with the regular optimizer. Inspecting the learned hyperparameters, we find that the algorithm raises the learning rate
5

---

Optimizer
Test error
Optimizer
Test error
SGD
8.99=0.05%
Adam
4.67=0.06%
SGD/SGD
4.81
Adam / SGD(10
3.03=0002%
SGD(O.)0769)
5.44
Adam(0.0004) 0.899, 0.999)
3.11=0006%
SGD / 1d1m(0.1)
4.86=20066%
Adam / SGD(10
3.12=0004%
SGD(0)4538)
2.80=0.09%
Adamf(0.002))
3.47=0.02%
SGD/ AdaGrad
4.85:10.21%
Adam / Adam
3.05%
SGD(0.0833)
5.10%%0033%
Adam(0. (0.0038, 0.870, 0.999)
3.24=0.13%
SGD / RMSprop(O..)
4.52=0.02%
Adam / Adam
0.08%
SGD(O))5220)
2.02+0.07%
Adam (0.0036)
0.022
(a) Experiments with SGD (Section 3.1)
(b) Experiments with Adam (Section 3.2)
Optimizer
Test error
Optimizer
Test error
AdaGrad
7.08%
RMSProp
4.19+0047%
AdaGrad SGD
6.90=0.169
RMSProp"/SGDDLI
3.55+:..23%
AdaGrad(0.0088)
7.75=0.02%
RMSProp((0.000))
3.93:10.70%
AdaGrad AdaGrad AdaGrad(0.0151)
5.03=0.23%
RMSprop /SGD((O
3.33:10.07%
6.67
RMSProp(0.0032, 0.9899)
3.25+0009%
(c) Experiments with AdaGrad (Section 3.2)
RMSProp"//MSSPA RMSPPrp((.002.1)
3.42:10.45%
3.6010.04%
RMSProp /RMSProp(100 RMSPProp(0.0000, 0.9962)
2.96+00.11%
3.65+
(d) Experiments with RMSProp (Section 3.2)
Table 1: Hyperoptimization experiments with MNIST. We denote hyperoptimizers by their constituent optimizers separated by slashes (the leftmost item adjusts the model's weights). Adam is an Adam optimizer where only is optimized as by Baydin et al. (2018); RMSProp" is similar. If not specified, initial hyperparameters are PyTorch defaults (10-² for learning rates except 10-3 for Adam; 31 = 0.9, B2 0.99 for Adam and y repeated using the final hyperparameters (typeset pink) learned by the algorithm.
0.99 for RMSProp). Each hyperoptimizer experiment
and slightly lowers 31. but does not significantly affect B2. Nevertheless, learning 31 does help slightly, though not when the top-level optimizer is itself another Adam optimizer.
Similarly, we can add any other optimizer to our system with just a few straightforward lines of code. Here, show results for AdaGrad (Table 1c) and RMSProp (Table 1d; also These experiments took less than an hour each to implement from scratch, and show that every hyperoptimizer stack outperforms the non-hyperoptimized baseline. We remark that AdaGrad is known to "stall" over time as the effective step size goes to zero; inspecting the learned over time, we find that the AdaGrad/ AdaGrad hyperoptimizer increases a make up for this effect. Additionally, we tried to hyperoptimize RMSProp's new 7 parameter, which modulates the accumulation of gradient RMS terms. This yielded even better results (compare a.7 trials), and required only llline change in our code.
epochs).
3.3 Hyperoptimization at scale
Next, we evaluate our hyperoptimizers on two different real-world neural network architectures.
3.3.1
Convolutional neural networks for computer vision
We train a ResNet-20 (He al., 2016) with and without hyperoptimization on the CIFAR-10 dataset (Krizhevsky, 2012). As baseline, replicate the training procedure of He al. (2016):

---

Comparison of tran loss with and without typerggiiiiiaaionn
Compurison test with without Ryperpiimizationn
H 0.009
p-9.
4-0.99
p-0.99
p-09
H200999
cts. 2015
3
Buselie
2x10
10
10
101
16
10
I00
200
100
200
100
200
100
10c
200
100
200
100
209
Fpods
(a) For wide range of "bad" initial hyperparameter configurations, the hyperoptimizer improves on (or least matches) final test accuracy, and often matches or even outperforms the "good" initial hyperparameters.
Train loss with initial parameters 4-0.9
Test crror with initial pazameters d- u-0.0
a over time
cl.
60
at. 2015
0.10
10'
Nidbont-byaaeetiiinn
4:100
0.06
Hypenqiniio
Trus 365
0.06-
He al 2015 Wilkos--ppsopiiiize
2-10
0.04
Eypengiiiios
0.10
0.00
100
200
3000
400
100
300
500
0
100
200
300
530
Egeche
Epocks
Dpado
(b) The hyperoptimizer matches performance of the hand-engineered learning rate decay schedule by He et al. (2016), learning a strikingly similar decay schedule (right plot).
Figure 2: Training ResNets on CIFAR-10 with hyperoptimizers (Section 3.3.1).
use the same network architecture, optimizer (SGD), step size (0.1), momentum (0.9), and weight decay (10-1), though without their custom learning rate decay schedule (which we will address later). Experiments were run for 200 epochs, which takes around 3 hours on our hardware.
First, we test how robust the hyperoptimizer is to "bad" initial choices of step size and momentum. We vary the initial step size and the momentum among "small." "good," and "large" values (that is, a E (0.01, 1.0, 1.0) and E {0.09, 0.9, 0.99]), and add a hyperoptimizer (a. = a² 10 103 On 1) 10-9). The results of this experiment are shown in Figure 2a. every configuration, the hyperoptimizer matches or outperforms the regular optimizer in final test accuracy. Furthermore, in nearly all of the configurations, the hyperoptimizer matches or exceeds the "good" hyperparameters' final test accuracy. Only when both hyperparameters are bad in the same direction (too small or too large) is it unable to manage this, and even then for the too-large case dramatically lowers the loss compared to no hyperoptimizer. We conclude that hyperoptimizers are indeed beneficial for tuning both step size and momentum in this real-world setting.
Next, we add in the learning rate decay schedule hand-enginen engineernd by He et al. (2016): the step size is divided by 10 epochs 100 and 150. We compare this with hyperoptimizer initialized with the same starting hyperparameters, training both variants for 500 epochs. Our results are shown in Figure 2b. The hyperoptimizer not only matches the final test loss of the hand-engineered learning rate decay schedule, but also learns a decay schedule strikingly similar to one hand-engineered by He al. Of course, both networks significantly outperform the baseline trained with a fixed step size.
3.3.2 Recurrent neural networks for language modeling
We train character-level RNN ("Char-RNN") on the Tolstoy dataset, as proposed by Karpathy et al. (2015) as convenient testbed for language models, which is now often used to benchmark optimizers (Schneider et al., 2018 8; Schmidt et al., 2021). We took the architecture (2-layer LSTM with 128 hidden nodes) and "expert" optimizer (Adam optimizer with a = 2 10 3 run for 50,000 gradient descent steps) directly from Johnson (2017) as recommended by Karpathy et al. We compare against
7

---

Training an RNN with and without hyperoplimization
G-1e
a-
0-1e
4x10
Hypenoptimizer
3x10"
2x10'
200
Baseline
Test perpiciiy
Hyperoptimizer
10'-
Borations (1e2)
Figure 3: Training RNNs with hyperoptimizers (Section 3.3.2). As the initial learning rate is lowered, the regular Adam optimizer's convergence slows, but the hyperoptimizer is able to accelerate it. The hyperoptimizer also slightly improves convergence when the initial learning rate is too high.
our HyperAdam optimizer on wide range of initial learning rates E {10- 4,2 10 10 10-0), 2}. with a, = a 10 2 We do not vary initial 31.2 because in our experience these hyperparameters are typically left at their default values. However, we allow the hyperoptimizer to vary $1,2 over the course of training (with 03 10 and a 2 10-4), All runs took around hour to train.
The results this experiment are shown in Figure 3. We find that hyperoptimizer performs comparably to the expert-chosen fixed step size (perplexity 5.41 = 0.26 with hyperoptimizer vs 5.27 + 0.31 without), and improves upon "bad" initial step sizes in both directions (5.45 + 0.76 vs 5.77 + 0.34 when too high; 6.51 + 0.88 8.71 + 0.91 when too low).
3.4 Higher-order hyperoptimization
In Section 2.4 developed an interface for building arbitrarily tall towers of optimizers. Baydin et al. (2018) hypothesized that taller towers would yield hyperoptimizers that were increasingly robust to the initial human-chosen hyperparameters. To validate this behavior of higher-order hyperoptimizers, we ran each of our benchmarks from above (MLP on MNIST, CNN on CIFAR-10, RNN on Tolstoy) with towers of hyperoptimizers of increasing heights, and with bottom-level step sizes initialized across many orders of magnitude. In practice we find that if the initial hyper-step sizes are too large, the computation diverges for networks larger than the MNIST MLP. So, we initialize each level's hyperparameter to be smaller than that of the previous level. Specifically, we use the following scheme: from 10-8 to 10-4 the higher layers' step sizes were initialized to (a 102 a 10 10-2] respectively, while for a 2 10 10-3 they were initialized to Ja 10
10 10 respectively.
Figure 4 shows results. It is indeed the case across these different benchmarks (each of which has a different dataset, architecture, and optimizer type) that the taller the hyperoptimizer stack, the less sensitive the results become to the human-chosen hyperparameters. With three-level optimizer stack, a single hyperoptimizer design obtains reasonable results all of our benchmarks across several orders of magnitude of base-level step size.
Further tests of scalability To test if our hyperoptimizers continue to work in even larger regimes, we fine-tuned a ResNet-152 (pretrained on ImageNet) to the Caltech-256 dataset Griffin et al. (2007). Figure 4e shows the results: height-1 hyperoptimizer recovers z 1 11% emror for both 10 and = 10 (without a hyperoptimizer, a = 10 gives 91.5% error). height-2 hyperoptimizer is additionally able to make significant progress when 10-2
We stress how lightweight and practical this method is. Figure 4f shows how runtime scales as function of hyperoptimizer stack height for the above benchmarks. We find that the scaling is linear:

---

Tesi vayynn intta value of
Test crior inttal value of
brghh-0
beggle-
heigh-1
langle
hrghh2
heightr
60-
60
Tee emee
40-
20-
20-
Log 0
Log
(a) Results MLP (Sec 3.1). where all layers (b) Results MLP (Sec 3.1), where all layers initialized with the same step size.
initialized in Sec 3.4.
Test errec varying inial volue of
Test peplexity varying inital value of
90-
hrighen
1nghh-1
60-
beighe1
10
hright-2
laigle-2
70-
hrightes
90
leighe-
Tee peplery
40-
Log
Log
(c) Results ResNet (Sec 3.3.1)
(d) Results Char-RNN (Sec 3.3.2)
Test error varying inital valuue of
Runtime Cost Highen--rdee Hlppropeimizaiion
hnigle-
Rustian pet saax lacaae over o0 a0-ooooo
MLL os MNST
98
beight1
+0000
teigle
RN0N on Tolauy
Test eator %
+500%
10.0%
20
-3.0%
T.og
(e) Results on fine-tuning a pretrained ResNet-155 (f) Our hyperoptimizers have minimal impact on to the Caltech- dataset (Sec 3.4)
runtime, which scales linearly in height (Sec 3.4)
Figure 4: Evaluating higher-order hyperoptimization across a variety benchmarks (Section 3.4). As we stack more layers of optimizers, the resulting hyperoptimizer is less sensitive to the initial choice of hyperparameters, but costs only 1-2% more in runtime.
each additional level costs only 1-2% in additional runtime above the non-hyperoppimized baseline, and negligible additional memory.
4 Related work
Hyperparameter optimization has a long history, and we refer readers to a recent survey by Feurer and Hutter (2019) for the full story. Most existing work on gradient-bases hyperparameter optimiza- tion (Bengio, 2000; Domke, 2012: Maclaurin et al., 2015; Pedregosa, 2016; Franceschi et al., 2017) has focused on computing hyperparameter gradients after several iterations of training, which computationally expensive. Baydin et al. (2018), building technique first published by Almeida

---

et al. (1999), propose instead updating hyperparameters at each step, and Rubio (2017) provides convergence analysis. Wu al. (2018) demonstrate a "short-horizon bias" in the related online "stochastic meta-descent" algorithm (Schraudolph, 1999), but Lichtarge et al. (2022) empirically find encouraging results when optimizing large sequence-to-sequeee
models. Luketina al. (2016) apply similar technique to regularization hyperparameters, though they note that their proposed method could work in principle for any continuous hyperparameter. Grefenstette et al. (2019) provide library for metalearning via hypergradients. As discussed above, we expand upon this line of work in three directions: (1) by fully automating this process, rather than requiring manual derivative computations: (2) by optimizing hyperparameters beyond just the learning rate; and (3) by realizing the vision of recursive higher-order hyperoptimizers and evaluating the resulting algorithms. We find that they are indeed more robust the initial human-chosen hyperparameter, which relates our work to other learning algorithms that minimize sensitivity to learning rates (Orabona and Tommasi, 2017; Vaswani et al., 2019).
5 Limitations and future work
As discussed in Section 3.4, one limitation of hyperoptimizers is that they cannot yet handle initial hyperparameters that are set far too high, because the system is unstable and diverges before the hyperoptimizer can have an effect. Designing hyperoptimizers robust in this regime requires further research, such as deeper theoretical analysis of convergence. Our implementation also requires some care in avoiding certain bugs related to computation graph management. For example, loggers must detach what is logged to avoid memory leaks because tensors are not garbage collected unless all children are detached. Similarly, certain PyTorch modules (e.g. the built-in LSTM) cannot be used because they silently modify the computation graph, which may lead to incorrect gradients with our system. Further research is needed to design differentiable programming languages where methods like ours can be expressed in modular and composable manner that minimizes the risk of such bugs.
Broader impact Training a modern deep learning system consumes tremendous amount of energy, and hyperparameter searches can multiply that energy impact by many orders of magnitude (Strubell et al., 2019). We hope that advances in on-line hyperparameter tuning can reduce this impact.
Conclusion
We presented a technique that enables gradient descent optimizers like SGD and Adam to tune their own hyperparameters. Unlike prior work, our proposed hyperoptimizers require manual differentiation, learn hyperparameters beyond just learning rates, and can be stacked recursively to many levels. We described an elegant recursive implementation of hyperoptimizers in a reverse-mode AD system and evaluated it on a variety of benchmarks, showing that as the stacks grow taller, they become less sensitive to the initial human-chosen hyperparameter.
Acknowledgments and Disclosure of Funding
We thank Samantha Andow, Emilio Arroyo-Fang, Irene Dea, Johann George, Melissa Grueter, Basil Hosmer, Steffi Stumpos, Alanna Tempest, and Shannon Yang for early discussions, Krishna Murthy Jatavallabhula and Josh Tenenbaum for their advice when preparing this paper, and the anonymous reviewers for their thoughtful feedback. KC and JRK were supported by NSF Grants #2105806, #CCF- 1231216 MIT. Additionally, KC supported by a Hertz Foundation Fellowship, the Paul and Daisy Soros Fellowship for New Americans, and an NSF Graduate Research Fellowship under Grant #2 141044 and AX was supported by the MIT Undergraduate Research Opportunities Program (UROP).
and #CCF- 184602,, and ONR Grant #00010803 at
References
I.. Almeida, T. Langlois, J. M. do Amaral, and Plakhov. Parameter adaptation stochastic optimization. In On-Line Learning in Neural Networks, 1999.
10

---

A. G. Baydin, R. Cornish, D. M. Rubio, M. Schmidt, and F. Wood. Online learning rate adaptation with hypergradient descent. Sixth International Conference on Learning Representations (ICLR). Vancouver, Canada, April 30 May 3, 2018, 2018.
Y. Bengio.
Gradient Cradient-based optimization of hyperparameters. Neural Computation, 12(8): 1889-1900, 2000. doi: 10. 10.116220899966000000011555 089976660300015187.
URI. https://doi.oon orgg 1162/
J. Domke. Generic methods for optimization-baseeee modeling, In N. D. Lawrence and M. Girolami, editors, Proceedings of the Fifleenth International Conference on Artificial Intelligence and Statis- fics, volume 22 of Proceedings of Machine Learning Research, pages 318-326, La Palma, Canary Islands, 21-23 Apr 2012. PMLR. URL http:ht/pprocceddnngg.g mlr press/v22 /dmk2e2. html.
Duchi, F. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12:2121-2159, July 2011. ISSN 1532-4435. URL http:
//dl. acm
2621068.
M. Feurer and F. Hutter. Hyperparameter Optimization, pages 3-33. Springer International Publishing. Cham, 2019. ISBN 978-3-033-005118-5 doi: 10. 000077977----33-05311-----3333 URL https: //doi. 0t./10..000 0000077977-20 3-00--33-----0-00 05318-5 1.
L. Franceschi, M. Donini, P. Frasconi, and M. Pontil. Forward and reverse gradient- based hyperpa- rameter optimization. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 Proceedings of Machine Learning Research, pages 1165-1173, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http://proceediins. ww..rrr... wwt.:/sw......ps/////000 http///fraaneschh1ee html.
E. Grefenstette, B. Amos, D. Yarats, P. M. Htut, A. Molchanov, F. Meier, D. Kiela, K. Cho, and Chintala. Generalized inner loop meta-learning. urXiv preprint 1rX1v:1910.01777, 2019.
G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. 2007. URL http: //authors. library. www.ecch.hc.... wal.tech.ddu/6694
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016. doi: 10. 00.100/CVPR.2011.900 URL https://doo http://ooo.orgg//// 10. 1109/1199/PP 2016.90.
J. Johnson. torch-rnn. Github repository, 2017. URL https://ggihubb.cmm/jjjj/oo/// torch rnn.
A. Karpathy, J. Johnson, and L. Fei-Fei. Visualizing and understanding recurrent networks. arXiv 2015. URL
preprint 150ivv:500.02000
02078. 78. pdf.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference Learning Representations, 12 2014.
A. Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1) (6(11)):278-233 Nov 1998. ISSN 0018-9219. doi: 10.1109/5.726791.
Lichtarge, C. Alberti, S. Kumar. Simple and effective gradient- based tuning of sequence-to- sequence models. AutoMI., 2022. URL
pdf / 22009 04683. p4f83.pdf.
Luketina, M. Berglund, Greff, and Raiko. Scalable gradient- based tuning continuous regu- larization hyperparameters. In Proceedings of the 33rd International Conference on International Conference on Muchine Learning Volume 48, ICMI'16, pages 2952-2960. JML..org, 2016. URL http:/ d/ acm www.org.cciaaaiiioooooommm

---

D. Maclaurin, D. Duvenaud, and R. P. Adams. tradient-basedd hyperparameter optimization through reversible learning. In Proceedings of the 32Nd International Conference on International Con- ference Machine Learning- Volume 37, ICMI''15, pages 21 131-2122. JMLR.org. 2015. URL
http://dl. acm. prggcitatiio..... cfm wtw.cccmmmmmmmmmmmmm 3045343.
Orabona and T. Tommasi. Training deep networks without learning rates through coin betting. Advances in Neural Information Processing Systems, 30, 2017.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
F. Pedregosa. Hyperparameter optimization with approximate gradient. In Proceedings of the 33rd International Conference International Conference on Machine Learning Volume 48, ICML'16. pages 737-746. JMLR.org, 2016. URL //tt:://d.... acm. wttp//rgg////iieee wwttttiinccmmidda333...... cfm?id=3045399. 3045469.
D. M. Rubio. Convergence analysis of an adaptive method of gradient descent. University of Oxford. Oxford, M. Sc. thesis, 2017. URL https:://ammrru2 htt.g//aaaea..eeeeeeeaaarrtttaaa
io/ wwt.///nencceeeeeeeeeeee anslysis. pdf.
R. M. Schmidt, F. Schneider, and P. Hennig. Descending through a crowded wally--benchmarking deep learning optimizers. In International Conference Machine Learning, pages 9367-9376. PMLR, 2021.
Schneider, L.. Balles, and P. Hennig. Deepobs: deep learning optimizer benchmark suite. In International Conference on Learning Representations, 2018.
N. Schraudolph. Local gain adaptation in stochastic gradient descent. 1999 Ninth International Conference on Artificial Neurul Networks ICANN 99. (Conf. Publ. No. 470), volume 2, pages ieee. org/
569-574 vol.2, 1999. doi: 10. 1049/c:: 1991 URI. document/817990.
E. Strubell, A. Ganesh, and A. McCallum. Energy and policy considerations for deep learning in NLP. arXiv preprint 190.:v:996.0024/// 2019. URL https: //ii:v////iii////// wttpp/wwwwo//////////////// 02243. pdf.
S. Vaswani, A. Mishkin, I. Laradji, M. Schmidt, G. Gidel, and S. Lacoste-Julien. Painless stochastic gradient: Interpolation, line-search, and convergence rates. Advances in neural information processing systems, 32, 2019.
Y. Wu, M. Ren, R. Liao, and R. Grosse. Understanding short-horizon bias in stochastic meta- optimization. ICLR, 2018. URL https://arxiv. wttp://...rg/////////////// http://838.002...111 020.1.222.... pdf.
M. Zeiler. ADADELTA: adaptive learning rate method. CoRR, abs/1212.5701, 2012. URI. http://dblp.. teiittrier.e
5701.
12


Reviews:
[{"review_id": "y1G0IDrGdcf", "reviewer": "Reviewer_oELp", "summary": "This work presents a way of optimizing parameters (hyperparameters) of first order optimizers as the learning rate and the momentum coefficient using another first order optimizer as SGD or ADAM, called hyperoptimizer. Furthermore, this method can be applied recursively to optimize the parameters of the hyperoptimizer and so on, creating a chain of optimizers. The gradient of the loss w.r.t. the hyperparameters is computed by backpropagating through only one step of the optimizer, so that the weights and biases of the neural network and the hyperparameters can be optimized jointly with a small increase in cost compared to standard training. \n\nThe authors provide a simple pseudocode implementation in pythorch which exploits Automatic Differentiation (AD), thus avoiding error-prone implementation of derivatives by hand. They also validate this method on several neural architecture and datasets: MLPs on MNIST, CNNs on CIFAR-10 and RNNs on Tolstoy. Results show that this approach often outperforms training with the fixed sub-optimal hyperparameters used as a starting point for the algorithm, and is comparable to using the hyperparameters found at the end of training. Furthermore, they show that, when optimizing only the learning rate,  the longer the chain of optimizers the less sensitive the final performance is to the initial learning rate.\n", "questions": "1. How much higher is the time and memory cost of the method compared to standard SGD/ADAM? how does it increase as the chain of (hyper-)optimizers becomes longer? \nThe computational cost is discussed only briefly in the text and all plots have the number of epochs in the x-axis. I think this aspect is worth an additional table/section or the inclusion of a plot with time in the x-axis.\n\n2. The authors should consider to specify that the function  $f$ in $f(w_i)$ and $f(w_{i-1})$ differ, since they are computed on different minibatches. This could be done by e.g. using $f_i$ in place of $f$. I think this would make the exposition clearer.\n\n3. Did the authors try to theoretically study the convergence of the proposed approach? \nI think this could be an interesting avenue for future work and could be discussed in the paper as a limitation of the current work.\n\n4. How many runs have been done for each experiment? How are errors/variance in the table/plots computed? This should be discussed when presenting the result or in the appendix.\n", "limitations": "The authors properly discussed the limitations of the method in Sec. 5. Main limitations are handling high learning rate values and implementation issues related to the PyTorch implementation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n1. The method is simple to implement, clearly explained and probably not too costly compared to standard optimizers. The authors present Pytorch code snippets and discuss the approach in great detail.\n2. Convincing experimental evaluation. Experiments are done with several architectures, MLPs CNNs and RNNs, and small to medium sized dataset. The advantages brought by the method are evident and match the authors’ claims.\n3. High significance. A Better optimizer for training neural networks is useful to the majority of the machine learning community. The ease of implementation could favor its adoption.\n\nWeaknesses:\n1. Comparison with related works can be improved. The authors discuss a connection with programming language theory without providing references. They also do not discuss or compare against other adaptive optimizers, as [1, 2], which are also less sensitive to the initial learning rate and are more theoretically grounded.\n2. Lacks large scale experiments (e.g. on ImageNet) and does not include transformers models, which are widely used neural architectures.\n3. Limited Novelty. The idea of optimizing the learning rate using a first order method with gradient computed by back propagating through one step of the optimizer was presented in [3]. The novelty of this work lies in (i) applying the idea recursively, (ii) optimizing other hyperparameters like the momentum coefficient, (iii) providing a simple Pytorch implementation exploiting automatic differentiation and (iv) an independent experimental evaluation.\n\n[1] Orabona, Francesco, and Tatiana Tommasi. \"Training deep networks without learning rates through coin betting.\" Advances in Neural Information Processing Systems 30 (2017).\n\n[2] Vaswani, Sharan, et al. \"Painless stochastic gradient: Interpolation, line-search, and convergence rates.\" Advances in neural information processing systems 32 (2019).\n\n[3] Baydin, Atilim Gunes, et al. \"Online Learning Rate Adaptation with Hypergradient Descent.\" International Conference on Learning Representations. 2018.\n\n**Post author's response**\nThe authors addressed most of my concerns and promised to address others. Therefore I am increasing my score from 6 to 7.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KQhhNelvS-N", "reviewer": "Reviewer_kxHt", "summary": "This paper proposes to apply reverse mode auto-differentiation to compute gradients w.r.t  hyperparameters of an optimizer and update them using gradient descent, in contrast to computing them manually as done in previous works. It offers the flexibility in terms of choosing the optimizer for updating the hyperparameters. It also proposes to apply this procedure recursively for each hyperparameter of hyperparameters, resulting in a stacked hyperoptimizer. Experimentally, it tests the hyperoptimization idea on the hyperparameters of SGD and Adam, and compare its performance against the case when hyperoptimization is not used. \n", "questions": "- To strengthen the paper, it would be good if the author can further test the stacked hyperoptimizer on more optimizers, such as AdaGrad, RMSProp etc, and report similar results as Figure 2 and 4. \n- It would be good to plot also the training loss of stacked and unstacked hyperoptimizers. It is interesting to see whether the hyperoptimizer converges faster than normal SGD or Adam. Also please report the computation time for different heights of the stacked hyperoptimizer. \n- I believe the experiments of this work are interesting and intriguing, and can be beneficial for the community to reduce the search cost of hyperparameter or build better optimizers. This paper is at the border line of acceptance. If all my concerns are addressed, I would raise my score. ", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- This paper is well-written and can be easily understood. The pseudocode together with the computation graph clearly demonstrate the hyperoptimization procedure. There is a minor typo in section 3.1, should be “4.08% and 4.48% test error for SGD/SGD and SGD/Adam, respectively”. The main findings are also highlighted. \n- The novelty of this paper is limited. As mentioned in the paper, [1] computed the derivative of loss w.r.t. the learning rate of SGD and performed gradient-descent type update. For optimizers with more hyperparameters and more complicated gradients such as Adam, the paper does not demonstrate the clear advantage of optimizing all the hyperparameters over only optimizing the learning rate. For example, in Table 1 (b), when only the learning rate of Adam is optimized, the resulting test error is lower than that when all the hyperparameters are optimized (3.00 % vs 3.03 %). Hence, autodifferentiation may not take full advantage over manual computation when simple expression for updating the learning rate can be derived as in the case of SGD or Adam. \n- Despite lack of novelty, some interesting results are presented. Figure 1a) shows that when hyperoptimization is used, the algorithm is less sensitive to the initialization of hyperparameters, and can make certain degree of corrections when bad initialization occurs. This is an intriguing property as a good initialization is difficult to know a priori and can be dependent on many factors such as optimizer and dataset. Another interesting observation is that the final step size can be either larger or smaller when compared against its initial value as shown in Table 1 (a) and Figure 2 (b). This shows that the hyperoptimization approach can tune the hyperparameter properly depending on its magnitude. \n- One main contribution of this paper is that with autodifferentiation, one can perform gradient -type optimization for the hyperparameter of hyperparameters. The results seem to show that test error decreases as height increases, but it is not very convincing when comparing Figure 4 b) with Figure 4 c). The claim regarding the sensitivity of different height to base-level step size does seem to be true. \n- Some main limitations of the experimental results are: given hyperparameters are properly initialized, the hyperoptimization does not demonstrate clear advantage over the case when they are fixed during training. Another limitation is that even hyperoptimizer can optimize hyperparameters of hyperparameters, but one still need to provide proper initializations for these higher-level hyperparameters. Hence, the search cost of hyperparameter may not be necessarily reduced. Moreover, as mentioned in the paper, it is difficult to provide a common initialization for different heights. This may limit the usage of the stacked hyperoptimizer. \n\n[1] Online Learning Rate Adaptation With Hypergradient Descent", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Dmgv4K5VgJf", "reviewer": "Reviewer_zqnY", "summary": "This paper outlines a method to computer hypergradients using automatic differentiation by modifying backpropagation, allowing hypergradient descent to be extended to arbitrarily large layers of hypergradient descent without consuming arbitrarily large amounts of programming labor. The paper then uses this technique to validate the hypothesis that taller towers of hyperoptimization yield increasingly robust hyperparameters, eventually converging to the optimal as the number of layers increases even with poorly chosen initial values.", "questions": "- The computational overhead of hypergradient descent using AD is stated to be negligible; what is the actual value –  is it even statistically measurable?\n", "limitations": "Authors address all limitations I can think of.\n", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Paper is extremely well written, and the best that I have reviewed recently. The writing is very easy to understand and follow.\n- I believe that the experimental results validating the higher order hyperoptimization hypothesis will be immensely valuable to the optimization and meta-learning community.\n- Well-documented, easily readable, and easily usable source code is provided.\n- The method presented will be very useful to the community in advancing future meta-learning and optimization research.\n\nWeaknesses:\n- The contributions of this paper are primarily in presentation, and not as deep or novel as the typical NeurIPS paper.\n- Evaluation networks are rather small compared to networks used in modern practice, and more comparable to those used by learning to optimize papers that are crippled by high memory and compute requirements. Since the overhead of hypergradient descent is much lower, it would be good to see a benchmark of at least a ResNet-152 or some transformer architecture, especially in a higher-order hyperoptimization experiment.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept