MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters
Authors: Arsalan Sharifnassab
Saber Salehkaleybar
Richard S. Sutton
Venue: ICML 2025 poster (2025)
Keywords: Continual optimization
meta-parameter optimization
Primary area: optimization->nonconvex

Abstract: We address the challenge of optimizing meta-parameters (hyperparameters) in machine learning, a key factor for efficient training and high model performance. Rather than relying on expensive meta-parameter search methods, we introduce MetaOptimize: a dynamic approach that adjusts meta-parameters, particularly step sizes (also known as learning rates), during training. More specifically, MetaOptimize can wrap around any first-order optimization algorithm, tuning step sizes on the fly to minimize a specific form of regret that considers the long-term impact of step sizes on training, through a discounted sum of future losses.  We also introduce lower-complexity variants of MetaOptimize that, in  conjunction with its adaptability to various optimization algorithms, achieve performance comparable to those of the best hand-crafted learning rate schedules across diverse machine learning tasks.

MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters
Arsalan Sharifnassab 12
Saber Salehkaleybar Richard Sutton
Abstract
optimal trial and error search methods.
We address the challenge of optimizing meta- parameters (hyperparameters) in machine Iearn- ing, key factor for efficient training and high model performance. Rather than relying ex- pensive meta- parameerr search methods, we in- troduce MetaOptimize: a dynamic approach that adjusts meta- parammttersss,s particularly step sizes (also known as learning rates), during training. More specifically, MetaOptimize can wrap around any first-order optimization algorithm, tuning step sizes the fly to minimize a specific form of re- gret that considers the long-term impact of step sizes on training, through a discounted sum of fu- ture losses. We also introduce lower-complexity variants of MetaOptimize that, in conjunction with its adaptability to various optimization algo- rithms, achieve performance comparable to those of the best hand-crafted learning rate schedules across diverse machine learning tasks.
Meta-paraman parameter optimization is particularly important in continual learning (De Lange et al., 2021), where continu- ally changing environments or evolving loss functions ne- cessitate adaptation of meta-past meta-parameters, such as step sizes, to track time-varying optima rather than settling on static value.
In this work, we propose MetaOptimize, a general frame- work for optimizing pettaparaeeters to minimize form of regret that explicitly accounts for the long-term influence of step sizes on future loss. Although this framework can handle various meta-parameters, we concentrate on step sizes they ubiquitous and crucial in practice.
MetaOptimize offers additional advantages beyond reduc- ing search overhead. First, it enables dynamic step-size updates during training, potentially speeding the learning process. Traditional methods typically rely on manually designed learning rate schedules (e.g., initial increase fol- lowed by decay (Amid et al., 2022)), whereas MetaOptimize automatically discovers similar patterns.
Second, adapting step sizes across different network blocks (e.g., layers neurons) can improve performance (Singh et al., 2015; Howard & Ruder, 2018), yet manually tuning such blockwise step sizes is impractical for large networks. By design, MetaOptimize handles these blockwise adjust- ments.
1. Introduction
Optimization algorithms used in machine learning involve meta peta-parameters (i.e., hyperparameters) that substantially influence their performance. These meta- metam-earsestrs are typ- ically identified through a search process, such as grid search or other trial-and-error methods, prior training. However, the computational cost of this meta-parameter search is significantly larger than that of training with optimal meta- parameters (Dahl et al., 2023; Jin, 2022). Meta- peta-parameter optimization seeks to streamline this process by concur- rently adjusting meta- tatta--armmeeers during training, moving away from the computationally expensive and often sub-
The concept of meta step-size optimization dates back to (Kesten, 1958), Delta-bar-Delta (Sutton, 1981; Jacobs, 1988), and its incremental variant, IDBD (Sutton, 1992). Numerous methods have emerged over the years (Section 8). This work distinguishes itself from prior efforts through the following key aspects:
We introduce a formal approach to step-size optimiza- tion by minimizing a specific form of regret, essentially a discounted sum of future losses, and demonstrate how to do this causally via the MetaOptimize framework.
'Opemmind Research Institute, Canada Department of Com- puting Science, University of Alberta, Edmonton, Canada 'Leiden Institute of Advanced Computer Science, Leiden University, Lei- den, Netherlands. Correspondence to: Arsalan Sharifnassab <ar- salan, sharifnassab @ @opamnndresaaacch.....
MetaOptimize is general and can wrap around any first- order optimization algorithm (the base update), such as SGD, RMSProp (Hinton, 2012), Adam (Kingma Ba,
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MetaOptimize
2014), Lion (Chen et al., 2023), while optimizing step sizes via separate first-order method (the meta update), such SGD, Adam, RMSProp, or Lion.
3. Forward and Backward Views
Because F depends on future losses, minimizing it causally requires an alternative view. Suppose hypothetically we had oracle access to future information (i.e., future loss values and weights). We could update
We develop approximation methods (Section 6) that, when incorporated into MetaOptimize, yield computa- tionally efficient algorithms outperforming sfate-of-the- art automatic hyperparameter optmiization methods on various stationary and continual (non-stationary) benchmarks (see Section 7).
d Be+1 = =,, -.,-10 d3 3, F?
(4)
d = -8-11 17 -17) Σx 7>6
fo (w,).
d
We show that some existing methods (like IDBD and its extensions, and hypergradient descent (Baydin et al., 2017)) are specific instances or approximations within the MetaOptimize framework (Section 5).
where 7) is meta step-size. This forward -iew update, however, is not causal because we do not have the required future information at time
To address this, we adopt an eligibility -racee--y-ree--yleee approach from reinforcement learning (Sutton, 1988; Sutton & Barto, 2018), introducing backward-vien view update:
2. Problem Setting
We introduce a general continual optimization setting that, for a given sequence of loss functions fe(.) : IR - IR, t = 0, 1, 2, aims to find a sequence of weight vectors W1, W22 ...... that minimize a discounted sum of future losses:
d KST d3, ft(w), (5)
Brt1 + -,-n(1-7)
Σ
r
F7 def (1
E f+(w),)
so that terms involving fr (and w,) appear at time T (instead of f), which is the earliest time that these quantities become available. In the small-n limit, the backward view closely approximates the forward view."
where y E [0, 1) is a fixed discount factor, typically close to 1, called the discount factor. For stationary supervised learning. ft are i.. samples from the same distribution, so loss.
Accordingly, we define causal gradient estimate
minimizing ! promotes rapid reduction of the expected
d
V F (1
Sr (www),
d 133
Consider an arbitrary first-order optimization algorithm (c.g., SGD, RMSProp, Adam, or Lion) for updating wt. At time t, it takes the gradient f. of the current loss, along with an m-dimensional meta meta-paaaaeeter vector Br to update ui and possibly some internal variables ã; (e.g.. momentum in Adam). Denoting 2, Stack(w,, iz) and calling this update rule Algaase we have
t=u
It follows from chain rule that
VAF , = -H V V. (w,),
(6)
where
dw,
H, (1
d
Ft+1 Algonse ( (.)
(2)
t=
Hence, H. encodes how past Gr values cumulatively affect w, under discounting by 7-
The goal of MetaOptimize is to determine a sequence Br such that plugging them into the above base update yields a trajectory } minimizing F?
4. MetaOptimize
Step-size adaptation is a natural special case: at each step the m-dimensional B, defines an n-dimensional step-size vector aa via some fixed function m + IR", i.e.,
Algorithm presents the general MetaOptimize frame- work for learning the meta-parameters Br. At each step
a₂ = (B,)
'Formally, assuming fe(.) = 0 for < 0 and for t > T, the final updates under (5) and (4) become arbitrarily close as (3)
A good choice for a(.) is exponential, ensuring a; is al- ways positive and making multiplicative changes in ar cor- respond to additive changes in B, (Sutton, 1992). By par- titioning the network weights into blocks, we can learn a shared scalar step-size per block, or even a unique step-size per weight, all handled automatically by MetaOptimize.
r) + 0. More specifically. (8)) Bf9)/n + 0, where 835 and S/ are the values of 3 at time T obtained from updates (5) and (4), respectively, starting from the same initial Ba. This is because
as 1) + 0. . remains almost constant over [0, 1"] interval, and the right hand sides of (5) and (4) match with accuracy Or7²).
when summed over [0, T]. See Section 9 for discussion more accurate approximations for large 17.
2

---

MetaOptimize
we replace the intractable gradient V P with the causal surrogate vgF, to ensure the update is feasible in real time, as discussed in Section 3. Specifically, we feed VgF, V f(((w)) (from (6)) into any first-order meta- update rule Algmeta just like conventional gradient.
Algorithm MetaOptimize Framework (for general
meta pueaa-paramtters)
Given: base-update Algnaee meta-update Alg gmeta and discount-factor y S 1. Initialize:
7
X 0(n+n)xm: Yo=
Qo 0
Formally, define yt de Stack (3,; 9r) as the stack of meta- parameters B3 and any internal states Ye of Algmeta (e.g.. momentum). The meta-update then:
0mxm
for 0, 2,,, do
x++1 Algaase (z), (
He first n rows of X
M1+1
Algmeta ye H. f.(w.)).
(8)
Vt+1 f Algmeta (ye, H V ft (we)).
Y++1
After applying the base update (2) produce #t+1, we compute H. V f((,) and plug it into (8) to update V1+1 (and thus 14+1). The remaining question is how to maintain H4, defined in (7), through application of the chain rule.
Update
X
from (14), using G, in (9).
Q1+1
end for
To compute Ht incrementally, let us stack the columns of the n X m matrix Hr into a single vector hr, and let
and then extract Hi by taking the top n rows of X, (since De SStak((w,,, (,,,,(,,,@)))))))) Rt)). The blocks of G, can be found for standard algorithms (SGD, Adam, Lion, etc.) as detailed in Appendix A. Notably, the first row of G. blocks depends only on Algmeta and the rest of G, blocks depend only on Algame Algorithm summarizes the procedure.
d 1
d
d yy
d
d/.
G2 def
d d dy,
d
d dhe
d
dhe41
dy.
dm
dh,
Applying the chain rule then implies
Remark 4.1. A distinction of MetaOptimize from exist- ing meta peramperr optimization methods is that it explic- itly captures dynamics of the meta- peeta-parametess B, and how changes in the current B3 affect ,3 in future. The term Y, in (11) links changes in past B, to future /3 values, which then influences H4. Intuitively, if G3 has been changing consis- tently in one direction (e-g., steadily increasing), amplifies Y, and thus H4, accelerating ongoing updates. Conversely, if B, stays nearly constant (indicating it may be close to optimal), Y, shrinks and do the subsequent updates to B,, stabilizing around the optimum.
ddy,
d
d
d G2
d
d,8,
ag,
1h.11
d
d,g,
[d3,]
which, when summed over T, turns into
dy a
dy
d 1 dB, d 1h d
a
G.
dæ
G.
d a
Jp.
T=0
dhi
7=0
db
a
(10)
5. Reducing Complexity
Defining
The matrix G, can be large and may involve Hessian terms, increasing the computational burden. We discuss two prac-
Ye del (1 1 7))
Σ
1-1
7/
1-T-1 dy
(11) tical approximations:
d33
7=0
1-1
dw
X, (1 7)
Σ
(12) 2x2 2 approximation. In (9), we zero out all blocks in the last row and column, effectively removing Qr- Em- pirically, this simplification often has negligible impact on performance. Intuitively, Hi does not affect the base up- date directly (d 2 /d h. = 0), so the extra blocks G, involving h, often have minor influence on the final meta- parameter trajectory.
d3,
7=0
t-1
d ht
Q (1 7)
(13)
d
0=0
and noting that y, = Stack ú.), one obtains a compact update of the form
Ye+1
Yt
X4+1
= G.
X;
+ +(1 - 7)
(14)
Q4+1
Q:
0
We go one step further, also zeroing out the block in the first row and second column of G.,
L-approximation.
0
3

---

MetaOptimize
Algorithm MetaOptimize with 2 x 2 approx..
6. Hessian-Free MetaOptimize
(Algtuse Stuse: Algmeu) (SGD, SGD), and scalar step-size Initialize: Ho = O Yo =
The G, matrix typically involves Hessian, V (( of the loss function, e.g., in the dw block where w w: Ox, V f(.w). Including second-order information in G, can be costly. Interestingly, for certain base and meta algorithms, we can eliminate the Hessian without much compromising the performance.
for t 1,2,.................. do
0r e
Base update:
1+++1 w² Ot V (w,)
Ho+1 7 [1 ar V f((w)) H₂ - Y ((())
Y41 = -YY (1 7) mH ((
For example, Lion (Chen et al., 2023) updates weights by taking the sign of the gradient (plus momentum). Since the derivative of the sign function zero almost everywhere, d dw4+1/d w, and related partials do not involve V f((w)) Hence, both base and meta updates use Lion, G. becomes Hessian-free throughout, avoiding second-order computa- tions entirely (Appendix A.1.3, A.3.2).
F For u-approximation let Yet1 =1
Meta update:
Be41 3r 1) V (w,,)
end for
Formally,
For other algorithms, we may consider their Hessian-free approximation by zeroing out any Hessian term in G The Hessian-free approximation turns out to be good approxi- mation, especially for base and meta algorithms that involve gradient normalization, like RMSProp and Adam. Note that, the sign function used in the Lion algorithm is an extreme form of normalization that divides a vector by its absolute value. We could instead use softer forms of normalization, such as normalizing to square root of a trace of squared vec- tor, vt, as in RMSProp. Such normalizations typically result in opposing Hessian-based terms in H.'s update (stem- ming from d diw+11 and d wre+ blocks of matrix Gt), aiming d.
d d
G def
dy.
(15)
d
d d
dy.
and the update in (14) simplifies to
1): 0
Y1+1 Xe+1
Ye X2
= G
7
+11 10--3 +11-7)
7)
0
(16)
This again discards Qt, but also certain cross cross-terms in Y''s update. Empirically, L-aaproximmation often matches the performance and sometimes improves the stability of the 2 x 2 approach.
d ue, to cancel out, particularly when consecutive gradients are positively correlated.
When Hessian terms are removed in the 2 x 2 approxi- mation, X, and Y, become diagonal or simply vectorized, drastically reducing matrix-multiplia muttiiiiulipiiiiiinnnn overhead. The overall complexity per step thus becomes similar to that of regular base and meta updates, requiring only few extra vector operations. Algorithm 3 in Appendix A illustrates these Hessian-free variants (SGDm, AdamW, Lion) under x 2 approximation.
Intuition. Algorithm illustrates the 2 approxima- tion for the case of SGD base/meta updates with single scalar step size. Observe how H., effectively accumulates (decayed) past gradients to decide whether to increase or decrease ot, If current and past gradients align, ae is raised for faster learning: if they oppose each other, a, shrinks. The decay 7(1 - |a] ft) of H. ensures that past gradi- ents poorly approximate future ones due to large 2 or a, their inffuence fades more rapidly. Meanwhile, Y, reflects how changing past (9 influences the current 3i large swings in 3 amplify H1+1, while near-constant B dampens updates. Under the L-approximation, Y, becomes constant in this particular setup, further simplifying the algorithm.
In summary, Hessian-free and 2 x or L-approximations yield a range of practical MetaOptimize instantiations that maintain strong performance at low additional cost.
7. Experiments
Containing some prior methods as special cases. Under L-approximation, and restricting both base and meta up- dates to plain SGD, MetaOptimize reduces to IDBD (Sutton, 1982) and its extension (Xu et al., 2018); see Appendix B.1 for derivations. Another notable special case is 7 0, which recovers the Hypergradienttddssee approach (Bay- din et al., 2017), updating step sizes to minimize the immedi- ate loss fe (w.) rather than the discounted sum F7, ignoring long-term effects of step size on future loss.
We evaluate MetaOptimize on image-classffiation and language-modeling benchmarks. Out of many possible base/meta-algorithm combinations and approximations (AI- gorithm 3), we showcase few Hessian-free variants that performed well in practice. In the experiments, MetaOpti- mize starts with step-sizes set one two orders of magni- tude below typical good fixed step-sizes, with no specific tuning. We compare MetaOptimize against some popular baselines whose meta parameersssssrss are well-tuned for each
4

---

MetaOptimize
task separately. See Appendix C for more details. Codes are available at https:/ggghhh..ccmm//aaaaaaallla
7.2. Non-stationary CIFARI00
We evaluated MetaOptimize in a non-stationary setting with 10 sequential tasks, each containing 10 classes from CI- FAR100. After training for one epoch on a task, it abruptly switches without explicit notification to the optimizer, and without resetting weights. We use a batch size of one, mean- ing cach data point is seen exactly once. The model is based on a simple CNN network consisting of two convolution (and max pooling) layers followed by a fully connected layer. Each curve is averaged over random seeds.
7.1. CIFARIO dataset
The first set of experiments involve training ResNet-18 with batch size of 100 the CIFARIO (Krizhevsky al., 2009) dataset. Fig. 1 depicts the learning curves of four combinations of (base, meta) algorithms for Hessian-free free MetaOptimize, along with the corresponding baselines with well-tuned fixed step sizes. Besides using a single scalar step-size, we also test a blockwise variant that partitions the ResNet18 parameters into six blocks (one for each linear layer and four blocks for the ResNet modules). In every tested combination, MetaOptimize outperforms its corre-
Average Cumulative Accuracy (Top 1)
40
38
sponding fixed-step step-size baseline.
36
(AdamW, Adam)
(Lion,Loo)
100
34
Train Accuracy (Top 1)
98
:22
AdamW, Fixed stepsire
96
MetaOptimize (AdamW, Adam). Scalar
MetaOptimize (AdamW,Adam) Blockiss
94
Fixed step size
Fixed step size
30
Scalar
Scalar
0
10000
20000
30000 40000 50000
92
Epoch
Blockwise
Blockwise
Figure 3. Learning curves for non-stationary CIFAR100.
90
(AMSProp. Adam)
(SCDm, Adam)
100
x10
x10
Thain Accuracy (Tbp 1)
10
20
98
Step-size of first block
96
Step-size second block >
16
94
Fixed step size
Fixed step size
12
Scalar Blockwise
Scalar
92
Blockwise
90
20
30
40
50
60
30
40
Epoch
Epoch
Figure 1. I.I. Learning curves for selected (hase, meta) combinations
in CIFARIO.
0
10000
20000
30000
40000
50000
1
Iteration
Figure 4. Blockwise stepsizes learned by MctaOptimize
Initial alpha 10e-6
(AdamW.Adam) on non-stationary CIFAR100. Note that the scale of the y-axis for the two curves differ by an order of magnitude. Step-sizes of both blocks are initialized at oxp = 10-4
8
Initial alpha 10e-7
Initial alpha - 10e-8
step size (alpha)
Initial alpha 10e-9
6
Figure presents cumulative top-1 accuracy averrged over all past training fires-for AdamW (with the best fixed step- size), MetaOptimize with a scalar step-size, and MetaOp- timize with blockwise step-sizes (two blocks: one for the first three layers and one for the last layer). MetaOptimize consistently outperforms the baseline. See Appendix D for additional plots.
0.00 0.25 0.50 0.75 1.00 1.25
1.50 1.75 2.00
Iteration
le4
Figure 2. Robustness to initial step-sizes, for (Lion, Lion) as (base, meta) update CIFARIO.
The learned step-sizes reveal an interesting pattern (Fig. 4):
Task adaptation: MetaOptimize increases step-sizes immediately after task switches to enhance adaptation.
Interestingly, as demonstrated in Fig. 2. the MetaOptimize algorithms show remarkable robustness to initial step-size choices, even for initial step sizes that are several orders of magnitude smaller than the optimal fixed siep-size.
Layer-wise behavior: In blockwise case, early-layer step-sizes decrease over time, indicating convergence
5

---

MetaOptimize
Algorithm 3 Hessian-free MetaOptimize algorithms with 2x2 approximation used in experiments Parameters: 0 (default 10 10-3), 3), y [0, (default = 1)
Initialize: ho = 0nx1.
for t
1,2.... do
0e = (3,)
exponential scalar/blockwier
M+++1 pm, (1 (1-p) f)
Base update
if Algrase is SGDm then
Aw 7, k
if Alghase is Lion then
Aw 0x, Sign (cm4 + (1 c) K0;
if Algease is A then
1e+1 Av (1 - ((w,))
Aw = m yu ka,w,
where pr = -V---x 1 (1-p))
Wi+1 = wt + Aw
h++1 = -1(1 ( h
Aw
h1 ((((w))
This for scalar step-sizes.
Meta update
for blockwise, should compute sum of h, ((W))) over each block.
M4+1 Mm; (1 p)
if Algmen is Lion then
B 1i-nSiggn (ēm, + (1 (1- 2)²)
if Algmeu is Adam then
1++1 = πū + (1 x)
B B m / 02
# where i== v1 X1(11 -7)
end for
to globally useful features, while last-layer step-sizes increase, reflecting the need to adapt to changing la- bels.
15M parameter model with batch size of 128 on the TinyS- tories dataset. Two combinations of Hessian-fiee MetaOp- timize with scalar step sizes were tested against Lion and Adam W with well-tuned fixed step sizes, Adam with a well-tuned cosine decay learning rate scheduler with Ik warmup iterations, and the four state-of-the-art step-size adaptation algorithms mentioned in the previous subsection. According to the learning curves, shown in Fig. 6, MetaOp- timize outperforms all baselines (with an initial delay due to small initial step-sizes), except for the well-tuned learning rate scheduler within 30k iterations.
7.3. ImageNet dataset
We trained ResNet-18 with batch-size 256 on ImageNet (Deng et al., 2009). We compared MetaOptimize with scalar step-size against four state-of-the-a art hyperparamter optimization algorithms, namely DoG (Ivgi et al., 2023), gdtuo (Chandra et al., 2022), Prodigy (Mishchenko & De- fazio, 2023), and mechanic (Cutkosky et al., 2024), as well as AdamW and Lion baselines with fixed step-sizes, and AdamW W with a well-tuned cosine decay learning rate sched- uler with 10k iterations warmup. Learning curves and complexity overheads shown respectively in Fig. 5 and Table 1, showcasing the advantage of MetaOptimize algo- rithms (learning curve of DoG is not depicted due to its relatively poor performance). Unlike CIFAR1O, here the blockwise versions of MetaOptimize showed no improve- ment over the scalar versions. Refer to Appendix D for further details.
7.5. Sensitivity analysis
Here, we briefly discussion the sensitivity of MetaOptimize to meta-mett-parameterr
For the meta-stepsize r) MetaOptimize, there is generally need for tuning, and the default value 7 10 works universally well in stationary supervised learning. All exper- iments in this section used this default value with no sweep- ing required. The rationale for this choice is that when using Adam, Lion, or RMSProp for meta- veta--pdates, the absolute change in 8 per iteration is approximately r) x O(1) 10- Unless the current stepsize a already near its optimal value, most 3 updates will consistently move toward the op- timal B. Within 1,000 steps. 3 can change by ((),, nearly doubling or halving a = exp(3). Over 10,000 iterations, can adjust to stepsizes that are e > 20, 000 times larger or smaller, allowing 1) = 10-3 3 to efficiently track optimal stepsizes while minimizing unnecessary fluctuations in a.
7.4. Language modeling
For language model experiments, we used the TinyStories dataset (Eldan & Li, 2023), a synthetic collection of brief stories designed for children aged 3 4. This dataset proves effective for training and evaluating language models that are significantly smaller than the current state-of-the-arn and capable of crafting stories that are not only fluent and coherent but also diverse.
Regarding the discount factor 7 we used the ? = 1 in all stationary experiments and observed minimal sensitivity
We used the implementation in (Karpathy, 2024) for training
6

---

MetaOptimize
Table Per-iteration wall-clock-time and GPU-memory overhead (compared AdamW)
Algorithm
ImageNet
TinyStories
Time
Space
Time
Space
AdamW W (fixed stepsize)
0%
0%
0%
0%
DoG (Ivgi et al., 2023)
+45%
1.4%
+268%
0%
gdtuo (Chandra et al., 2022)
+85%
64%
+150%
21%
mechanic (Cutkosky et al., 2024)
+42%
88%
+9%
0%
Prodigy (Mishchenko & Defazio, 2023)
-2%
13%
+9%
0%
MetaOptimize (Adam Lion)
44%
33%
+13%
0%
ing rates via adaptations of classical line search (Rolinek & Martius, 2018; Vaswani et al., 2019; Paquette & Schein- berg, 2020; Kunstner et al., 2023) and Polyak step size (Berrada et al., 2020; Loizou et al., 2021), stochastic prox- imal methods (Asi & Duchi, 2019), stochastic quadratic approximation (Schaul et al., 2013), hyper-gradient descent (Baydin et al., 2017), nested hyper-gradient descent (Chan- dra et al., 2022), distance solution adaptation (Ivgi et al., 2023; Defazio & Mishchenko, 2023; Mishchenko & De- fazio, 2023), and online convex learning (Cutkosky et al., 2024). A limitation of most of these methods is their po- tential underperformance when their meta- meta--arameters are not optimally configured for specific problems (Ivgi et al., 2023). Moreover, the primary focus of most of these meth- ods is minimizing immediate loss rather than considering the long-term effects of step sizes future loss.
85
Train Accuracy (Top 5)
AdamW, LR scheduler
AdamW, Fixed stepsize
75
Lion, Fixed stepsize
MetaOptimize (AdamW, Lion), Scalar
MetaOptimize Lion), Scalar
MetaOptimize (SGDm, Lion)
70
gdtuo
mechanic
Prodigy
20
40
60
80
Epoch
Figure ImageNet learning curves.
2.2
AdamW, LR scheduler
AdamW, Fixed stepsize
Normalization techniques proposed over past few years, such as AdaGrad (Duchi et al., 201 I), RMSProp, and Adam have significantly enhanced the training process. While these algorithms show promise the stationary problems, these normalization techniques do not optimize effective step sizes and are prone have sub-optimal performance especially in the continual learning settings (Degris et al., 2024).
2.0
Lion, Fixed stepsize
MetaOptimize (AdamW, Lion)
MetaOptimize (Lion, Lion)
1.8
DoG
Train Loss
gdtuo
mechanic
1.6
Prodigy
1.4
An early practical step-size optimization method was the Incremental-Dede Deltt--ar--Deltt Bar-Delta (IDBD) algorithm, introduced in (Sutton, 1992), which aimed to optimize the step-size vec- minimize specific form of quadratic loss functions in continual setting. This al algorithm for neural networks in (Xu et al., 2018; Donini et al., 2019). and further adapted in (Mahmood et al., 2012; Javed, 2020; Micaelli & Storkey, 2021) for different meta or base up- dates beyond SGD. However, the development IDBD and its extensions included some implicit assumptions, notably overlooking the impact step-size dynamics on the formu- lation of step-size update rules. These extensions are, in essence, special cases of the L-approximation within the MetaOptimize framework. The current work extends the IDBD research, significantly broadening the framework and establishing a solid basis for the derivations. IDBD and its extensions have been used in various machine learning tasks including independent component analysis (Schraudolph
1.2
5000
10000
15000
20000
25000
30000
Iteration
later extended
Figure 6. TinyStories (language model) learning curves.
for values 7 2 0.999 in a series of preliminary tests. How- ever, performance begins to degrade with smaller values of y. the non-stationary CIFARIO0, 7 = 0.999 performed slightly better than
8. Related Works
Automatic adaptation of step sizes, has been an important research topic in the literature of stochastic optimization. Several works aimed remove the manual tuning of learn-
7

---

MetaOptimize
Giannakopoulos, 1999), human motion tracking (Kehl & Van Gool, 2006), classification (Koop, 2007), and reinforce- ment learning (Xu et al., 2018; Young et al., 2018; Javed et al., 2024). Refer to (Sutton, 2022) for a comprehensive history of step-size optimization.
Hessian: We confined our experiments Hessian-free methods for practicality, though Hessian-based algorithms could offer superior performance. These methods, however, face challenges requiring additional research. The Hessian matrix is notably noisy, impacting He+1 multiplicatively, necessitating smoothing and clipping techniques. Addition- ally, the Hessian approximates the loss landscape's 's curva- ture but fails to account for non-differentiable curvatures, such as those from ReLU unit breakpoints, significant training's s end. From computational perspective, devel- oping low-complexity methods for approximate Hessian matrix products, especially for adjusting step-sizes at the layer and weight levels, is essential.
Hypergradient Descent (HD) (Baydin et al., 2017) adapts learning rates using immediate loss gradients. MADA (Ozkara al., 2024) extends HD by parameterizing a space of optimizers and navigating it via hypergradient descent. Both focus on short-term effects, whereas MetaOptimize in- troduces a discount factor to model long-term influences, generalizing HD as a special case when y = O.
related line of work gradient- based bilevel optimization, initially introduced by (Bengio, 2000) and later expanded in (Maclaurin et al., 2015; Pedregosa, 2016; Franceschi al., 2018; Gao et al., 2022). Recent advances, such as (Lor- raine et al., 2020), enable the optimization of millions of hyperparameters. While bilevel optimization focuses on tuning hyperparameters to minimize validation loss through repeated full training runs of the base algorithm, MetaOpti- mize diverges significantly. Designed for continual learning, MetaOptimize optimizes meta-parameters on-the-fly during single streaming run, without relying on validation loss. Instead, it minimizes online loss (or regret) directly, aligning with the continual learning framework where no validation or test sets exist, and data arrives sequentially.
More accurate traces: As discussed in Section 3, accuracy of the backward approximation (5) may degrade for larger values of the meta-stepsize n). Eligibilitt traces in RL suffer from similar problem, to resolve which more-sophisticated traces (e.g., Dutch traces) have been developed (see Chapter 11 of (Sutton & Barto, 2018)). Developing more accurate backward approximations for meta- pertamerreeerr optimization can result in considerable improvements in performance and stability.
While step sizes can vary much in granularity, our experiments focused on scalar and block- wise step-sizes. While increasing the number of step sizes is anticipated to enhance performance, our experimental findings in Section 7 reveal that this improvement is not con- sistent across the MetaOptimize approximations evaluated. Further investigation needed in future research.
Blockwise step-sizes:
Another relevant literature is learn to optimize (L20), which aim to learn optimization strategies from data. Classical 1.20 methods such as (Andrychowicz et al., 2016) train op- timizers offline and deploy them unchanged. Later works, including (Metz et al., 2020; 2022), develop more effective or scalable architectures, often using neural networks to modulate optimizer behavior. While powerful, these meth- ods typically lack the ability to adapt online. In contrast, MetaOptimize updates its meta- parammersssssss continuously during training, which is advantageous in nonstationary or continual learning scenarios.
Other approximations: We explored limited set of MetaOptimize's possible approximations, leaving com- prehensive analysis of various approximations for future research.
Other meta- peta-parameters: Our study was limited to differen- tiable meta- paramaaaaetters, not covering discrete ones like batch size network layer count. We also did not investigate several significant differentiable meta-paarseetess beyond step-sizes, deferring such exploration to future work.
There is also line of research on the so-called parameter- free optimization that aims to remove the need for step-size tuning with almost no knowledge of the problem properties. Most of these methods are primarily designed for stochastic convex optimization (Luo & Schapire. 2015; Orabona & Pal, 2016), while more recent ones (Orabona & Tommasi, 2017; Ivgi et al., 2023) were applied to supervised learning tasks with small or moderate sample sizes.
Automatic Differentiation: While certain versions of MetaOptimize, such as the L-Approximation, could be im- plemented using standard automatic differentiation software, its applicability to the general case of MetaOptimize remains unclear. Unlike updates for and /3 (base and meta param- eters), the H matrix lacks an explicit incremental formula that can be easily handled by automatic differentiation. For some versions of MetaOptimize, including the Hessian-free approximations used in our experiments, automatic differen- tiation is unnecessary, as meta updates do not require addi- tional differentiation. Exploring the scope and applicability of automatic differentiation across different MetaOptimize instances is an interesting direction for future research.
9. Limitations and Future Works
Our work represents step toward unlocking the potential of meta-parameter optimization, with substantial room for further exploration, some of which we outline here:
8

---

MetaOptimize
Discount factor 7 1: Our backward formulation (Eq. (14)) formally assumes y 1 due to normaliza- tion factor used in the definition of the surrogate gradi- ent. For y 1, a simple workaround is to remove this scaling factor, which makes the derivation fully valid and ceeiiseent-maachigg our actual implementation and exper- iments. That said, the case y = 1 presents subtle theoretical differences, much like in reinforcement learning and dy- namic programming where additional centering is often helpful. Adapting similar techniques for meta-optimization may yield benefits and is a promising direction for future work.
Advances in Neural Information Processing Systems, 35:
8214 8225, 2022.
X Chen, C Liang, D Huang, E Real, K Wang, Y Liu, H Pham, X Dong, Luong, CJ Hsich, et al. Symbolic discovery of optimization algorithms. arxiv 2023. arXiv preprint arXiv:2302.06675, 2023.
Ashok Cutkosky, Aaron Defazio, and Harsh Mehta. Me- chanic: A learning rate tuner. Advances in Neural Infor- mation Processing Systems, 36, 2024.
George E Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, et al. Benchmarking neural network training algorithms. arXiv preprint arXiv:2306.07173 2023.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3366- 3385, 2021.
References
Ehsan Amid, Rohan Anil, Christopher Fifty, and Manfred K Warmuth. Step-size adaptation using exponentiated gra- dient updates. arXiv preprint arXiv:2202.00145, 2022.
Aaron Defazio and Konstantin Mishchenko. Learning-rate- free learning by d-adaptation. In International Confer- ence on Machine Learning, pp. 7449-7479 7479. 2023.
Marcin Andrychowicz. Misha Denil, Sergio Gomez, Matthew W Hoffmann, David Pfau, Tom Schaul, Bren- dan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
Thomas Degris, Khurram Javed, Arsalan Sharifnassab, Yuxin Liu, and Richard Sutton. Step-size optimization for continual learning. arXiv preprint arXiv:2401.17401 2024.
Anonymous.
Metaoptimize framework. 4open. science/z/
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. IEEE, 2009, doi: 10.1 (L00///CVP.22099555066
Metapptimmze-25000 2024.
Hilal Asi and John C Duchi. The importance of better models stochastic optimization. Proceedings of the National Academy of Sciences, 116446)22924-22930, 2019.
Michele Donini, Luca Franceschi, Massimiliano Pontil, Or- chid Majumder, and Paolo Frasconi. Marthe: Scheduling the learning rate via online hypergradients. arXiv preprint arXiv: 190.0855 2019.
Atilim Gunes Baydin, Robert Cornish, David Martinez Ru- bio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. arXiv preprint arXiv: 1703.07703.4488... 2017.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive sub- gradient methods for online learning and stochastic opti- mization. Journal of Machine Learning Research, 12(7), 2011.
Yoshua Bengio. tradient-based optimization of hyperpa- rameters. Neural Computation, 12(8): 1889-1900 2000. doi: 10. 10.116208999766000000155551
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint
Leonard Berrada, Andrew Zisserman, and M Pawan Kumar. Training neural networks for and by interpolation. In International Conference on Machine Learning, pp. 799- 809. PMLR, 2020.
2023.
Luca Franceschi, Michele Donini, Paolo Frasconi, and Mas- similiano Pontil. Bilevel programming for hyperparam- eter optimization and meta-learning. In International Conference on Machine Learning, volume 80, pp. 1568- 1577, 2018
Kartik Chandra, Audrey Xie, Jonathan Ragan-Kelleg and Erik Meijer. Gradient descent: The ultimate optimizer.

---

MetaOptimize
Boyan Gao, Henry Gouk, Hae Beom Lee, and Timothy M Hospedales. Meta mirror descent: Optimiser learning for fast convergence. arXiv preprint arXiv:2203.02711, 2022.
Frederik Kunstner, Victor S Portella, Mark Schmidt, and Nick Harvey. Searching for optimal per-coordinate step- sizes with multidimensional backtracking. arXiv preprint (rxi::230.00252., 2023.
Geoffrey Hinton. Neural networks for machine learning, lecture 6.5 rmsprop, 2012. URL https: /wwww cs www.:ccc.tooooooo co..trrono..dd/ttt....... imen/ csc321/slides Lecture slides slides_lec6.pon pdf. Coursera Lecture.
Nicolas Loizou, Sharan Vaswani, Issam Hadj Laradji, and Simon Lacoste-Julient Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence. In International Conference Artificial Intelligence and Statistics, pp. 1306-1314. PMLR, 2021.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint 1801.06 2018.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Opti- mizing millions of hyperparameters by implicit differen- tiation. Proceedings of the 23rd International Confer- ence on Artificial Intelligence and Statistics, volume 108, pp. 1540-1552. PMLR, 2020.
Maor Ivgi, Oliver Hinder, and Yair Carmon. DoG is SGD's best friend: A parameter-free dynamic step size schedule. In International Conference on Machine Learning, pp- 14465-14499, PMLR, 2023.
Haipeng Luo and Robert E Schapire. Achieving all with no parameters: Adanormaltedge. In Conference on Learn- ing Theory, pp. 1286-130K 1304. PMLR, 2015.
Robert Jacobs. Increased rates of convergence through learning rate adaptation. Neural networks, 1(4):295-307, 1988.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- hyperparameter optimization through re- versible learning. In International Conference on Ma- chine Learning, pp. 2113-2122. PMLR, 2015.
Khurram Javed. Step-size adaptation for mmsrrpp. Tech- nical Report, 2020. com/ com/ hemprtepott///// dreepots/idbd__oe iiidddddrrrrrpppooee pdf.
Ashique Rupam Mahmood, Richard S Sutton, Thomas De- gris, and Patrick M Pilarski. Tuning-free step-size adap- tation. International Conference on Acoustics, Speech and Signal Processing, pp. 2121-2124. IEEE, 2012.
Khurram Javed, Arsalan Sharifnassab, and Richard S Sutton. Swifttd: A fast and robust algorithm for temporal differ- ence learning. In Reinfocement Learning Conference, 2024.
Luke Metz, Niru Maheswaranathan, C Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. Tasks, stabil- ity, architecture, and compute: Training more effec- tive learned optimizers, and using them to train them- selves. arXiv preprint arXiv:2009.11243 2020. URL https://azxiv. wttp:/iww..r//aa///0...... attes///azaav.oorr//////00..... 112433
Honghe Jin. Hyperparameter importance for machine learn- ing algorithms. arXiv preprint arXiv:2201.05/322 2022.
Andrej Karpathy. llama2.c: Inference Ilama 2 in one file of pure 2024. URL /tctps://githhn com/
Sarpathy/1lamas 2ma.. c. GitHub repository.
Luke Metz, C Daniel Freeman, James Harrison, Niru Ma- heswaranathan, and Jascha Sohl-Dickstein. Practical tradeoff's between memory, compute, and performance in learned optimizers. In Conference on Lifelong Learning Agents, pp. 142-164, 2022. URL https: arrxiv. o org/ abs / 2203. 11860.
Roland Kehl and Lue Van Gool. Markerless tracking of com- plex human motions from multiple views. Computer Vi- sion and Image Understanding. 104(2-3):900-09, 2006.
Harry Kesten. Accelerated stochastic approximation. The
Annals of Mathematical Statistics, pp. 41-59, 1958.
Paul Micaelli and Amos J Storkey. Gradient-based hyper- parameter optimization over long horizons. In Advances in Neural Information Processing Systems, pp. 10798 10809, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 4r2.v:1412.6980, 2014.
Konstantin Mishchenko and Aaron Defazio. Prodigy: An expeditiously adaptive parameter-free Iearner. arXiv preprint vrXiv:2306.06100 2023.
A Koop. Investigating Experience: Temporal Coherence and Empirical Knowledge Representation. University of Alberta MSc. PhD thesis, thesis, 2007.
Francesco Orabona and Dávid Pál. Coin betting and parameter-fiee online learning. In Advances in Neural Information Processing Systems, volume 29, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
10

---

MetaOptimize
Francesco Orabona and Tatiana Tommasi. Training deep networks without learning rates through coin betting. In Advances in Neural Information Processing Systems, vol-
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Sharan Vaswani, Aaron Mishkin, Issam Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien, Pain- less stochastic gradient: Interpolation, line-search, and convergence rates. Advances in Neural Information Pro- cessing Systems, 2019.
ume 30, 2017.
Kaan Ozkara, Can Karakus, Parameswaran Raman, Mingyi Hong. Shoham Sabach, Branislav Kveton, and Volkan Cevher. Mada: Meta-adaptive optimizers through hyper- gradient descent. arXiv preprint www.i::20000088999 2024.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta- gradient reinforcement learning. In Advances in neural Information Processing Systems, 2018.
/0/0 08893.
Courtney Paquette and Katya Scheinberg. A stochastic line search method with expected complexity analysis. SIAM Journal on Optimization, 30(1):349-376) 2020.
Kenny Young, Baoxiang Wang, and Matthew Taylor. Metatrace: Online step-size tuning by meta-gradient de- scent for reinforcement learning control. arXiv preprint arXiv:1805.04511 2018.
Fabian Pedregosa. Hyperparameter optimization with ap- proximate gradient. International Conference on Ma- chine Learning, pp. 737-746, PMLR, 2016.
Michal Rolinek and Georg Martius. L4: Practical loss- based stepsize adaptation for deep learning. In Advances in Neural Information Processing Systems, 2018.
Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. In International conference on machine learning, pp. 343-351. PMLR, 2013.
Nicol Schraudolph and Xavier Giannakopoulos. Online independent component analysis with local learning rate adaptation. Advances in neural information processing systems, 12, 1999.
Bharat Singh, Soham De, Yangmuzi Zhang, Thomas Gold- stein, and Gavin Taylor. Layer-specific adaptive learning rates for deep networks. In International Conference on Machine Learning and Applications, pp. 364-368. IEEE, 2015.
Richard S. Sutton. Adaptation of learning rate parameters. Wright-Patterson Air Force Base, Ohio, 1981. Technical Report SFWAL-TR-81-1077
Richard S Sutton. A theory of salience change dependent on the relationship between discrepancies on successive trials on which the stimulus is present. Unpublished working paper, 1982.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3:9-44, 1988.
Richard S Sutton. Adapting bias by gradient descent: An in- cremental version of delta-bar-delta, In AAAI, volume 92, pp. 171-176. San Jose, CA, 1992.
Richard Sutton. A history of meta-gradient: Gra- dient methods for meta-learning arXiv preprint arXiv:2202.09701, 2022.

---

MetaOptimize
Appendices
A. Step-size Optimization for Different Choices of Base and Meta updates
In this appendix, we derive G, defined in (9) for different choices of algorithms for base and meta updates, and propose corresponding step-size optimization algorithms.
Consider the following partitions of Gt,
d
d
(17)
y.
des
dh
d #0++1
dm
d
Gaas def
1/+
d
dhs
(18)
d 11
d
d
d
d/
Then,
d MM+1
d
dy
des
dl
Ganeta
G.
dæ
d
d
(19)
dy:
daz
dh
Couse
dhet1
dhes
d
dy.
d
d
In the sequel, we study base and meta updates separately, because Algnuse and Algmeta impact disjoint sets of blocks in G.. In particular, as we will see, the choice of Algane only affects Ghase while the choice of Algmeta only affects Gmmeta
Notation conventions in all Appendices: For any vector u, denote by [e) diagonal matrix with diagonal entries derived from We denote by (B,) the Jacobian of a with respect to Br.
Before delving into computing Ghase and G. for different base and meta algorithms, we further simplify these matrices.
A.1. Derivation of Gueta for Different Meta Updates
We start by simplifying Grmeaa and introducing some notations.
Note that the meta update has no dependence on internal variables, ã, of the base algorithm. As a result,
d dyt+1 0.
(20)
d dæ;
Then,
G mmeta
d
d
d
d3e4s dy.
dvee1 dws.
diee d.c.
dye+1 dh.
d Me+
d wa
d
(21)
dhk
yr
dh.
where the third equality is due (20). Let
v f((w))"
0
0
0
f((w.,)
0
L
(22)
0
0
0
0
0
Ve((w,))
m
and recall that h; is a vectorization of Hg. Then,
H4 V f(we) = L,h4-
(23)
We now proceed to derivation of metta for different choices of Algmeta
12

---

MetaOptimize
A. 1.1. Meta SGD
Here, we consider SGD for the meta update (8),
B4+1 B. -
(24)
where 7) is a scalar, called the meta step size. In this case, yr - B, It then follows from (24) that
d dB+11 h4
d (Hl (((wi) -1 -1 dh h
hh -" (Lete)) -1/4,
(25)
where the second equality is due to (23). Consequently, from (21), we obtain
d
d d
0
yets
dy.
d ww
d he
d d,,
d
d
(26)
d
dh.
nH V f((e) 0 -Tp
where the last equality follows from (25) and simple differentiations of (24). Here, V2 denotes the Hessian of St t.+-
A. 1.2. Meta Adam
The meta update based on the Adam algorithm is as follows,
mc+1 pm; + H. V fi(w:))
8141 = Xv, + (HL V f((w)))
14 1 (1-0) V
(27)
mi
Be11 Be n jtt
where m, is the momentum vector, is the trace of squared mrrrogate-mett-rrddien Since Adam algorithm needs keep track of B,, my, and i2 we have
B
V.
mi;
(28)
0;
Recall the following notation convention the end of the Introduction section: for any & 2 1, and any k-dimensional vector [tr...., U&J, we denote the the corresponding diagonal matrix by [w]:
0
[0]
(29)
13

---

MetaOptimize
Consequently, from (21), obtain
G
d
d d
d
d/
d
d
d
d d
d
d
d
d
d d
d
dr
dm
dm
dmail
0
dp.
d
dw
d
diit+11
d
d
d
(30)
a,;
d dim
dw
dh
I
my
0x
0
pl
0
H. V
0
dmiil
XT
2
V
0
d de+1
d
where the last equality follows by calculating derivatives of (27). For the two remaining terms the last column of G,, we have
d m++ = d h dh:
d
d
(H) ())w)) =1 a'dh h
(Iehe) T4-
(31)
where the first equality follows from the update of m441 (27), and the second equality is due to (23). In the same vein,
d d0+11 d V((w))) d h4
d
2( d dhe (Lihe) 2[L, L, 2 [[((w))] Le, (32)
a h (L,ti)²
d dh,
where the first equality follows from the update of uc+1 in (27), the second equality is due (23), and the last equality is again from (23).
Plugging (31) and (32) into (30), we obtain
I
7i
0
6
Gmeta
pl
O
H
0
17 Lr
(33)
0
XI
2 V
0
2
2(H
Vf
L2
A. 1.3. Meta Lion
The meta update based on the lion algorithm is as follows
M++1 pm, + (1 p) Vol.
(34)
B = 3, Sign (cm, + (1 (1-c)V+F.).
(35)
where 17 is scalar, called the meta step size, and p. [0, 1). Note that the meta algorithm operates low dimensional space. Therefore, we drop the regularizers like weight-decay in the meta updates, they are primarily aimed to resolve the overfitting problem in high dimensional problems. Substituting V&P, with H.V f((e) we obtain the following meta updates
m++1 = pm- + (1- p) (w:)),
(36)
B = B, - Sign (cm;: + (1 - (.(()))
(37)
In this case,
B
mi
14

---

MetaOptimize
and
G
d
dy--
0
dy++1 dh
d
dm; d d
d d
d 1
d
di
d
m1111
0
mi
(38)
d
d mi;
dim
he
I
0
0
0
0
1 d Ma
0
1
d
dmm.
dh
where the last equality follows from (37). Consider the following block representation Y;:
B, Y
Y,
(39)
Since the base algorithm, does not take mn as input, will see in (41) and (42) of next subsection (Appendix A.2), d the only non-zero block of G in its column of blocks (i.e., dssil "im. 0 for every variable other than m²) Consequently, it follows from (14) that y defined in (39), has no impact the update of X B and Q++1. Therefore, can zero-out the rows and columns of Geeta that correspond to derivative of m. As such obtain the following equivalent of Gmeta in (38) from an algorithmic perspective:
mmxm 0
G
(40)
0
0
As result, get B, for all times t.
A.2. Derivation of Cusee for Different Base Updates
We now turn our focus to computation of Chase Let us start by simplifying Ghase and introducing some notations.
Note that the base update has no dependence internal variables, ÿ. of the meta update. As a result,
d ax+11 y,
0.
(41)
Moreover, it follows from the definition of 74 in (7) that
d d = -11 (1-7) Σ d (dwr-1 (1 - Σ t-r d (dwr+1) ) (1 7) Σ 7 t-r d
(0) 0,
dÿ.
dÿ, d
dB,
d
dB
0=0
1=0
1=0
where the third equality follows from (41). Therefore,
d h 0. y
(42)
Note also that Algase does not take H6, as input, and therefore,
d 1x+1 O.
(43)
d h
Consequently, we can simplify Ghase as follows,
d
d d
d0re1 dh,
d 1 d
dæ
d
d
dæ
bbsse
d dyy
d
dig dig. de:
db;
dg.
da
(44)
d hha1
dbea dpp
dh
d
0
d
d
7
dy.
dm
d
d'g.
dam
d
dpp
d
d
15

---

MetaOptimize
where the last equality is due to (41), (42), and (43). On independent note, consider the following block representation of Y,,
B: 1-71 I
(45)
Yt
Therefore,
?it + (1 - [
[
B. %x
=
then follows from (19) and (14) that
B
Y1
Xe+1 Q++1
= = G
Chuse
(46)
X
Qt
Moreover, from the definition of Y in (11), we have
d (1 - 7) d
d d3,
(d
d
(1 7)
Σ
T-
(1 - Σ 7
dB, (0) =0,
dæ;
d3,
d
dx
==0
7=0
T=0
d dB,
(1 - BB 1=0
dB
(1 7)
d
- (1 -1) Σ 7
d
(7) 0, (47)
dB
dB
d3
a,,
d3
T=0
T=0
d B,
d
d
= (1-7) d dh
(1 7)
--T d
(1
(0) = 0.
dh:
d
dh.
d3
==0
==0
T=0
Finally, recall the definition
o(,,)) def d da
(48)
d 3
the Jacobian of az with respect By-
We now proceed to derivation of husee for different choices of Algbase
A.3. Base SGD
Base SGD algorithm makes the following base update in each iteration:
Wr+1 w. at fw..).
(49)
In this case, x. = we and X, He. Then, Gease in (44) can be simplified to
d d,P. d dhat d.g.
d Amse1 me d hees d ai Awerl daw
Chase
dhes d
dw dp, heus dB,
(50)
d d dh.
dw.
[... (B,)
- [ax]] 2 ((w))
he-1
d dhit d mrc
d ha dhs
dB,
where the last equality follows by computing simple derivatives of w 1 in (49).
16

---

MetaOptimize
We proceed compute the three remaining entries of Gt i.e., M 3.. h141/d we and d h441/d / hg. Note that by plugging the first row of Gease given in (50), into (46), and noting that He X we obtain
He+1 7 [ V (r(w:))
[V o' (3,,) B.,
(51)
for all 2 0. By vectorizing both sides of (51) we obtain
(1 - [ [c H,
[VA]]] (33,, B. [V (33,) B,
(1 - [ V2 ft) Hg
h+41
(52)
[ar] v H mx
[i] o' o3(), Bi
Note that for any pair of same-size vectors a and b, we have [a] [b] a where [a] and [b] are diagonal matrices of a and respectively. Therefore, (52) can be equivalently written in the following form
I - (ard 5a H (3,, B
V
h141 =
(53)
(T [oxg] V2 mm]l - [o,, mm 4
By taking the derivative of (52) with respect to he, we obtain
I - [oc]] 2 (2s((w))
0
1st
[xi]
V f(w)
2nd
dhi+1 d dh,
7
(54)
0
[os]]
V ((((()))))
mtth
In the above equation, note that d B,/d h; = due to (47). Let [.[x]] and we]i] denote the ith and jth entries Br and wis for 1,... m and j 1...., n, respectively. then follows from (52) and (47) that
da 1
ft
H
[a]
B(a
dos
H 11
+
[.
(p'
B
J..
d
Jpp.m
d dh dB,
(55)
doy d&
2
fz H4 m)]
+
[.
(0'(3,)
[m]
da: [p.[m)
V fe H [m)
00'(9,) [.
B!n]]
(
9 stands for the entry-wise partial derivative of matrix with respect scalar variable B. In the same vein, (53) where 23 and (47) imply that
[ar]]
d (V2 (10. H4(A))
+ [o((3,) B! V fww,)
dw,,
d h
(56)
d ww
[oc]]
d (.V"),(w.) mm
[o,(8,) m!!l V2 f((ns)
w.
Finally, Ghase obtained by plugging (54), (55), and (56) into (50).
17

---

MetaOptimize
In the special that [3 is scalar (equivalently m 1), and furthermore = = e matrix G would be simplified to
1j h V2 f((ww)
G base (scalar)
a (V)((
a --
0
((m
V - ((a.)
-yaa
dwe:
Br a ((
7(1 - a f((i))
A.3.1. Base W
The base update according to the AdamW algorithm (Loizou et al., 2021) is as follows,
M441 = p + f((ww),
1i+1 A Y + ff((w;)
(57)
#4 =
X
mc /o:
W1+1 = Wt ai14
ko;
where m. the momentum vector, v, is the trace of gradient square used for normalization, and x is weight-decay parameter. Therefore the base algorithm needs keep track of wt my, vt, i.e.,
Wt
3t
171:
(58)
0.
It then follows from (44) that
d dmee4
d 2:41
0
Ghase
d
d
dbr11 d.B.
d hett
de
d d ww d dmiil d'w dertl daa.
d dmm
d 0 d dm 0 de:
d B mi dB.. delil d
dmeil
0
deeil O
dmm.
de.
(59)
d 1hri d
d het1 d wt
d
hee-1
d'm
du
ah;
m.
+ kaw
(B,)
0
I -
-44
aam. w
0
0
pl
0
2
XI
hh+1
dha+ dm
d d
d d
0
d&g.
d urz
d
where the last equality follows from simple derivative computations in (57).
We proceed to compute the terms the last row of the G Crass above. Consider the following block representation of X,,
H
X,
X
(60)
X
Plugging the first row of Ghme, given in (59), into (46), implies that
m. + kw d' + 7(1 [as)
a4 V.
a,
Hot1 -YPt
He
7p4
x
+
xy.
(61)
2
v.
18

---

MetaOptimize
for all / 2 O. Note that for any pair of same-size vectors and b, have [a] b [b] where [a] and [6] are diagonal
matrices of and b, respectively. Therefore, the ith column in the matrix equation (61) be equivalently written
Hdl -- -7 (33,)
m vt
www; + r(T - xi [ous) H TMpe X²
a Vor
+
[d]
a,
(62)
2
where B H64h X
[s] and X, stand for the ith columns of B,, He, X and X², respectively. Following similar arguments as in (47), it is easy to show that
d d dx
=
0,
dB,
d3;
d.x"" JXp 0,
dw; 10
d =
ZXpp 0,
(63)
dm
177
d X
3X O,
d d
4a
dx;"".
x
0.
dh
h
Note that h2 is an nm-dimensioonll vector derived from stacking the columns of H4. Therefore, we consider block representation of h, consisting of blocks, each of which corresponds to column of H. By taking the derivative of (61) with respect to he, and using (63), we obtain
xi [ay]]
0
0
0
+- 1st
ki [o:]]
0
0
2nd
dh dh;
(64)
0 0
0
0
0
0
k [ac]
with
Let [.[d] and we]l]] denote the ith and jth entries of B, and wy, for i 1,..., and j 1, ...... n, respectively. Note that d h.f41/d /d B, is a block matrix, in the form of an m x m array of n. x blocks, then follows from (61) and (63) that, for i,j 1,.... m,
d d,p. [t, det xdd
for i, = 1,..., m.
a0.
d [i
d !
d d33
[
(8,)(B,)
d d 1
7144
+ kw.
B
(65)
/0, 1
7PC
Vv,
d d.
d da
X
[
P4
2
m
155
d 1.
d d
x [
where 03
& stands for the entry-wise partial derivative of matrix with respect to scalar variable B.
In the same vein, it follows from (62) and (63) that
BB,))B [1]
d h 111
(66)
d dw:
④ d(
19

---

MetaOptimize
Xx
(8,11
2
v
dh
7P4
(67)
dm;
X, [ms]
20:
v0r
(o'( mr + Or, X
m0 [9]
2
dhi+1 d
7/44
(68)
(o.( ((").)
m: + Oz X, [m]
30., m, X (m)
Finally, GA is obtained by plugging (64), (65), (66), (67), and (68) into (59).
A.3.2. Base Lion
The lion algorithm, when used for base update, is as follows
M&++1 p m, + (1 p) (w:)
(69)
W++1 Wt Cxe Sign (em: + (1 - ft) KK,,
(70)
where m, is called the momentum, the weight-decay parameter, P. E (0, constants, and Sign is function that computes entry-wise sign of vector. Let
W;
3t.
(71)
m74
then follows from (44) that
d d
d
Ghase
d,3,
da:
d hhss
hett
hec41 dh
d/9,
dæ:
aur d dmaa1 d&g
dweet dl wr m dim
duree d.m
0
dm
d mim
(72)
he4s d
d
d hese dm
diaett dhe
d we
[[Sign (cm, (1 Vi) + [w)]o'(3,))
I
[ax]]
d 1m+11
d mms+1
dme+1 dm
d.B.
ure
d
d dims
d.G.
d
where the second equality is due to (71) and the last equality follows from (70). Consider the following block representation of X₁,
Hls X
X,
=
(73)
Plugging the first row of Ghase, given in (72), into (46), implies that
Het1 - Sign (cm; + v)
B, >(T x (axi) )
(74)
For simplicity of notation, we define the diagonal matrix S. as
def (iggn (cm, + (1 e) V.) + kwr]
(75)
20

---

MetaOptimize
Then,
S, (',, B + >(I [a.
h4+1 = 7
: - S. 0'3,) B, + 7(1 K [(xx)) )
(76)
It follows that
d dh+t1 O, dm;
(77)
and
[ei] d'(3,) B!
[en] o( B,")
h Ww
-y
(78)
[ei] o (B,) D!
[en] o' (B,) B
where e; is the ith unit vector (i.e., n-dimensional vector whose ith entry is 1 and all other entries zero). Let 3e((] H1! be the ith entry of 3, and ith column of H4, respectively, for = I,..., m. Then,
and
7/6
d a
H,
+ S.
(g, B,")
83(11]
7/K
d a a
Hg + St 8 [m)
(r.)
B!")
d h44
(79)
d 1,
das
H|mm [m)]
+ S. (3,
B, 03.11]
7/6
d a..m
H|ml [m]]
+ S. 2 p.
(B.)
B|md
and
I x [cc]]
<- 1st
1 [oy]
2nd
d dh
(80)
d h
0
O
I x [ow]
with
dm.e1 It follows from (21), (72), and (77) that in the G, matrix, "dm, is the only non-zero block in its corresponding column of blocks. Consequently, it follows from (14) that XT", as defined in (73), has no impact on the update of H4+1, Y++1, and
Q++1- Therefore, the rows and columns of hase that correspond to derivative of m can be completely removed from Ghase By removing these rows and columns from G the matrix update (14) simplifies to
d d
Me44
d d
[
1 1
Y(+1 Hep1 =7
up
Sign (cm; + vf)l
k.
[oc]]
H4
11--))
Q4+1
dh+s1
dhet1
d
Qi
0
d.P.
w.
dh.
(81)
where (h1+1/ /d Br, he+1 /d w., and d hef1/d h4 are given (79), (78), and (80), respectively; and the blocks in the first row depend on the meta update.
B. Exiting Step-size Optimization Algorithms as Special Cases of MetaOptimize
In this appendix we show that some of the existing step-size optimization algorithms are special cases of the MetaOptimize framework. particular, we first consider the IDBD algorithm (Sutton, 1992) and its extension (Xu et al., 2018), and then discuss about the HyperGradient algorithm (Baydin et al., 2017).
21

---

MetaOptimize
B.1. IDBD and Its Extensions
(Sutton, 1992) proposed the IDBD algorithm for step-size optimization of class of quadratic loss functions. In particular,
considers loss functions of the form
fe(w:)) | 2 (a w. be)²
(82)
for given sequence of feature vectors a and target values b,, for t 1, Moreover, Sutton (1992) assumes weight-wise step sizes, in which case B, has the same dimension as we The update rule of IDBD is as follows:
9, - (aj wi be) az.
(83)
B411 + B h4 9:.
(84)
0:41 f exp (3++1),
(85)
W141 f- wt ac++1 9:'
(86)
h,41 + (1 - 1--aa at) h 14+
(87)
where (-)+ clips the entries at zero to make them non-negative, aimed improve stability. Here, gr is the gradient of fe fe(w,)
and af in the last line is vector that contains diagonal entries of the Hessian of fi- The updated values of 33 and would remain unchanged, if instead of the vector h4 we use diagonal matrix Hi and replace (84) and (87) by
B211 < 3 1j He It> H4+1 + (1 [a4+1a²1) Hi 1ar+191)
(88)
Note that [a²] is matrix that is obtained from zeroing-out all non-diagonal entries of the Hessian matrix of ft. It is easy to see that the above formulation of IDBD, equals the L-approximation of MetaOptimize framework when we use SGD for both base and meta updates, and further use diagonal approximation of the Hessian matrix along with rectifier the update of Hc.
An extension of IDBD beyond quadratic case has been derived in (Xu et al., 2018). Similar to IDBD, they also consider weight-wise step sizes, i.e., n. The update of step sizes in this method is as follows:
Bet1 t 3,- 17 H4 ((w.))
0++1 < exp(B.+1);
Wi+1 f- w. 01+1 Vi((ws),
H441 (I- [at+1) 4 fe(w.))) Hi [ao+1 Vi((w:)).
Similar IDBD, it is straightforward to check that the above set of updates is equivalent to the L-approximation of MetaOptimize framework that uses SGD for both base and meta updates, except for the fact that the above algorithm uses 0441 in u1+11 and H4+1 updates whereas MetaOptimize uses Qz,- This however has considerable impact since ar varies slowly.
B.2. Hyper-gradient Descent
HyperGradient descent was proposed in (Baydin et al., 2017) step-size optimization method. It considers scalar step size with straightforward extensions to weight-wise step sizes, and at each time t, updates the step size in direction to minimize the immediate next loss function. In particular, they propose the following additive update for step sizes, that can wrap around an arbitrary base update:
Br 1nx11
df((w:)
d w Br 1/ Vi.(w))
(89)
3 Br d d
d d
The last update can be equivalently
written as
8+11 B, - 11 H v(((w),
d d
(90)
H441 = 0 H4
d
22

---

MetaOptimize
The step-size update in (90) can be perceived as special case of MetaOptimize in two different ways. First, MetaOptimize algorithm that uses SGD as its meta update and approximate the G. matrix in (9) by zeroing out all of its blocks except for the top two blocks in the first column. From another perspective, the additive HyperGradient descent in (90) is also equivalent to a MetaOptimize algorithm that uses SGD as its meta update and sets = O. Note that setting equal to zero would eliminate the dependence of H on X, and Qt. as can be verified from (14). This would also render the 3 updates ignorant about the long-term impact of step size on future losses.
C. Experiment Details
In the appendix, we describe the details of experiments performed throughout the paper. In our experiments on CIFARIO, non-stationary CIFAR 100, and ImageNet dataset, we used machine with four Intel Xeon Gold 5120 Skylake @ 2.2GHz CPUs and a single NVIDIA V100 Volta (16GB HBM2 memory) GPU. For TinyStories dataset, we used a machine with four AMD Milan 7413 @ 2.65 GHz 128M cache L3 CPUs and a single NVIDIA A100SXM4 (40 GB memory) GPU. In all experiments, the meta step size y) is set to 10-3, The meta-parameters used in the considered optimization algorithm for CIFARIO, non-stationary CTFAR 100, ImageNet, and TinyStories are given in Table 2, Table 4, and Table 5, respectively. In the experiments, we performed a grid search for p. P. E {0.9, 0.99, 0.999). X, À E {0.99, 0.999]. and c, 2 E {0.9, 0.99). Regarding baselines with fixed step sizes, we did grid search for the learning rate in the set {10 -,005,0000 10 4, 10-3. 10 We set o equal to one in all experiments. Moreover, in ImageNet (respectively TinyStories) dataset, for Adam with the learning rate scheduler, we considered a cosine decay with 10k (respectively 1k) steps warmup (according to extensive experimental studies in (Chen et al., 2023) (respectively (Karpathy, 2024))) and did a grid search for the maximum learning rate in the set {10 10 10-
10-1).
Regarding other baseline algorithms, for DoG, although it is parameter- algorithm, its performance still sensitive to the initial step movement. We did grid search for the initial step movement in the set {10-9, 10- 10-7, 10-0) -I and reported the performance for the best value. In all experiments of DoG, considered the polynomial decay averaging. For Prodigy, we used the default values of parameters as suggested by the authors in github repository. For gdtuo, we considered the following (base, meta) combinations: (RMSprop. Adam), (Adam, Adam), and (SGD with momentum, Adam) and chose the best combination. For mechanic, we did experiments for the base updates of SGDm, Lion, and Adam and considered the best update. In order to have a fair comparison, in mechanic and gdtuo, we used the same initial step size as MetaOptimize.
Regarding the complexity overheads reported in Table 1, for Adam with fixed step-size we used the Pytroch implementation of AdamW. For all other baselines, used the implementation from the Github repository provided along with (and cited in) the corresponding paper. For MetaOptimize, we used the implementation in (Anonymous, 2024). Note that the implementation of MetaOptimize in (Anonymous, 2024) is not optimized for time or space efficiency, and smaller complexity overheads might be achieved with more efficient codes. For each algorithm, the wall-clock time overhead and GPU space overhead are computed by (Talg TAdamw) / Adamm and ( [ Adamw/ BBxax) - 1, respectively; where T and T daamw are per-iteration runtimes of the algorithm and Adam W, and Big and Buax that did not cause GPU-memory outage for the algorithm and AdamW.
AdamW are the maximum batch-sizes
Base Update
Meta Update (if any)
p
K
ē
Olo
"
Fixed step size
0.9
0.999
0.1
10-3
AdamW
Adam, Scalar
0.9
0.999
0.1
0.9
0.999
10
10-3 3
Adam, Blockwise
0.9
0.999
0.1
0.9
0.999
10 6
10 3
Fixed step size
0.99
0.1
0.9
10
Lion
Lion, Scalar
0.99
0.1
0.9
0.99
0.9
10
10
Lion, Blockwise
0.99
0.1
0.9
0.99
0.9
10
10
Fixed step size
0.999
0.1
10
RMSprop
Adam, Scalar
0.999
0.1
0.9
0.999
10
10
Adam, Blockwise
0.999
0.1
0.9
0.999
10
10
Fixed step size
0.9
0.1
10
SGDm
Adam, Scalar
0.9
0.1
10
10
Adam, Blockwise
0.9
0.1
10
10
Table 2. The values meta- parameeasssssrs used CIFARIO dataset.
23

---

MetaOptimize
Base Update
Meta Update (if any)
p
X
000
Fixed step size
0.9
0.999
0.1
10
AdamW
Adam, Scalar
0.9
0.999
0.1
0.9
0.999
10
10-3
10
0.999
Adam, Blockwise
0.9
0.999
0.1
0.9
0.999
10
10 3
0.999
Table 3. The values of meta-parameters used in non-stationary CIFAR 100 experiment.
Base Update
Meta Update
p
A
C
p
0k0
Fixed step size
0.9
0.999
0.1
10
AdamW
Lion, Scalar
0.9
0.999
0.1
0.99
0.9
10
10
Lion, Blockwise
0.9
0.999
0.1
0.99
0.9
10-0
10-3
Fixed step size
0.99
0.1
0.9
10-5
Lion
Lion, Scalar
0.99
0.1
0.9
0.99
0.9
10-0
10 10-3
Lion, Blockwise
0.99
0.1
0.9
0.99
0.9
10-6
10
10
SGDm
Lion, Scalar
0.9
0.1
0.9
10
10
Table 4. The values of meta patam-arreeters used in ImageNet datasct.
D. Further Experimental Results
Continual CIFARI00 experiment: In Figure 3, plotted the average top-1 accuracy-averagee over all past training times. This metric used in continual learning mainly because it summarizes algorithmic performance across multiple tasks, avoiding difficul//miseadding interpretations from task-specifi accuracy variations. Here, we also include the accuracy curves to reveal such variations. Figure depicts test accuracy at the end of each task.
52
50
Test Accuracy
48
46
44
AdamW, Fixed stepsize
42
MetaOptimize (AdamW.Adam) Scalar
MetaOptimize (AdamW,Adam), Blockwise
40
4
10
Task number
Figure 7. Top test accuracy of each task in the non-stationary CIFAR100 experiment, computed at the end of the task.
ImageNet dataset: In Figure we depict the train accuracy (top 1) and test accuracy (top of the considered algorithms in ImageNet dataset. As can be seen, in the train accuracy (top 1), MetaOptimize (SGDm, Lion) and MetaOptimize (AdamW, Lion) have the best performance. Moreover, in the test accuracy (topl), these two combinations of MetaOptimze outperform other hyperparameter optimization methods and only A with a handcrafted learning rate scheduler has a slightly better performance at the end of the training process.
Figure shows the results for the blockwise version of MetaOptimize for two combinations of (Adam W, Lion) and (Lion, Lion). As can seen, they showed no improvement over the scalar version.
24

---

MetaOptimize
Base Update
Meta Update (if any)
p
C
p
X
e
11
Fixed stepsize
0.9
0.999
0.1
10
AdamW
Adam, Scalar
0.9
0.999
0.1
0.9
0.999
10
10
Fixed stepsize
0.99
0.1
0.9
10
Lion
Lion, Scalar
0.99
0.1
0.9
0.99
0.9
10
10
Table 5. The values of meta-parameters used in TinyStories dataset.
70
70
60
Train Accuracy (Top 1)
Test Accuracy (Top 1)
50
50
AdamW, scheduler
AdamW, LR scheduler
AdamW, Fixed stepsize
AdamW, Fixed stepsize
40
Fixed stepsize
40
Lion, Fixed stepsize
MetaOptimize (AdamW, Lion), Scalar
MetaOptimize (AdamW, Lion), Scalar
MetaOptimize (Lion, Lion), Scalar
MetaOptimize (Lion, Lion), Scalar
30
MetaOptimize (SGDm, Lion)
30
MetaOptimize (SGDm, Lion)
gdtuo
gdtuo
mechanic
mechanic
20
Prodigy
20
Prodigy
20
40
60
80
Epoch
Epoch
(a) Train Accuracy (Top 1)
(b) Test Accuracy (Top 1)
Figure 8. ImageNet learning curves.
TinyStories experiment: In Figure 10, provide the test loss of considered algorithms for the TinyStories datasets. As can be seen, the learning curves have the same trends as the training loss in Figure 6.
25

---

MetaOptimize
85
Top Accuracy
80
75
MetaOptimize (AdamW, Lion), Scalar
70
MetaOptimize (Lion, Lion), Scalar MetaOptimize (AdamW, Lion), Blockwise MetaOptimize (Lion, Lion), Blockwise
0
20
40
60
80
Epoch
Figure 9. Comparison of blockwise version of MetaOptimize with the scalar version in ImageNet dataset.
2.2
AdamW. LR scheduler
AdamW, Fixed stepsize
2.0
Lion, Fixed stepsize
MetaOptimize (AdamW, Lion)
MetaOptimize (Lion, Lion)
1.8
DoG
Test Loss
gdtuo
mechanic
1.6
Prodigy
1.4
1.2
5000
10000
15000 Iteration
20000
25000
30000
Figure 10. TinyStories learning curves.
26


Reviews:
[{"review_id": "rhExh3CgtY", "reviewer": "Reviewer_Liw6", "summary": "The proposed approach seeks to dynamically adjust hyper parameters during the optimization process. The MetaOptimize framework, which utilizes historical iteration data, seeks to minimize regret builds on a discounted sum of future losses. This framework, coupled with several approximations, calculates meta-gradients to update the hyper parameters. The approximations suggested in the paper significantly reduces computational overhead, particularly in relation to Hessian matrix calculations. To demonstrate its effectiveness, the new optimizer was evaluated on both visual and textual datasets. Performance metrics, including learning trajectories and computational efficiency, were analyzed and presented as evidence of the method's efficacy.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Liw6 for their thoughtful and constructive comments. We carefully considered each suggestion and provide detailed clarifications below. We will apply the suggested improvements in the final version, which we believe significantly helps improve the clarity and presentation of the paper.\n\n---\n\n### Reviewer’s Concern 1: Relevance of Eligibility-Trace-Style Approximation\n\n> “The technique proposed...follows the eligibility-trace-style approach, well-known in RL. Why do the authors believe this RL-inspired approach is relevant to approximating gradients, as these seem like very different tasks?”\n\nThe eligibility-trace-style approach is relevant and beneficial in our setting because it efficiently approximates the meta-gradient in a causal way: each term in the RHS of Eq. (5) directly appears in Eq. (4), though at the earliest feasible time it can be computed (ensuring causality). An alternative, would be direct gradient computation through finite differences which requires unrolling the optimization procedure far into the future at each iteration, resulting in prohibitive computational and numerical challenges; and also results in large delays in updates not acceptable in online and continual applications. In contrast, the eligibility-trace approach provides an instant and tractable approximation by summarizing gradient effects over time.\n\nAs noted in the footnote on page 2, our approximation becomes exact when the meta-stepsize tends to zero. For larger meta-stepsizes, approximation accuracy naturally decreases—a phenomenon also observed in RL. In RL, sophisticated techniques like Dutch traces have been developed to mitigate such inaccuracies (van Hasselt et al., 2014). Similarly, extending these advanced RL-inspired methods to our meta-optimization framework is an exciting direction we highlight explicitly as future work (Section 9).\n\n---\n\n### Reviewer’s Concern 2: Comparison with Other Meta-learning Techniques (e.g., MADA)\n\n> “In reducing complexity, several approximations are used. It raises the question of comparing this technique explicitly with recent methods such as MADA (ICML 2024, Ozkara et al.).”\n  \nThe MADA method builds upon the Hypergradient-descent approach of Baydin et al. (2018), updating meta-parameters based on immediate hypergradients. We have already thoroughly compared MetaOptimize against Hypergradient-descent in our experiments (referred to as the \"gdtau baseline\" in Section 5 and Appendix B). Crucially, we demonstrate theoretically and empirically that Hypergradient-descent corresponds exactly to the special case of MetaOptimize with the discount parameter $\\gamma=0$. Thus, MADA inherently shares this limitation by considering only immediate-step effects, whereas our proposed method explicitly models long-term impacts via nonzero $\\gamma$.\n\nWe appreciate your suggestion and will explicitly reference and clarify this comparison with MADA in the final version of the paper.\n\n---\n\n### Reviewer’s Concern 3: Deterministic vs. Stochastic Meta-Optimizer\n\n> “In practice, stochastic optimizers are employed due to gradient intractability, whereas your paper presents a deterministic meta-optimizer. How would a stochastic version look, and was the deterministic version used in experiments?”\n\nThank you for highlighting this. We clarify that, despite the formal deterministic derivation in Section 4, our practical implementation is inherently stochastic. Specifically, the meta-update at each step (including the matrix $G_t$ computation) relies on gradients estimated using stochastic samples (such as mini-batches). Thus, in practice, MetaOptimize is already stochastic.\n\nIf desired, additional stochasticity could be introduced by employing further random samples (e.g., from a validation set) within each meta-update. We avoided this approach in our experiments to minimize sample complexity.\n\nWe will clarify the stochastic nature of our experimental implementation explicitly in the final paper revision.\n\n---\n\n### Reviewer’s Concern 4: Complexity of Section 4 Derivations\n\n> “Derivations in Section 4 are complex. More explanations would greatly improve readability.”\n\nThank you for pointing out this readability issue. We will significantly enhance clarity in Section 4 by explicitly providing a concise outline of the derivations at the section's outset. Our goal is to clarify the key conceptual steps, making the mathematical reasoning easier to follow.\n\n---\n\n### Reviewer’s Minor Comments\n\nWe are grateful for the reviewer’s meticulous attention to detail. We confirm that all minor suggestions (notation inconsistencies, typos, equation corrections) are valid and helpful. All indicated issues, including font inconsistencies, typographical errors, and minor mathematical clarifications, will be carefully fixed in the final version.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There are no proofs in the paper, but there are some mathematical developments, which seems correct to me.", "experimental_designs_or_analyses": "Yes, the results look valid.", "relation_to_broader_scientific_literature": "As far as I know the relevant literature, the idea proposed in this paper is novel.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths. This paper presents a novel and general MetaOptimize framework.\nThe framework is made practical through proposed approximation and Hessian-free versions that reduce computational complexity.\nThe effectiveness of the approach is thoroughly demonstrated through extensive experiments on both image classification and language modeling benchmarks\n\nWeaknesses. The technique proposed in this paper is designed to handle the setting where we a sequence of loss functions {f_t}. However, this comes with the price of approximating the true gradient with a sum of the gradients over the past iterations. This idea follows the eligibility-trace-style approach, which is well-known in RL. Why the authors believe that this approach is also relevant in approximating gradients as it seems to be a very different task than RL? Can we measure the error produced by this approximation? \n\nIn reducing the complexity of the proposed optimizer, the authors suggest some approximations other matrix G_t. This is done on top of the approximation mentioned in the first point (there is also the approximation that is done in Section 5). This raise the question of comparing the technique proposed in this paper with other possible techniques for learning hyper parameters. For example, comparing to the recent paper MADA: Meta-Adaptive Optimizers through hyper-gradient Descent (ICML 2024) by Ozkara et. al..\n\nIn practical settings, stochastic optimizers are used to overcome the intractability of computing gradients. However, in this paper, the authors develop a deterministic meta optimizer. How a stochastic version would look like? In the numerical experiments, do you use the deterministic version that is presented in the paper?\n\nThe derivations in Section 4 are complicated to follow. I understand that space is always an issue but more explanations will be very helpful.", "other_comments_or_suggestions": "On page 2, There are two font versions of  w_t in the text (see lines 73 and 84, for example) and I am not sure if they are the same.\nOn page 3, left column, line 121 , w_{t+1} --> x_{t+1}?\nOn page 3, left column, line 122, You don't need w_{t+1} for the computation of H_{T}^{T}\\nabla f_{t}(w_{t}), right? \nOn page 3, right column, line 127, It seems you start a new sentence here as formula (13) ends with a period.\nOn page 10, last reference, RS Sutton --> Richard S Sutton :).\nOn page 12, line 630, Maybe you want to mention when you write \\sigma'(\\beta_{t})) the relation in (3) defining \\alpha_t.\nOn page 13, line 682, inequality --> equality.\nOn page 16, line 833, there is a missing \\gamma on the right hand side (unless I missed something).", "questions_for_authors": "See my list of weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jghlmiFx8x", "reviewer": "Reviewer_wz4K", "summary": "This paper introduces MetaOptimize, an approach for automatically adapting the learning rates of base optimization algorithms like SGD, Adam, and Lion.  MetaOptimize maintains a set of learning rates updated with a separate stochastic gradient optimizer to minimize the discounted sum of future losses.  Naively updating the learning rates requires maintaining multiple states and computing the Hessian of the base objective; the authors propose a tractable Hessian-free formulation with multiple approximations to reduce computational overhead.  Experiments show MetaOptimize outperforms previous meta-optimization methods like Prodigy and mechanic and matches a well tuned AdamW for ResNet on Imagenet and outperforms AdamW with a fixed learning rate for GPT on TinyStories (lags well-tuned AdamW).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer wz4K for thoughtful and valuable feedback. Below we provide clarifications, which we will also highlight in final version of the paper.\n\n---\n\n### On Discount Factor γ:\n\n>Authors state γ=1 was used in stationary experiments, and performance meaningfully degrades for γ<0.999, making discounting somewhat contrived.\n  \nWe clarify two important aspects:\n\n-  Our general framework naturally accommodates discount factors less than one. While stationary experiments favored γ≈1 due to implicit decay (discussed in the next bullet), having the flexibility to set γ<1 is beneficial for non-stationary tasks (e.g., continual learning scenarios, as shown in Section 7.2), where explicit discounting significantly improves performance. Thus, the framework’s generality allowing γ<1 is not contrived but genuinely beneficial for practical settings.\n\n-  The use of weight decay implicitly introduces discounting in meta-updates. To see why clearly, consider the base optimizer being SGD with weight decay parameter $\\kappa$. The eligibility trace updates (under L approximation) become:\n  $$\n  h_{t+1} = \\gamma (1 - \\alpha_t \\kappa) h_t - \\gamma\\alpha_t \\nabla^2 f_t(w_t) h_t - \\alpha_t \\nabla f_t(w_t).\n  $$\n  Even if we set γ=1 and use a Hessian-free approximation (i.e., ignore the Hessian term), the trace $h_t$ still decays at rate $(1 - \\alpha_t \\kappa)$, exhibiting a discount-like effect. Thus, explicit discounting (γ<1) was unnecessary here primarily due to this implicit decay from weight decay.\n\n>Does backward formulation rely on γ<1? How does γ=1 impact derivation?\n\nThis is an important subtlety, and you correctly identified that our backward derivation (Eq. (6)) formally assumes γ<1 because of the scaling term $(1-\\gamma)$ used for normalization. A straightforward workaround for γ=1 is simply removing this scaling factor $(1-\\gamma)$ from the definition and all subsequent formulas. This slight adjustment makes the derivation fully valid and consistent for all γ≤1, matching precisely what we implemented and tested experimentally.\n\nYour comment also highlights an interesting direction for future research. In RL and dynamic programming, the formulation and algorithms for the case γ=1 differ subtly, requiring additional  reward-centering. Exploring analogous techniques for meta-optimization when γ=1 could be beneficial and is a promising avenue we will include in our future work discussion.\n\n---\n\n### On Practical Value:\n\n>MetaOptimize lags behind well-tuned AdamW on TinyStories; thus, its practical value is unclear.\n\nWe acknowledge this observation and provide some clarification:\n\n - The AdamW baseline used a carefully tuned step-size schedule through extensive search, specifically optimized for this task; while MetaOptimize involved no manual tuning. Thus, we did not expect scalar MetaOptimize to outperform such an extensively optimized schedule.\n\n-  In addition to avoiding manual hyperparameter tuning, MetaOptimize benefits from generality and applicability to generic meta-paramters beyond stepsizes. The biggest advantage however appear in continual learning (see experiment in Section 7.2), where tasks evolve over time and pre-defined stepsize schedules are ineffective.\n\n- Crucially, we emphasize this paper’s primary contribution as foundational, laying the groundwork for a highly promising research direction. The greatest benefits of meta optimization methods are likely still ahead. We explicitly outline multiple clear pathways for future improvements (Section 9), highlighting that the current results are only the beginning. Thus, while initial performance is competitive rather than groundbreaking, the demonstrated potential and clear roadmap for future research represent the key strengths of this work.\n\n>Sensitivity to the base optimizer .., For example, base AdamW worked best on ImageNet while base Lion worked best on TinyStories.\n\nThe key point is that regardless of the base optimizer, applying MetaOptimize to learn the step sizes consistently outperforms the same base algorithm with best fixed step size. Note that the  Lion optimizer tends to work better in training Transformers even when using fixed step sizes.\n\n---\n\n### Scale of Experiments:\nWe agree scaling is important. Due to resource constraints, we focused on models <20M parameters, but included diverse architectures to show broad applicability. MetaOptimize is theoretically scale-agnostic and compatible with any first-order optimizer, so we expect it to extend naturally to larger models. Scaling up is a valuable next step and will be highlighted in future work.\n\n---\n\n### Continual Learning:\nSection 7.2 includes a continual CIFAR-100 experiment where MetaOptimize shows strong adaptation: step sizes increase after task switches to support quick adaptation, and decrease in early layers over time, reflecting stable feature reuse. These behaviors align well with continual learning needs, showcasing MetaOptimize’s potential in nonstationary settings.", "claims_and_evidence": "Figure 2 and 3 provide good evidence that MetaOptimize is able to recover from poor initial learning rates and also adapt to nonstationary data.  \n\nResults in Figure 6 shows a well-tuned MetaOptimize outperforms prior approaches like Prodigy, mechanic, and gdtuo.  However, MetaOptimize is sensitive to the selection of the optimizer used for base and meta updates and still lags a well-tuned AdamW on TinyStories.", "methods_and_evaluation_criteria": "The evaluation methods make sense for the problem.", "theoretical_claims": "There are no error bounds/convergence guarantees presented in the paper.  I did not check the derivation of MetaOptimze updates for different base optimizer.", "experimental_designs_or_analyses": "The experimental design and analysis are sound.", "relation_to_broader_scientific_literature": "This work follows prior work on learning-rate free optimizers and provides a formulation based on minimizing sum of discounted future losses.", "essential_references_not_discussed": "References to prior work is sufficiently discussed.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is well written and easy to understand.  \n\nWeaknesses:\n- In Section 7.5, the authors state that a discount factor of 1 was used in all stationary experiments and performance degrades meaningfully with $\\gamma\\leq 0.999$.  This makes the discounting of future losses somewhat contrived.  \n- It is unclear how valuable MetaOptimize is in practice given it lags a well-tuned AdamW on TinyStories by a meaningful amount.  \n- Experiments are on fairly small models with <20m parameters.  It is unclear how well MetaOptimize performs on larger models.\n- The performance of MetaOptimize seems sensitive to the base and meta optimizers selected.  For example, base AdamW and Lion for MetaOptimize worked best on ImageNet but base Lion and Lion for MetaOptimize worked best on TinyStories.", "other_comments_or_suggestions": "Given the suitability of MetaOptimize for continual learning and nonstationary data, I think the experiments would benefit from a practical nonstationary learning problem from RL or continual learning.", "questions_for_authors": "Does the backward formulation depend on the discount factor being less than 1?  How does a using a discount factor of 1 as is done for the stationary experiments impact the derivation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DaByTucp0B", "reviewer": "Reviewer_51Lj", "summary": "This paper proposes a method for optimising hyperparameters online in first order optimisation algorithms. Using this framework, performance is drastically improved over using fixed hyperparameters in a number of experiments. There are a number of qualitative approximations used which can simplify the complexity of the method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback and helpful suggestions, which we address below and plan to incorporate into the final version of the paper. We believe this will significantly enhance the clarity and quality of the work.\n\n---\n\n### Continual Learning\n>I think the missing benchmark that would really complete this paper would be in the method's application to continual learning problems... the paper discusses that MetaOptimize can be applied to continual learning problems but... it is not evaluated on any.\n\nWe already include a continual learning experiment in Section 7.2 on continual CIFAR-100 benchmark. This benchmark sequentially trains on 10 tasks (10 classes each) without explicit task boundaries or weight resets. As discussed in the paragraph preceeding Figures 3 and 4, MetaOptimize demonstrates substantial advantages in continual learning, including fast adaptation to new tasks, and favourable layer-wise adaptive behaviour.\n\n>I would love to see more discussion around figure 3 and 4—in particular, why you think this happens. \n\nHere are some intuition for the observed phenomena:\n\n- **Why later-layer step-sizes are consistently larger:**  \nThis aligns with established best practices in supervised learning (Howard & Ruder, 2018), where larger step-sizes in later layers accelerate training. MetaOptimize autonomously discovers this pattern, despite equal initial step-sizes (10⁻⁴).\n\n- **Why step-sizes of earlier layers decrease and later-layers increase in continual learning:**  \nEarly layers converge toward stable low-level image features shared across tasks, thus needing smaller updates over time. Later layers must continually adapt to changing labels, thus requiring constantly larger step-sizes.\n\n- **Saw-tooth pattern in step-sizes (Fig. 4):**  \nEach spike corresponds exactly to task transitions. MetaOptimize momentarily boosts step-sizes to facilitate rapid adaptation to new tasks.\n\n>I would want to see this phenomenon occur... over multiple seeds...\n\nFigures 3 and 4 report averages over **5 random seeds**, an important detail we mistakenly omitted in the submitted manuscript. We will explicitly clarify this in our revision.\n\n---\n\n### Methods and Evaluations\n\n>The one comparison I think is missing is between their implementation and the specific instances of their framework (i.e. hypergradient descent and IDBD).  \n\nThank you for pointing this out. Hypergradient Descent is already included in our experiments as gdtau (Chandra et al., 2022), an efficient implementation of hypergradient descent.\n\nRegarding IDBD, its original design targets linear regression tasks, making it inapplicable to the neural-network-based scenarios presented in our work. \n\n>In figure 3, average cumulative accuracy is a very strange metric. Why not report the accuracy?\n\nAverage cumulative accuracy is used in continual learning mainly because it summarizes algorithmic performance across multiple tasks, avoiding difficult/misleading interpretations from task-specific accuracy variations. In the final version of the paper, we will also include the accuracy curves to reveal such variations.\n\n>figure 4 is a bit misleading, since the two lines use different scales.\n\nThank you for highlighting this. Initially, both blocks have identical step-sizes (10⁻⁴). To eliminate confusion, we will plot step-sizes for the two blocks in separate subfigures.\n\n---\n\n### Relation to Broader Scientific Literature (Learned Optimization)\n\nThank you for highlighting the interesting relevant works on Learned-to-Optimize (L2O). Traditional L2O approaches typically learn optimizers or hyperparameters offline, then fix them at deployment. In contrast, MetaOptimize dynamically adjusts hyperparameters (step sizes) concurrently during training. This real-time adaptation to changing conditions or tasks is specifically beneficial in continual learning. In the revised manuscript, we will carefully contrast MetaOptimize with L2O, explicitly citing and discussing the references you kindly suggested. Additionally, we are thinking that a combination of the two areas (e.g., learning to meta optimize, or using L2O-discovered optimizer as base algorithms MetaOptimize, etc) might be promising research directions.\n\nThe reference on L2O for RL is particularly interesting. While the present manuscript focuses on supervised and continual learning (as more controlled settings), we plan future research extending MetaOptimize explicitly into RL.\n\n---\n\n### Stability of MetaOptimize\n\n>How stable is your method?\n\nAll reported MetaOptimize experiments exhibit stable, consistent behavior across multiple random seeds. However, we observed instability in certain variants not included in the experiments (e.g., weight-wise or layer-wise step sizes in very deep networks). We explicitly mention these stability challenges in our \"Limitations\" section, suggesting directions for future research. Solving these underlying issues could significantly enhance MetaOptimize performance.", "claims_and_evidence": "The papers claims are as follows:\n- They produce a formal approach for optimising step-size and other hyperparameters. This is true.\n- Their algorithm is general and can be applied to any first-order optimisation algorithm. This is true.\n- Their algorithm is computationally efficient thanks to a number of approximations. It is true that there are a number of approximations, and a small demonstration that their method does not have too large an overhead; it would be interesting to see compute-normalised experiments too, although this may not be practical.\n- They demonstrate that some prior work is purely specific instances of this more general framework, which is true. I would, however, be curious to see how these specific instances compare to the algorithm used in this work empirically.", "methods_and_evaluation_criteria": "There are 3 algorithms, growing in specificity, proposed in this work. These are sensible applications from the more theoretical framework generated earlier in the paper. They compare against a number of sensible baselines - in many cases, using a standard learning rate, but also occasionally against other online hyperparameter optimisation algorithms. The one comparison I think is missing is between their implementation and the specific instances of their framework (i.e. hypergradient descent and IDBD, which are discussed in the paper). I would be curious to know whether their more general system enables improved performance.\n\nTheir method is evaluated on a number of standard stationary datasets, which is good. However, I think the missing benchmark that would really complete this paper would be in the method's application to continual learning problems, where having adaptive hyperparameters could lead to huge performance improvements but which may prove difficult to operate in; the paper discusses that MetaOptimise can be applied to continual learnign problems but as far as I can tell it is not evaluated on any.", "theoretical_claims": "There are a number of derivations, which I followed through in the main body of the paper, but there are not theoretical proofs.", "experimental_designs_or_analyses": "All experiments seem to be run for a single seed, without error bars. Besides this, the apper includes hyperparameters and meta-parameters needed for the experiments, and seems like it should be fully reproducible. There is a relatively brief sensitivity analysis, and a larger analysis considering how the step size of a model optimised using MetaOptimise changes over training. I believe that the necessary analyses are covered; more is always better, but I think the necessary bases are covered.", "relation_to_broader_scientific_literature": "The paper contextualises prior literature as specific instances of its more general framework. This is done rigorously. There is a large related work section which seems to provide good coverage of prior literature, albeit missing one crucial field in my opinion.\n\nI personally think contextualisation of this work against learned optimisation would provide additional value - in L2O, the optimiser is learned upfront and then not changed at test-time, which provides an interesting comparison to this work where the hyperparameters are learned at the same time as the model.\n\nI highlight a few referencess below with some reasons why I think they should specifically be included:\n\n- Learning to learn by gradient descent by gradient descent, 2016, Andrychowicz et al. - kicked off a lot of the L2O work.\n- Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves, 2020, Metz et al. - a strong demonstration of learned optimisers\n- VeLO: Training Versatile Learned Optimizers by Scaling Up, 2022, Metz et al. - a large scale learned optimiser with no hyperparameters\n- Practical tradeoffs between memory, compute, and performance in learned optimizers, 2022, Metz et al. - Introduces a python library for learned optimisation which, among others, includes an optimiser 'NN_Adam' where the hyperparameters of Adam are replaced with a neural network.\n- Can Learned Optimization Make Reinforcement Learning Less Difficult, 2024, Goldie et al. - a learned optimiser designed for reinforcement learning which exhibits much of the desired behaviour in MetaOptimise, such as having different update sizes for different parts of the network and a step size which changes throughout the course of training, including at the start of every new batch (which I assume is what causes the periodic structure in figure 3 and 4).", "essential_references_not_discussed": "See above.", "other_strengths_and_weaknesses": "Strengths:\n- I liked the plots showing robustness to idfferent meta-parameters (i.e. the initial alpha). I also think it is interesting to see that the step size differs between the first and second block.\n\nWeaknesses:\n- In figure 3, average cumulative accuracy is a very strange metric. Why not report the accuracy?\n- I would love to see more discussion around figure 3 and 4 - in particular, why you think this happens. To properly draw conclusiosn, I would want to see this phenomena occur at least over multiple seeds but, preferably, over multiple different experiments (eg the second block step size growing through training). I also think that the way results in figure 4 are displayed is a bit misleading, since the two lines use different scales - I think a naive reader might think the step sizes are crossing over, but it is actually more interesting to note that the second block has much larger step sizes!", "other_comments_or_suggestions": "I think most of my other comments are interspersed into the review.", "questions_for_authors": "- Why do you think the second block needs larger learning rates, and grows over time?\n- How stable is your method?\n- How would your method work in a non-stationary dataset that is standard for continual learning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VApAgpjufs", "reviewer": "Reviewer_TH46", "summary": "This paper proposes to optimize step size by viewing step size as a differentiable parameter, and minimizing the objective of cumulated loss by recording a temporal trajectory.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's honesty in stating clearly that this paper falls outside their area of expertise. However, it seems there was a critical misunderstanding regarding the approximation in Equation (5).\n\n> **Reviewer:** _\"Using eqn (5) as a surrogate of eqn (4) is the basic to make the proposed method practical. However, such approximation is not reasonable. I cannot see any rationality in footnote 1 as justification...\"_\n\nWe clarify the approximation carefully below, as there appears to be a misunderstanding:\n\n- The statement in Footnote 1 (p.2) specifically analyzes the case as $\\eta \\to 0$. In this scenario, indeed $\\beta_T^5 - \\beta_T^4 \\to 0$. However, the non-trivial insight is that $(\\beta_T^5 - \\beta_T^4)/\\eta \\to 0$, meaning the difference between these terms vanishes faster than $\\eta$. Concretely, this implies:\n  - While both $\\beta_T^4$ and $\\beta_T^5$ scale roughly as $\\eta T$, their difference scales strictly slower ($o(\\eta T)$), making the approximation increasingly accurate for small $\\eta$.\n\n- Intuitively, each term in the RHS of Eq. (5) directly appears in Eq. (4), though at the earliest feasible time it can be computed (ensuring causality). Thus, Eq. (5) serves as a valid practical approximation to Eq. (4).\n\nFinally, we emphasize that this forward-backward view approximation is well-established in related fields, notably in eligibility trace methods widely used in reinforcement learning literature.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "No. Using eqn (5) as a surrogate of eqn (4) is the basic to make the proposed method practical. However, such approximation is not reasonable. I can not see any rationality in footnote 1 as justification of such approximation. If as it says, $\\beta_T^5=\\beta_T^4$, then $\\beta_T^5=\\beta_T^4=\\beta_0$: we can not update $\\beta$ under such assumption.", "theoretical_claims": "Yes. Please refer to Methods And Evaluation Criteria", "experimental_designs_or_analyses": "No.", "relation_to_broader_scientific_literature": "I could not identify the contribution as the proposed method can not be understood.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "I am not familiar with related works in this area at all, and I feel confused why OpenReview assign this paper to me to review. \n\nBut I can understand the proposed method and find fatal error as mentioned above.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an online approach to hyper parameter optimisation. Unlike L2O approaches, hyper parameters are adjusted during training, which is particularly helpful in continual learning settings, for example. Reviewers generally agreed that this is an interesting and valuable line of work. 
The outstanding divisive issue, which generated some debate during the discussion phase, is what scale of experiments should be required to empirically validate a new optimisation method. This paper considers ResNet18 (CIFAR, ImageNet) and a small language model. I agree that bigger experiments would be preferred and probably generate more attention for the work. However, it’s already interesting and hopefully other members of the community can help to apply it at scale. 
I recommend accept.
Decision: Accept (poster)