$\mu$LO: Compute-Efficient Meta-Generalization of Learned Optimizers
Authors: Benjamin Thérien
Charles-Étienne Joseph
Boris Knyazev
Edouard Oyallon
Irina Rish
Eugene Belilovsky
Venue: ICLR 2026 Poster (2026)
Keywords: Learned Optimizer
Meta Generalization
MuP
Maximal Update Parameterization
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a new way to meta train learned optimizers, allowing them to generalize from small meta-training tasks to large unseen tasks for the first time.

Abstract: Learned optimizers (LOs) have the potential to significantly reduce the wall-clock training time of neural networks. However, they can struggle to optimize unseen tasks (*meta-generalize*), especially when training networks wider than those seen during meta-training. To address this, we derive the Maximal Update Parametrization ($\mu$P) for two state-of-the-art learned optimizer architectures and propose a simple meta-training recipe for $\mu$-parameterized LOs ($\mu$LOs). Our empirical evaluation demonstrates that LOs meta-trained with our recipe substantially improve meta-generalization to wider unseen tasks when compared to LOs trained under standard parametrization (SP) using the same compute budget. We also empirically observe that $\mu$LOs exhibit unexpectedly improved meta-generalization to deeper networks ($5\times$ meta-training) and surprising generalization to much longer training horizons ($25\times$ meta-training) when compared to SP LOs.

Published as a conference paper at ICLR 2026
pLO: COMPUTE-EFFICIENT META-GENNRALIZATION OF LEARNED OPTIMIZERS
Benjamin Thérien 1.2
Charles-Etienne Joseph!.2
Boris Knyazev
Edouard Oyallon
Irina Rish 2.2
Eugene Belilovsky2..
Université de Montréal: Mila - -Quebec AI Institute; Concordia University, Montréal; Samsung Al Lab, Montréal; "ISIR Sorbonne University, CNRS, Paris, France.
ABSTRACT
Learned optimizers (LOs) have the potential to significantly reduce the wall-clock training time of neural networks. However, they can struggle to optimize unseen tasks mheta-generallie), especially when training networks wider than those seen during meta-training. To address this, we derive the Maximal Update Parametriza- tion (μP) for two state-of-the-art learned optimizer architectures and propose a simple meta-training recipe for u-parameterized LOs (jLOOs). Our empirical eval- uation demonstrates that LOs meta- meta-traines trained with our recipe substantially improve meta-gentn eittageenerllization wider unseen tasks when compared to LOs trained under standard parametrization (SP) using the same compute budget. We also empirically observe that pLOs exhibit unexpectedly improved veeta-genrralizttoon to deeper networks (5 meta-training) and surprising generalization to much longer training horizons (25x meta-training) when compared SP LOs.
INTRODUCTION
While deep learning (DL) has largely replaced hand band-designed algorithms, one crucial component of DL training remains hand-crafted: gradient-base based optimizers. While popular optimizers such as
Adam or SGD provably converge to local minimum in non-convex settings (Kingma & Ba 2017]
Li al. 2023a Robbins 1951), the existing literature provides no evidence that these optimizers converge to the global optimum at the optimal rate. With the lack of theory certifying the optimality of existing optimizers and the clear strength of data-driven methods, it is natural to turn towards data-driven solutions for improving the optimization of neural networks.
Taking a step in this direction, Andrychowicz et al. (016)); Wichrowska el al. (2017); Metz et al. 20199 2022ab replace hand-designe optimizers with small neural networks called learned optimizers (LOs). LOs are meta-lcarned on a task distribution by minimizing the loss the inner learning problem (e.g. neural network training in case) across a batch of tasks. Being neural networks themselves, these optimizers are advantaged by their substantially larger parameter counts than Adam or SGD, making them suitable to large-scale meta- veeaa-training For instance, |Metz et al. (022b) showed that scaling up learned optimizer meta-training to 4000 TPU months can produce an optimizer, VeLO, that significantly outperforms well-tuned hand-designed optimizers without requiring hyperparameter tuning. However, even VeLO has limitations in meta- tatta-gnnraalization optimizing unseen problems. Specifically, VeLO (Metz al. 2022b) known to (1) have difficulty optimizing models much wider and deeper than those seen during meta-training (See Figures 6 and 9 of Metz et al. (2022b)) and (2) generalize poorly to longer optimization problems (e.g., training for more steps) than those seen during meta-traine vetaa-raining.
The problem of meta-generan generaaeaallinooonnn is fundamental to learned optimization due to the requirement for tractable meta-training and the expectation of strong performance across combinatorially large set of downstream tasks. Meta- deta--eneralization refers to the ability of meta-learned algorithm to generalize, that is, perform well when applied to unseen tasks. In the case of LOs, a learned optimizer
Correspondence to: Benjamin Thérien (benjamin.theriin @umontreal.ca) and Eugene Belilovsky (eugene.belilossee www.eelllosooooooooooooo.oo @concordia.ca). Our code is open-sourced: bttps://githuue http:///tt...b.ccoo////ccaaaaaaa learned Spt Intimizaiio

---

Published as a conference paper at ICLR 2026
Meta-training Tasks (In Distribution)
Meta Generalization (Out of Distribution)
μLO
LOw
AdamW HAdam
μVooOO
VeLOw
Unroll Length
Width
Unroll Length
Width
25000
8162
4096
5000
2048
1000
1024
512
fre M/ f/
Average Rank
128
MLP
VIT
Architesture
ImageNet-32
Depth
Architecture
Depth
ImageNet-64 64
CIFAR1O
LM1B
Dataset
Dataset
(a) Axes of Meta Generallizzzion
(b) Performance by Average Rank
Figure 1: Meta-generallizttinn is severely limited without approach. Subfigure (a) illustrates meta-genent gentraggenaaliztiion axes by distinguishing between meta- meea-training tasks used herein (blue) and out- of-distribution tasks (red). Subfigure (b) reports the average rank across tasks within our evaluation suite that are out-of-distribution with respect to the corresponding axis. Both Adam and 4Admm undergo task-specific hyperparameter tuning across more than 500 configurations per task. Learned Optimizers of the same architecture are meta-learned on the same tasks with FLOP-matched budget.
trained on a tractable and, thus, limited distribution of meta-training tasks should nevertheless exhibit strong performance when applied out-of-distribution tasks: new combinations of architecture, dataset, and training objective (Figure D.. Even changes as small as increasing the hidden dimension of the architecture (width), the number of layers (depth), or the number of training steps (unroll length) can cause meaningful distribution shifts between meta- veta-training and testing tasks, leading to poor generalization. Consequently, understanding and improving meta-gent getaa-enerallzation is central to making learned optimizers practical for real-world machine learning workloads.
In this work, we focus on the problem of LO meta-generalizationn to tasks of larger hidden dimension (width) than those seen during meta-learning A related problem is that of transferring hyperpa- rameters of hand-designed optimizers to wider tasks. Introduced by Yang et al. (002). juP is an optimizer-dependen and width-dependent parameterization (e.g.. rule for initializing a model, scaling its pre-activations and scaling the optimizer's 's updates) that allows hyperparameter transfer to larger width tasks for Adam and SGD. Making the connection between encerparameter-rranster and meta-generalizatton, we ask: Are existing learned optimizer architectures compatible with μP? Does mela-learning optimizers under p.P improve mela- reniazzgeenallztaionn To answer this question, theoretically analyze two recent I.O architectures (Metz et al. 2022a b) (sec. derive the appropriate maximal update parameterization for them, and carefully design low-cost meta-training recipe to bring out their meta getta-ennraalzatioon capabilities. We then provide extensive experimental evidence demonstrating that μlOSs generalize to large unseen tasks. Our contributions are as follows:
We derive j-parameterizztion for two popular learned optimizer architectures (Vel.O and small fc fc lopt) and demonstrate theoretically that our parameterization satisfies pP desider- ata.
We design a set of meta-training and meta-testing tasks enabling a systematic study of meta- gettraeennaliioionn and demonstrate that our pLOs significantly outperform strong baseline LOs and hand-designed optimizers.
We demonstrate empirically that our fa.Os show surprisingly good generalization to deeper networks (5x meta-training) and longer training horizons (25x meta- veta-training) when com- pared to baseline LOs.
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND
Learned optimizer objective. A standard approach to learning optimizers Meezz et al. 2019 is to solve the following meta- learnigg problem:
T T
min WOD
EXXYY
C(X,Y ((.,X:f((u,))) wr,
(1)
1=0
Where T is a distribution over optimization tasks defined as tuples of dataset D, objective function L, and initial weights wo associated with a particular neural architecture (we refer to this network as the optimizee); á represents the weights of the learned optimizer,fa with input features uy; and T is the length of the unroll which we write our experiments, the sum of per- timestet loss is the quantity being optimized. It should be noted, however, that one could also optimize the final loss, final accuracy, or any other performance metric. Gradient descent is the preferred approach to solving equation [ However, estimating meta-gradient via backpropagation known
fixed quantity for simplicity. In equation and in
problematic for long unrolls (Metz al., 2019). Therefore, learned optimizer meta- titta-graiietss are estimated using evolution strategies and their variants (Vicol
ct al.
2021 Buckman et al. 2018 Nesterov & Spokoiny 2017 al.
Parmas et al. 2018 Vicol 2023
2023b).
Learned optimizer input, output, and update. Learned optimizer neural architectures have taken many forms over the years, will briefly review two recent architectures, small fc lopt of Metz et al. (222a)) and VeI.O of Metz et al. (2022b), as they are used our experiments. These learned optimizers construct input features u, based momentum accumulators, variance accumulator, and multiple adafactor accumulators, we provide full list in Tables 2 3 and El of the Appendix. At
every gradient descent step, small_fc_loe [c_l__fc_lopt and VeLO are applied to each parameter of the optimizee, producing two outputs: the magnitude (m) and direction (d) of the update. VeLO additionally outputs a tensor-lever learning rate, ow. The per-parameter update for both optimizers is given as
Wt Wt-1 1 aw awwAidexp exp (A2m),
where w is parameter of weight matrix W, A and A2 constant values set 0.001 to bias initial step sizes to be small. For small_fc fo_l_fe_loop, ow = 1 always. We refer interested readers to appendix sections J.I.aadd for more details.
RELATED WORK
Generalization in LOs. There are three main difficulties of learned optimizer generalization (Chen al. 2022 Amos 2022): (1) optimizing unseen tasks; (2) optimizing beyond maximum unroll length seen during meta-training; (3) training optimizees that do not overfit. Among these, (3) has been most extensively addressed as this problem has been well studied in classic optimization literature. For example, extra-regularrizaton terms can be directly applied to a learned optimizer (Harrison) validation set objective (Metz et al. 2019) or parameterizing LOs as hyperparameter controllers (Almeida et al. 202T). The problem (2) has been mitigated by regularization (Harrison al. 2022 Yang al. (023) and larger-scale meta-training (Metz et al. 2022b). However, (1) has remained a more difficult and understudied problem.
al. 2022 Yang et 2023). In addition, (3) can be addressed by meta-training
To the best of our knowledge, the only current approach to tackle this problem is to meta-train LOs on thousands of tasks (Metz et al.) 20226). However, this approach is extremely expensive and seems bound to fail in the regime where the optimizer is expected to generalize from small meta-training tasks in standard parameterization large unseen tasks: figures and 9 Metz et al. (202bb) demonstrate that this was not achieved even when using 4000 TPU-months of compute. Generalization would expected if all tasks, no matter the size, were included in the meta-training distribution, but such an approach is simply intractable and is likely remain so.
Maximal Update Parametrization and Hyperparameter transfer. First proposed by Yang & Hu (2021), the Maximal Update Parametrization is the unique stable Pooo-arammttizationn where every layer learns features. The parameterization was derived for adaptive optimizers by Yang & Littwin (2023) and was applied by Yang al. 2022) enable zero-shot hyperparameter transfer
3

---

Published as a conference paper at ICLR 2026
for Adam and SGD. Most recently, in tensor programs VI, Yang et al. (0244) a parameterization allowing for hyperparameter transfer in infinitely deep networks. While it is appealing, Depth-uP is only valid for residual networks with a block depth of 1, so it does not apply most practical architectures (e.g.. transformers, resnets, etc.). For these reasons, we do not study Depth-juP herein. Following from the original discovery of hyperparameter transfer in Yang et a (2022), a number of follow-up works have emerged that are not part of the tensor programs series. Dey al. (2024) investigates transferring hyperparameters across different sparsity levels and widths. Blake et al. (2025) investigates combination of jul and unit scaling, which results in easier tuning and more stable low-precision training. [vereett et al. (0224) investigate the alignment assumptions of Yang et al. (2022) and find that appropriate per-layer learning rate prescriptions can also enable hyperparameter transfer standard, mean field, and NTK parameterizations. In their empirical investigation of scaling exponents across these parameterizations, the authors find that SP with layer-wise learning rates outperforms J.P. While we study the impact of meta-lcarning optimizers in jPP on meta- aataa-eneralization herein, it is still open question which parameterization is best for meta-learning optimizers. Finally, in concurrent work, Dey et al. (2025) propose CompleteP, parameterization that can achieve transfer of optimal hyperparameters across depth and width.
propose Depth-uuP,
J-PARAMETTRIZATIO FOR LEARNED OPTIMIZERS
Parameterizing optimizee neural network in μP requires special handling of the initialization variance, pre-activation multipliers, and optimizer update for each weight matrix W E RR"* the network. Specifically, these quantities will depend on the functional form of the optimizer and the dependence of n (FAN_OUT) and m (FAN_IN) on width. We will refer to weight matrices in network of width h as hidden layers if ((n) ((m) (hh), as output layers if (n) ((m)) ((h). and as input layers if ((n) = ((h) A ((m) = @(1). Here, θ is standard asymptotic notation. Note that all biases and the weights of normalization layers are considered input layers and should be scaled as such. With this in mind, consider an arbitrary neural network whose weight matrices are denoted Wi, where I indexes the layers; the following modifications are then required obtain pP for learned optimizers.
Optimizee Initialization-xu. If Wi belongs to hidden or input layer, its weights should be initialized as (O, PAN_IR) Output layers should have their weights initialized as N (O,
Optimizee Multipliers-ja. Output layer pre-activations should be multiplied by FAN_IN during the forward pass.
Optimizer Update Scaling-Ju.
The learned optimizer's update (eq. 2 is re-scaled as follows:
Wt-1 FAN_IN (aw, didexp (Aam) Ow, A1d exp (>2m)
Wi is hidden layer
Wt
W4-1
otherwise.
Where w is parameter of not made explicit for simplicity. For transfer to the largest width optimizees, it may also become necessary to re-scale numerical underflow constants (e) by fan_in suggested by (Everett et al. 2024). However, for the scales reported on by our experiments, we did not find this to be necessary.
the weight matrix, W,, and the dependence of d and
w,-1 is
We now prove that our parameterization satisfies the μP Desiderata (YYagg et al. 2022 Sec. 1.2.1). Proposition 4.1 (small Lonall_f__lae fc_lopt μP). Assume that the Learned Optimizer fo has the form small small_sc_lppl is fed with features given in Appendix and that during training the optimizee's ss parameters and input data become aligned, leading Law of Large Numbers (LLN) scaling, then the update, initialization, and pre-activation multppiier above is sufficient to obtain a Maximal Update Parametrization.
Proposition 4.2 (VeLO juP). Assume that & in Proposition 4.1 is generated using an LSTM with the input features described in Appendix A. and that during training the optimizee's 's parame- ters and input data become aligned, leading Law of Large Numbers (LLN) scaling, then the update, initialization, and pre-activation multiplier above is sufficient to obtain a Maximal Update Parametrization.
Proof. The proof is provided in Appendix
The yl.O parameterization be applied to any neural network architecture.
4

---

Published as a conference paper at ICLR 2026
5 EMPIRICAL EVALUATION
We construct suite of optimization tasks of varying width to evaluate the mettaggnneralizttinn properties of our faLOs meta- trained on MLPs vs per-task tumed μAAdam (Yang et al. 2022 per-task tuned SP AdamW (Loshchilov & Hutter 2019) and baseline SP LOs treinet Our main focus is to evaluate meta veetageneralizztionn to wider networks as this is a key weakness of learned optimizers in previous works. However, we also establish the generalization properties of jl.Os to deeper networks and longer training horizons. Please note that while pl.Os inherit the theoretical properties of μP for width scaling, our findings with respect deeper networks and longer training are purely empirical.
MLP tasks).
5.1 SETUP
Baseline LOs and plO.. The meta-training configuration of each learned optimizer is summarized in T Each learned optimizer (ours and the baselines) in our empirical evaluation is meta-trained using the multiple-width single-task meta-training recipe proposed in section 5.2.1 Notably, these tasks only include MLPs (see Fig M. while the hand-desinged optimizers in our study are tuned individually on each task. The SP baselines sheds light on whether simply varying the SP optimizee width during meta-training is enough to achieve generalization of the 1.0 to wider networks in SP. During meta- teta-training, we set the inner problem length to be 1000 iterations. Therefore, any optimization beyond this length is considered -i--o---dstribuiionn For all meta-training and hyperparameter tuning details, including ablation experiments, see section Cof of the appendix.
μAdam is strong hand- designed μP baseline. It follows the Adam μ paraamttrizatioon and does not use weight decay as this is incompatible with μP (Yagg et al. (2022). μAAdam is tuned on a width=1024 version of each task this is the width of the largest meta-training task seen by learned optimizers (see Table 5) We tune the learning rate (17) and accumulator coefficients (11 and (32) using grid search over more than 500 different configurations. This is repeated once for each task in our suite. Section of the appendix provides more details about the grid search including the values swept and the best values found.
AdamW W (oohhhhlovv & Hutter 2019) is a strong hand-designed SP baseline. It is tuned on the largest meta- tetta-triniig task seen by our learned optimizers (Table 5). AdamW tuned on a width=1024 version of each task as this is the width of the largest meta-training task seen by our learned optimizers (see T 5 We tune the learning rate (7)), accumulator coefficients (3, and (32), and weight decay (A) using a grid search over more than 500 different configurations. This is repeated once for each task in our suite. Section of the appendix provides more details about the grid search including the values swept and the best values found.
Evaluation tasks. Our evaluation suite includes 35 tasks spanning image classification (CIFAR- 10, ImageNet) using MLPs and Vision Transformers (ViTs) (Dosovitskiy et al.) 2020) and autoregressive language modeling with a decoder-only transformer on LMIB (Chelba et al. 2013). To create the tasks, we further vary image size (for image classification). width, and depth of the optimizee network, and the number of optimization steps. See Table 10 of the appendix for an extended description of all the tasks.
5.2 RESULTS
In the following sections, we evaluate different meta-training distributions for training jLOS (Sec. 5.2.1: ; we present results empirically verifying the pre-activation stability of our JdLS (Sec. 5.2.2): y we present the results of our main empirical evaluation of meta-generalizztion to wider networks (Sec. 5.2.2); study of pLOs generalization to deeper networks (Sec. 5.2.4); and a study of pl.Os generalization to longer training horizons (Sec. 5.2.4). All of our figures report training loss and report the average loss across 5 random seeds. Each seed corresponds to a different ordering of training data and a different initialization the optimizee. All error bars our plots report standard error across seeds. Standard error is Vx and n is the number of samples.
where is the population standard deviation
5.2. EVALUATING META- TRAINING DISTRIBUTIONS FOR μLOs
In je-transfer (Yang et al. 2022), hyperparameters are typically tuned on a small proxy task before being transferred to the large target task. In contrast, learned optimizers are typically meta-trained
5

---

Published as a conference paper at ICLR 2026
(a) SP Adam
(b) LOs
(c) LOO
4096
1.7
sta(x XX
127
1.0
167
102
2048
Model Widths (log Scale)
1024
512
(d) jAAdam
(e) 5LOss
(0 MLLM
stdOk: Xo]
1.5
256
1.7
100
128
200
Training Step (t)
Training Step
Training Step (t)
Figure 2: Layer 2 pre-activations behave harmoniously in μP for pLOs and μAdam alike. We report the evolution of coordinate- wise standard deviation of the difference between the initial (t 0) and t-th second-layer pre- pre-activations of an MLP during training for the first 500 steps of single run (the remaining layers behave similarly, see Sec. G We observe that all models parameterized in jP enjoy stable coordinates across widths, while the pre-activations of larger-width models in SP blow up after a number of training steps.
Vilds
VeL.e
Final Loss
Final Loss
5
Pinal Locs
al trané
Final Loves
2.5-
VoLLOw founs)
L0p
(ours)
1.5
236 512 1024 2048 6096 8192
128
236
512
1024 2045 4096 Width
8192
255
512
1824 Width
2048
4206
0.0
256 512
Width
Width
(a) Iteration 1000
(b) Iteration 5000
(c) X 64 ImageNet (d) 32 x Cifar10
Figure 3: Generalization beyond meta-training widths is severely limited without approach. Each point is the average final training loss over seeds with standard error bars. Subfigures (a) and (b) report the results of our meta-training task ablation the ImageNet-32 meta-training tasks at 1000 and 5000 steps. Subfigures (c) and (d) report the performance of pl. and vVeLOO on OOD datasets.
distribution of tasks. To verify the effectiveness of each approach for meta-training pLOSs, compare pls,s, meta-trained on a single width=128 MIP ImageNet classification task (see Tab. 5 pl.Oas, meta-trained optimizer targets 1000 step problems. We include equivalent standard parameterization baselines for reference (LOg and LOM)) Figure r the performance of each optimizer on suite of MLP classification tasks of increasing width. When training for 1000 steps (meta-training unroll length), we observe that μLO outperforms μLOs the width of the model is increased (Fig. B (a)). Moreover, we observe that there is discrepancy in performance between both models after 5000 steps (Fig. (b)), showing that meta-training with multiple tasks different widths has benefits for generalization to longer unrolls in addition to improved generalization to larger optimizees. Given the improved generalization of μLOM compared to pLOgg we adopt the multiple-width meta-training recipe part of method. Subsequent experiments (c.g., Figures 3 and is also effective for meta-training y VeLO.
width E {128,55 512, 1024) MLP ImageNet classification tasks. Each
will show that our recipe
5.2.2 EVALUATING PRE-ACTIVATION STABILITY
We verify that desiderata J.1 of Yang et al. 2022) is satisfied empirically. In Figure 2 report the evolution of the coordinate- wise standard deviation of the difference between initial (t=0) and current (t) second-layer pre-activations of MI.P during the first 500 steps of training for

---

Published as a conference paper at ICLR 2026
single trial. We observe that all models parameterized in enjoy stable coordinates across widths, suggesting that desiderata J.1 is satisfied by our parameterization. In contrast, the pre-activations of the larger MLPs in SP blow up immediately for SP Adam while they take noticeably longer for LOs and M- Section G of the appendix contains similar plots for the remaining layers of the MLP which show similar trends. In summary, we find, empirically, that pre-activations of fl.Os and AAdam are similarly stable across widths, while the activations of SP Adam and SP LOs both blow up but behave qualitatively differently.
5.2.3 META-GERT GENERALIZAZAIIIONN TO WIDER NETWORKS
Given our goal of improving LO generalization to unseen wider tasks, the bulk of our empirical evaluation is presented in this section. Specifically, evaluate the behavior of μl.Os as the width of tasks increases well beyond what was seen during meta-training. To accomplish this, we fix the depth of each task and vary the width (see Table I0 for a full list of tasks), leading testbed of 32 different tasks. We then train each task using the baselines and p-optimizers outlined in section S for 5000 steps for different random seeds. This involves training 1120 different neural networks. To make the results easily digestible, we summarize them by width and final performance in Figure g and by average optimizer rank Table I We also highlight the smooth training dynamics of our optimizers at the largest widths in Figure
Performance measured by final loss function of width. Figure compares the training loss after 1000 steps of SP learned optimizers to y-parameterized learned optimizers for different widths. This is shown in three subfigures for three MLP image classification tasks: (a) Imagenet 32 x 32 x 3 (IN32), (c) Imagenet 64 x 64 x 3 (IN64), and (d) Cifar- 10 32 x 32 x 3 (C10). Subfigure (a) shows the performance of learned optimizers the μLOs achieve lower final training loss as the width of the task is increased. In contrast, L diverges for widths larger than 2048. Subfigure (b) evaluates our pLOs on 64 x 64 x 3 ImageNet images (e.g., when the input width is larger). Similarly, we observe smooth improvements in the loss as the optimizee width increases for pl.Os, while their SP counterparts either diverge width 512 ML or fail to substantially improve the loss beyond width 1024 (VeLO Finally, Subfigure (c) shows the performance of our ,LOs on Cifar- 10 (smaller output width) as the width is increased. Similarly, we observe smooth improvements in the loss the width increases for pLOs, while their SP counterparts either diverge immediately at small widths (VeLO)z) or diverge by width 1024 MLOM).
larger versions of the meta-training tasks. We observe that
Training dynamics at the largest widths Figure the training curves of diffferent optimizers on the largest width tasks in our suite. Despite training for 5x longer than the maximum meta- training unroll length, our pl.s are capable of smoothly decreasing the loss for the largest out- of-distribution tasks in our suite. In contrast, the strong SP LO baselines diverge by 1000 steps (subfigures (C)) or fail to decrease the training loss (subfigure (e)), demonstrating the clear benefit of μLOS for learned optimization. Our pLOS also substantially best the per-task-tume AdamW and pAddam baselines (subfigures (a) and (b)), match the best performing hand-designed optimizer in subfigure (c), and nearly matches or outperforms the strongest hand deeiide baseline performance on far out-of-distribute demonstrate that, under our μl. meta-training recipe, learning optimizers that smoothly train large neural networks (e.g., demonstrated an 8B parameter model typically uses width=4096) is possible at low cost (μLOM is meta-trained for 100 GPU hours).
LM and ViT tasks (subfigures (d) and (c)). These results
Table 1: Summary of optimizer performance on large tasks. We report the average rank of different optimizers across the five tasks in suite. We evaluate each optimizer large- width tasks: Large (2048), XL (4096 for MLPs and 3072 for vit and LM), and XXL (largest size for each task see Tab To of the appendix). We bold the strongest, underline the second strongest, and italicize the third strongest average rank in each column. We observe that, across all iterations, μLO and 4 VeLO consistently obtain the best and second- best ranks for all tasks.
Loss at Ik steps
Loss at 3k steps
Loss at Sk steps
Optimizer
(I
Ool) (XI.)
OoD (XXI.
OnD (Large)
(XL)
OoD (XXI.)
Onl (Large)
Ool)
OoD (XXL.)
Adaon
3,00
4.00
2.40
3,80
eAdom
3.40
2,20
2,20
3.00
2.40
2.40
3.20
2.60
2.60
VeL.Our
4.60
4.00
5.00
5.40
5.40
5,80
6.00
5.40
5.80
LOwr
5.60
5.40
5.60
5.60
4.50
5.20
5.00
4,80
5.20
VeeLO (ours)
2.60
2.10
2,10
1.40
2.00
pLO (ours)
1,50
2,00
2.00
1.80
1.60
1.20
1,50
2.20
1,60
7

---

Published as a conference paper at ICLR 2026
7
Max Mera-cralning Utroll
2.0
1.5
1288
1,00
1,000
0.5
0.0
1000
2000
3000
6000
3000
1000
2000
3000
6000
5000
1000
2000
3000
4000
5000
Training Steps
Training Steps
Training Steps
(a) MLP IN32 W=8192
(b) MLP IN64 W=8192
(c) MLP C1O W=8192
10
6.5
AdamW
6.0
μAdam
VeLOm
Lozs
5.0
LOM
4.5
(ours) μVeLOM
(ours) μlO0M
3.5
3.0
2000
5000
1000
2000
4000
Training Steps
Training Stegs
(d) LM W=4096
(c) ViT W=4096
Figure 4: Evaluating generalization to wider networks for different tasks. All optimizers are meta- trained or hyperparameter tuned for 1000 inner steps (dotted red line), therefore, any optimization beyond 1000 steps is considered out-of-distributtin. We plot average training loss over seeds with standard error bars. We observe that μLOM and @VeLOMgeneraaa and all unseen tasks, unlike their SP counterparts which diverge or fail to make progress. μLOs outperform the extensively tuned Adam and μAdam baselines in subfigures (a).(b), match or surpass them in subfigure (c), and exceed or nearly match their performance on far out-of out-of-distributio LM and VIT tasks (subfigures (d) and (e)). Note that all Adam W and μAdam are tuned on smaller versions of each task, while our pLOS are only meta-trained on MLP tasks.
smoothly longer unrolls
Performance measured by average optimizer rank Table reports the average rank of different optimizers on out-of-distribution w.r.t. width tasks (Large (width 2048), XL (width 3072 for trans- former and 4096 for MLPs), and XXL (maximum width)). Each entry of the table corresponds to the optimizer's s average rank (within the 6 optimizers evaluated) over the 5 tasks in our suite: Cifar 10 MLP image classification, ImageNet 32 MLP image classification, ImageNet 64 MLP image classification, ImageNet 32 VIT image classification, and LM1B transformer language modeling. The optimizers are ranked by their training loss at the given iteration. We report average ranks for 1000 iterations mooneer length), 3000 iterations, and 5000 iterations. We bold the strongest, underline the second strongest, and italicize the third strongest average rank in each column. We observe that, across all iterations and all task sizes (Large, XL., XXL), either yLO j VeLO consistently obtain the best and second-best ranks for all tasks. The per-task-tune hand-designed baselines consistently occupy third and fourth rank, while the SP learned optimizer baselines perform worst, typically failing to optimize at this size. These results demonstrate that meta-training learned optimizers under the p-aarmetterization we propose and using our simple meta- meta-training recipe yields substantial improvements in meta-gent peeta-gnerralizttion (across various tasks and widths) over SP LOs (previous work) and strong per-task tuned hand-designed baselines.
5.2.4 EVALUATING CEETTAGGNERAAIZATIION BEYOND WIDTH
While our main focus is mettageneralliztionn to wider networks While the focus of our paper is improving the mntaagenerallzzation of LOs on wider tasks, it is also important to evaluate how these modifications to learned optimizer meta-training impact other axes of generalization. As such, we now study meta-gentt generalizatoon deeper networks and longer training. While provide strong AdamW W and μAdam baselines for reference, our focus will be to establish the relative performance fLOS to SP LOS. Note that μP theory leveraged by fl.Os specifically concerns transferring hyperparameters to
8

---

Published as a conference paper at ICLR 2026
7.5
7.0
Has truasyy Vari
AdamY
6.0
uAdam
LOss
VeLOw
(ours) jiVeOOw
40-
4.-
(ours) pl.Ox
30
1000
3386
3000
.020
5000
1260
3000
2006
1000
5003
4.-
1856
2003
3000
(003
2006
Training Stops
Training Stags
Training Steps
(a) VIT W=1024 D=16
(b) LM W=1024 D-16
(c) MLP IN32 W=1024 D=16
Figure 5: Evaluating generalization capabilities of pLOs to deeper networks. Our focus is on
comparing the generralnilnnzationn to deeper tasks pl.Os to SP LOs (all meta-trained exclusively on MLPs). We also report the performance per-task tuned AdamW and μAdam for reference. Each plot reports average training loss over 5 seeds with standard error bars. each case, pLOs show improved generalization and performance when compared to their SP counterparts.
pAdam
1.05
VeLOn
10w
(ours) μVeLOw
(ours) pl.Ox
19050 13000 20000 25000 Training Stego
5928 20000
15000 Trasning Steps
20000
22002
3000 20039
1093s 200300 Training Stops
20020
(a) VIT W=1024
(b) LM W=1024
(c) MLP IN32 W=1024
Figure 6: Evaluating meta-generalization to longer training horizons. Note that Adam and μAdam are evaluated on their tuning tasks here, while LOs are trained on MLPs. We plot average training loss over seeds with standard error bars. We observe that μLOs seamlessly generalize to training horizons 25 x longer than teatntraining. In contrast, the best performing SP LO fails to decrease training loss (a), decreases it but suffers instabilities (b), diverges after 8000 steps (c).
larger-width networks, not longer training horizons or deeper networks. Therefore, any improvements we observe are purely empirical.
Meta- gecteeeerrlaiztiionn to deeper networks In this section, we evaluate LO meta-generalization to deeper networks. Specifically, we increase the number of layers used in MLP, ViT. and LM tasks from 3 to 16, while keeping width=1024 within the range of tuning/meta-traiinn the performance of our learned optimizers on deeper networks. We observe that both pLO M and y ae optimize stably throughout and generally outperform their counterparts, 1.0 and Vel.O,, by the end of training on each task, despite being meta-trained on MI.Ps of exactly the same depth. Moreover, LOM immediately diverges when optimizing the deep MLP while uLO experiences instability. Similarly, VeLOM diverges on ViTs and Transformers, while μ pVVLO M performs well, especially ViTs. This is remarkable as, unlike width, there is no theoretical justification for JP's benefit to deeper networks. We hypothesize that juP' stabilizing effect on the optimizee' 's activations leads to this improvement in generalization (see Sec. F.1.2 for more details).
Figure repoots
Meta-generalization to longer training In this subsection, we empirically evaluate the capability of during meta-training. Specifi-
Il.Os to generalize much longer training horizons than those cally, we use μLO and LO as well as vVeLO and VeLO to train three networks with width 1024: a 3-layer MLP, VIT on 32 32 3 ImageNet and 3-layer Transformer for autoregressive language modeling on LMIB, Each model is trained for 25, 000 steps (25x the longest unroll seen at meta-training time). Figure reports the training loss averaged over random seeds. We observe that μLOM and j VeLoO, stably decrease training loss over time for each task, while LO and VeLO as fail to decrease training loss (a), decreases it but becomes unstable (b), or diverges after 8000 steps (c). While we are uncertain of the exact cause of this improved generalization, we hypothesize that it may be due to the improved pre- are-activation stability (see Sec. F.1.2 for more details). These results suggest that generalization to longer training horizons is another benefit of using pLOs

---

Published as a conference paper at ICLR 2026
6 LIMITATIONS
We have conducted systematic empirical study and shown strong results within the scope of our study, there are some limitations of our work. Specifically, (I) we do not meta-tram on tasks other than MLPs for image classification, (2) we do not provide an evaluation of models wider than 8192 (MLPs) and 5072/12288 (transformer hidden/FFN size) due to computational constraints in our academic environment, and (3) We did not include an oracle SP Adam W baseline whose hyperparameters are swept at every width due computational constraints in our academic environment.
7 CONCLUSION
We have theoretically and empirically demonstrated that it possible to obtain valid p parameterization for two state-of-the learned optimizer architectures. Under proposed meta- training recipe, meta-learned optimizers show substantial improvements in meta-generalization properties when compared to strong baselines from previous work. Remarkably, our fllOss, meta- trained only on MI.P tasks, surpass the performance of per-task-tuned hand-designed baselines in terms of average rank on wide OOD tasks. Moreover, meta-trained with our recipe generalize better to wider and, unexpectedly, deeper out-of-distribution tasks than their SP counterparts. When evaluated on much longer training tasks, observe that μLOs have a stabilizing effect, enabling meta-generallzztionn to much longer unrolls (25x maxi- mum meta-training unroll length). All of the aforementioned benefits of μLOs come zero extra computational cost compared to SP LOs. Our results outline a promising path forward for low-cost meta-training of learned optimizers that can generalize to large unseen tasks.
experiments also show that μLOs
In future work, it will be important to investigate the benefits of meta-learning optimizers under parameterizations other than pP that have been shown admit hyperparameter transfer (Everetl et al. 2024). Another important direction of inquiry is to investigate the meta-learning optimizers under parameterizations, like CompleteP (Dey et al., 2025). that have the potential to improve meta- generalization across depth and width. Finally, combining such parameterizations with improved meta-generalization and scalable meta-learning recipes is required for learning truly general-purpos optimizers.
ACKNOWLEDGMENTS
We acknowledge support from the Mila-Samsung Research Grant, FRQNT New Scholar [E.B.], the FRQNT Doctoral (B2X) scholarship [B.T.], the Canada CIFAR AI Chair Program [I.R.], and the Canada Excellence Research Chairs Program in Autonomous AI [.R.R. We also acknowledge resources provided by Compute Canada, Calcul Québec, and Mila. [E.O.] acknowledges funding from PEPR IA (grant SHARP ANR 1ER---3-PEAA-0000 He was granted access to the Al resources of IDRIS under the allocation 2025- AD011005884R)
REFERENCES
Diogo Almeida, Clemens Winter, Jie Tang, and Wojeiech Zaremba. generalizable approach to learning optimizers. arXiv preprint arXiv:2106.00953 2021. 3 26
Brandon Amos. Tutorial on amortized optimization for learning to optimize over continuous domains.
arXiv e-prints, pp. arXiv-2202, 2022.
S 26
Marein Andrychowiez, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016. 26
Charlie Blake, Constantin Eichenberg, Josef Dean, Lukas Balles, Luke Y. Prince, Björn Deiseroth, Andres Felipe Cruz-Salinas, Carlo Luschi, Samuel Weinbach, and Douglas Orr u-up: The unit-scaled maximal update parametrization, 2025. URL htttttts///aatttt http://awwwwvrg////../////
www.orrg/ss... abs 2407
17465
4
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample-eficiient reinforcement learning with stochastic ensemble value expansion. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.),
10

---

Published as a conference paper at ICLR 2026
Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8234-8222 2234-8244, 2018.
26
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn. One billion word benchmark for measuring progress in statistical language modeling. CoRR, abs/1312.3005, 2013. 5
Tianlong Chen, Weiyi Zhang, Zhou Jingyang, Shiyu Chang, Sijia Liu, Lisa Amini, and Zhangyang Wang. Training stronger baselines for learning to optimize. Advances in Neural Information Processing Systems, 33:7332-73443 2020. 26
Tianlogg Chen, Xiaohan Chen, Wuyang Chen, Zhangyang Wang, Howard Heaton, Jialin Liu, and Wotao Yin. Learning to optimize: A primer and a benchmark. The Journal of Machine Learning
Research, 23(1) 23(1):5522-862) 2022.
3 26
Nolan Dey, Shane Bergsma, and Joel Hestness. Sparse maximal update parameterization: A holistic approach to sparse training dynamics. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 33836 33862. Curran Associates, Inc., 2024. doi: 10.522022079017--006
URL
3b6aa fec (9 Fece:311188887753356666666333--3
4
Nolan Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Bill Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don't be lazy: Completep emables enables compute-efficient
deep transformers. CoRR, abs/2505.01618, 2025. URL sttps:///ai..rqq//
arXiv. 01618
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010. 11999, 2020. S
Katic E. Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jachoon Lee, and Jeffrey Pennington. Scaling exponents across parameterizations and optimizers. In Forty-first International Conference on Machine Learning, 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
10
James Harrison, Luke Metz, and Jascha Sohl-Dickstein. A closer look at learned optimization: Stability, robustness, and inductive biases. Advances in Neural Information Processing Systems, 35:3758-3773, 2022.
3 26
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. I
Taku Kudo and John Richardson. Sentencepiece: simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 November 4, 2018, pp. 66-71, Association for Computational Linguistics, 2018 27
Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. Convergence of adam under relaxed assumptions. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023a.
Oscar Li, James Harrison, Jascha Sohl-Dickstein, Virginia Smith, and Luke Metz. Variance-reduced gradient estimation via noise-reuse in online evolution strategies. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023b. S
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019. New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019.
25
11

---

Published as a conference paper at ICLR 2026
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. In International Conference Machine Learning, pp. 4556-4566 PMLR, 2019.
26
Luke Metz, C. Daniel Freeman, James Harrison, Niru Maheswaranathan, and Jascha Sohl-Dickstein. Practical tradeoff's between memory, compute, and performance in learned optimizers, 2022a. I
23 25 26
Luke Metz, James Harrison, Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury. Naman Agrawal, Ben Poole, Igor Mordatch, Adam Roberts, et al. Velo: Training versatile learned optimizers by scaling up. arXiv preprint urXiv:221..0976. 2022b.
Yurii E. Nesterov and Vladimir G. Spokoiny, Random gradient- minimization of convex functions.
Found. Comput. Math., 17(2):527-5666 2017.
26
Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. PIPPS: flexible model-based policy search robust to the curse chaos. In Jennifer G. Dy and Andreas Krause (e Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 4062 4071. PMLR, 2018.
26
Isabeau Premont-Schwarz, Jaroslav Vitkuu, and Jan Feyereisl. A simple guard for learned optimizers.
arXiv preprint arXivv2201.1243333 2022. 26
Herbert E. Robbins. A stochastic approximation method. Annals of Mathematical Statistics, 22:
400-407, 1951.
Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent
networks. Neural Compulation, 4(1):131-139, 1992. 26
Sebastian Thrun and Lorien Pratt. Learning learn. Springer Science & Business Media, 2012.
Paul Vicol. Low-variance gradient estimation unrolled computation graphs with es-single. In
International Conference on Machine Learning, pp. 35084-351199 PMLR, 2023. g 2G
Paul Vicol, Luke Metz, and Jascha Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 10553-10563. PMLR, 2021.
26
Olga Wichrowska, Niru Maheswaranathan. Matthew W Hoffman, Sergio Gomez Colmenarejo. Misha Denil, Nando Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. In International conference on machine learning, pp. 3751-3760. PMLR, 2017. 26
Greg Yang. Tensor programs I: wide feedforward or recurrent neural networks any architecture are gaussian processes. CoRR, 0tos/190..12478 2019. 27
Greg Yang. Tensor programs II: neural tangent kernel for any architecture. CoRR, abs/2006.14548 2020a.
2
Greg Yang. Tensor programs III: neural matrix laws. CORR, abs/2009. 10685, 2020b.
Greg Yang and Edward J. Hu. Tensor programs IV: feature learning in infinite-width neural networks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on
Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of
Machine Learning Research, pp. 11727-11737 PMLR, 2021. B 27
Greg Yang and Etai Littwin. Tensor programs ivb: Adaptive optimization in the infinite-widh limit. CoRR, abs/2308 abs/2308.018844 01814, 2023.
27
Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, David Farhi, Jakub Pachocki, Xiaodong Liu. Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. In NeurlPS 202/, March 2022.
12

---

Published as a conference paper at ICLR 2026
Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs VI: feature learning in infinite depth neural networks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
29
Junjie Yang, Tianlong Chen, Mingkang Zhu, Fengxiang He, Dacheng Tao, Yingbin Liang, and Zhangyang Wang. Learning to generalize provably in learning to optimize. In International Conference on Artificial Intelligence and Statistics, pp. 9807-9825. PMLR, 2023.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc Emily B. Fox, and Roman Garnett (eds.), Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver. BC, Canada, pp.
12360- 12371, 2019. URL httptttttttttttttteeeet /ttpstpp////prreeelinn hew.eeeedddinnnnnnness.s..... / 1e8a1 1 ca 5 Abst html
15
13


Reviews:
[{"review_id": "Gr5oQOdHte", "reviewer": "Reviewer_tB8g", "summary": "The authors of this submission investigate the generalization ability of learned optimizers in zero-shot-like scenarios, where an optimizer trained on one architecture can be directly applied to unseen architectures. To achieve this, models are initialized following the $\\mu$P (Maximal Update Parameterization) principle. The authors leverage their hyperparameter reuse property, which allows optimal parameters discovered on one architecture to be transferred to architectures with the same depth but different widths via a well-defined mapping. Under this framework, learned optimizers are treated as a form of hyperparameter, enabling transfer to new architectures. The proposed approach is evaluated across various neural network training tasks and demonstrates promising results.", "questions": "1. Figure 5 suggests meta-overfitting: the learned optimizer performs well on meta-training tasks but fails to generalize effectively to unseen tasks.\n\n2. Across the experiments, only training curves are presented. It remains unclear whether the learned optimizer contributes to improved generalization performance beyond faster or more stable training.\n\n3. The level of novelty appears limited. The proposed $\\mu$P parameterization closely follows the original $\\mu$P paper, and several propositions in the submission echo theoretical results already established in prior work.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is interesting and well-grounded. The idea of viewing optimizers as hyperparameters that can generalize across $\\mu$P-initialized networks of varying widths is both novel and conceptually appealing.\n\n2. The learned optimizers show encouraging results in zero-shot-like transfer scenarios.", "weaknesses": "1. Figure 5 indicates the meta overfitting with the trained optimiser performing well on the meta-train tasks while failing to generalise well on the unseen tasks. \n\n2. From all the experiments, only the learning curve of training stages is illustrated, with the question of whether the learned optimizer leads to advanced generalization ability not answered. \n\n3. Limited novelty: the Mup parameterization proposed in this submission is very close to a direct application of the original $\\mu$P paper. In addition, the propositions in submission are similar to the theoretical results from the $\\mu$P papers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mYJDTAyUAt", "reviewer": "Reviewer_PtQP", "summary": "The paper derives muP for the small_fc_lopt and VeLO learned optimizer architectures. The authors then test the resulting metageneralization, meta-training on ImageNet classification with MLPs and applying the optimizers to other instances of that as well as using transformers for that and language modelling. They empirically find that this achieves hyperparameter transfer not only for width, but also to a significant degree for depth and training duration, outperforming other learned optimizers and also competing with task-tuned standard/hand-designed ones like AdamW and μAdam. There is also a short subsection on pre-activation stability, and similar investigations in the appendix.", "questions": "How much compute went into the hyperparameter sweeps for the baseline optimizers? The values swept looked reasonable, but I think it'd be useful context to compare with the meta-training figures, to see how much the LOs need to be used to make up for that (mainly thinking in seemingly underparametrized settings where that may be the right frame, like language modelling).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "While I think applying muP to this domain was inevitable, the authors make a good case that the failure to meta-generalize is the major blocker for LOs, and that they make substantial improvements there.\n\nI thought the experiments were reasonable, namely the baselines, and the paper was clear throughout, including the maths (though I haven't gone line by line in the proofs).", "weaknesses": "I think the findings of [\"Scaling Exponents Across Parameterizations and Optimizers\"](https://arxiv.org/abs/2407.05872) should've made an appearance somewhere, since they indicate that standard parametrization can also achieve hyperparameter transfer. They also show that (in larger problem instances than here) that epsilon should be tuned in Adam.\n\nAlso, since depth scaling is mentioned (albeit as a bonus), I think the related works and perhaps some experiments would ideally address more heuristic methods for scaling with depth (and width, or rollout length, if the LO community has investigated this at all). In NeurIPS 2025 (i.e., contemporary, ignoring arXiv) work, [ComputeP](https://arxiv.org/abs/2505.01618) is quite relevant\n\nMinor fix:\n* §B.1 \"When using a schedule, we always use linear warmup and cosine annealing with\" just cuts off", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XbA7XfhmxO", "reviewer": "Reviewer_P3eP", "summary": "This paper addresses the challenge of learned optimizers (LOs) struggling to optimize unseen tasks, particularly when dealing with networks larger than those encountered during meta-training. The authors introduce the Maximal Update Parametrization (P) for two state-of-the-art LO architectures and propose a new meta-training recipe for parameterized LOs. Empirical results show that LOs trained with this recipe significantly improve meta-generalization to larger unseen tasks, outperforming standard parametrization (SP) LOs within the same compute budget. Additionally, the paper observes enhanced meta-generalization for deeper networks and surprisingly effective generalization for longer training horizons compared to SP LOs.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces the Maximal Update Parametrization to address the meta-generalization problem in learned optimizers. The idea is novel and interesting.\n\n2. The experimental results are thorough and effectively demonstrate the method’s validity.", "weaknesses": "1. I recommend adding experiments with convolutional neural networks (CNNs). Although this limitation is mentioned, I believe CNNs are currently mainstream in neural network research, and testing the method on them would further strengthen the validity of the findings.\n\nI am not very familiar with this field, so I will rely on the feedback from other reviewers for the final score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8HPIPgLY91", "reviewer": "Reviewer_oqd9", "summary": "This paper introduces \\mu-parameterized Learned Optimizers to address the meta-generalization limitations of existing learned optimizers when optimizing neural networks wider than those encountered during meta-training. \n\nAuthors derive the Maximal Update Parametrization  for some architectures and propose a straightforward meta-training recipe. They find that proposed \\mu-P parameterization significantly improves meta-generalization to wider, deeper, and longer unseen tasks compared to Standard Parameterization.", "questions": "I also have some qns for the authors:\n- Given that µLOs exhibit improved generalization to deeper networks and longer training horizons, and these improvements are currently hypothesized to be due to activation stability, do the authors plan to conduct further theoretical analysis to formalize these observations?\n- The µLOs were meta-trained only on MLP tasks. How do the authors anticipate the performance and meta-training cost of µLOs would change if they were meta-trained on a more diverse set of architectures, including ViTs and Transformers?\n- While the paper acknowledges the absence of an oracle SP AdamW baseline, could the authors discuss any ongoing efforts or future plans to conduct such a computationally intensive comparison to provide an even stronger benchmark?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has following strengths in my opinion:\n- tackles the fundamental challenge of meta-generalization in learned optimizers, which is a major hurdle for their practical adoption.\n- derivation of µP for LOs provides a solid theoretical basis for the proposed approach\n- results show substantial improvements in meta-generalization across various axes (width, depth, training horizons) compared to  Standard Parametrization\n- authors emphasize that the benefits of µLOs come at zero extra computational cost during meta-training compared to SP LOs, which is a crucial practical advantage\n- paper provides a clear description of the meta-training recipe, evaluation setup, and baselines, making the work reproducible\n- good suite of evaluation tasks and architectures", "weaknesses": "My main concerns are following:\n- While the improved generalization to deeper networks and longer training horizons is a strong empirical result, the explanations for these benefits are currently speculative e.g. mentions of pre-activation stability. Wondering if this could be made more precise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)