Narrowing the Focus: Learned Optimizers for Pretrained Models

## Abstract

Abstract In modern deep learning, the models are learned by applying gradient updates using an optimizer, which transforms the updates based on various statistics. Optimizers are often hand-designed and tuning their hyperparameters is a big part of the training process. Learned optimizers have shown some initial promise, but are generally unsuccessful as a general optimization mechanism applicable to every problem. In this work we explore a different direction: instead of learning general optimizers, we instead specialize them to a specific training environment. We propose a novel optimizer technique that learns a layer-specific linear combination of update directions provided by a set of base optimizers, effectively adapting its strategy to the specific model and dataset. When evaluated on image classification tasks, this specialized optimizer significantly outperforms both traditional off-the-shelf methods such as Adam, as well as existing general learned optimizers. Moreover, it demonstrates robust generalization with respect to model initialization, evaluating on unseen datasets, and training durations beyond its meta-training horizon.

## 1 Introduction

The optimization loop forms the backbone of machine learning algorithms. The choice of optimizer and its hyperparameters heavily influences the final model’s performance. Despite its importance, optimizer selection often relies on heuristics and domain-specific knowledge. For instance, Adam optimizer (Kingma & Ba, ) excels with language problems, but struggles with image classification problems.
The potential of second-order methods for stochastic data remains largely untapped, except for a few notable exceptions (Martens & Grosse, ; Gupta et al., ). Every year, more and more optimization methods are proposed, making it increasingly challenging for practitioner to select the best one (Choi et al., ; Schmidt et al., ).

Learned optimizers or learning-to-learn methods (Schmidhuber, ; Bengio et al., ; ) offer a potential solution to this challenge by automating the optimizer selection process. These methods automate this process by either learning the hyperparameters of existing optimizers or developing entirely new optimization algorithms. However, two significant challenges limit widespread adoption of learned optimizers: meta-optimization difficulties, and meta-generalization.

Meta-optimization difficulties arise from the sensitivity of gradients in bi-level optimization with a high number of training iterations (aka optimization horizon) (Metz et al., ). Evolutionary methods can help mitigate this because they do not rely directly on gradients, and thus are less susceptible to gradient sensitivity issues. However, the complexity of the problem increases with the length of the optimization horizon. Short horizon bias (Wu et al., ) constitutes another problem, where meta-optimization over a specified time horizon introduces a bias that prevents it to be applicable to a longer time horizon.

Meta-generalization refers to a learned optimizer’s ability to perform well on novel tasks it was not specifically trained for. One approach to achieving this is to train the optimizer on the widest possible range of tasks. Previous research focused on developing a single, adaptable algorithm by training it on a very broad domain. For example, Versatile Learned Optimizers (VeLO, Metz et al., ) was trained on hundreds of different problems, ranging from simple linear regression to reinforcement learning. However, this “one-size-fits-all” approach has significant drawbacks. The computational cost of training such an optimizer is immense, requiring $4\,000$ TPU-months in the case of VeLO. Furthermore, a single optimizer struggles to effectively adapt to the vast array of loss surfaces encountered across such a diverse set of tasks. This makes it challenging to achieve consistently high performance across all tasks.

Instead of aiming for broad applicability, we focus on the increasingly relevant domain of fine-tuning pretrained models. This specialization allows us to develop learned optimizers in a more targeted way. By narrowing the scope of the meta-training domain, the learned optimizer can specialize and excel in the specific types of tasks it is designed for. Fine-tuning also simplifies the optimization problem by leveraging existing knowledge encoded in pretrained checkpoints (Wei et al., ) and is typically done for a relatively few iterations, since only a small alignment with the current task is needed to achieve good results.

To achieve this, we introduce L3RS (Learned Layer-wise Learning Rate Scheduler, pronounced “lers”), a novel learned optimizer designed specifically to leverage the performance of a set of predefined base optimizers within a narrower domain. L3RS uses a variety of performance features, such as adaptive exponential moving averages (EMA) across different time scales. EMAs, or model averaging, have demonstrated improved generalization in real-world applications (Tarvainen & Valpola, ; Izmailov et al., ). Based on these features, L3RS produces layer-wise updates as a linear combination of the predefined base optimizers, along with a corresponding learning rate for each layer.

Our contributions are as follows.

- •
We propose a novel learning-to-learn method that leverages the strengths of multiple base optimization methods. This optimizer is simple and intuitive, having only a fraction of parameters compared to other black-box learned optimizers.
- •
We demonstrate that meta-training a learned optimizer on a narrow domain results in an optimizer that can outperform existing methods, with more that $50\%$ speedup compared to another learned optimizer baseline, and more than $200\%$ speedup compared to best performing traditional optimizers.
- •
We evaluate the meta-generalization of the proposed optimizer across several components: generalization to longer training horizon, different pretraining and model initializations, and a different evaluation dataset.

## 2 Preliminaries

Figure: Figure 1: *Left:* Inner loop evaluation. Given a task $T:=\left\{\theta_{0},\{D^{t}_{K}\},D^{e}\right\}$, the learned optimizer $f_{\psi}$ uses optimization statistics $\Phi$ to update model parameters starting from $\theta_{0}$ for each of $\{D^{t}_{K}\}$ batches. After $K$ update steps, the final model parameters are evaluated on the evaluation set $D^{e}$ using meta-loss $L_{\psi}(\theta_{K},D^{e})$. *Right:* Outer loop NES meta-training iteration. Given a task $T$ and meta-parameters $\psi_{T}$, Gaussian noise is added to $\psi_{t}$ to produce a number of candidates equal to the population size $c$, $(\psi_{t,0},...,\psi_{t,c})$. An inner loop evaluation is performed on the given task for all candidates. The fitness of each candidate is then used to perform an NES update step, resulting in the next learned optimizer parameters, $\psi_{t+1}$.
Refer to caption: x1.png

Consider a loss function $L(\theta_{n},X)$, parameterized by weights $\theta_{n}$ for a given data $X$ at cetain optimization step $n$. Learned optimizers $f_{\psi}$, parameterized by $\psi$, provide a direction to update the model’s weights $\theta_{n}$ as

$$ $\theta_{n+1}=\theta_{n}+f_{\psi}(\Phi;\theta_{n},X),$ (1) $$

where $\Phi$ is a collection of features that describes the current (or past) optimization statistics, such as the gradient or the loss value. For example, the simplest learned optimizer might simply optimize the learning rate $\lambda$ of SGD, as $f_{\psi}(\Phi;\theta_{n},X)=\lambda\nabla L(\theta_{n},X)$, in which case $\Phi:=\{\nabla L\}$ and $\psi:=\{\lambda\}$. In case of Adam, the optimization parameters would be $\psi:=\{\lambda,\beta_{1},\beta_{2}\}$, where $\lambda$ is the learning rate, $\beta_{1}$ and $\beta_{2}$ are the exponential decay rates for the first and second moments.

In Figure 1 we show both the inner and the outer loop of a typical meta-training process. We train $\psi$ on a distribution of tasks from a given dataset. A task is a set $T:=\left\{\theta_{0},\{D^{t}_{K}\},D^{e}\right\}$, where $\theta_{0}$ represents initial model weights (or a distribution of initial model weights), $\{D^{t}_{K}\}$ is a sequence of $K$ training batches, and $D^{e}$ represents an evaluation batch. A batch typically consists of data $X$ and, for classification problems, may also include labels. We assume that $\theta_{0}$ is given as input, and we have no control over its generation process (unlike initialization-based meta-learning approaches, such as MAML, Finn et al., ).

Learned optimizer parameters $\psi$ are evaluated by performing $K$ update steps (equation 1) on every batch from $D^{t}_{K}$, to get $\theta_{K}$ and calculating the loss using the evaluation batch $L_{\psi}(\theta_{K},D^{e})$.

We use Natural Evolution Strategies (NES, Salimans et al., ) to meta-train the learned optimizers. We define a task meta-loss $L_{\psi}^{M}(T)=L_{\psi}(\theta_{K},D^{e})$, the loss on the evaluation batch after K training steps of a task $T=\left\{\theta_{0},\{D^{t}_{K}\},D^{e}\right\}$. We then define a fitness function $F(\psi,(T_{1},...,T_{b}))=-\frac{1}{b}\sum^{b}_{i=1}L_{\psi}^{M}(T_{i})$, using a batch of $b$ tasks for each generation. NES is used to maximize the fitness function during meta-training.

After the learned optimizer is trained, it can be evaluated on a new task distribution $\widetilde{T}:=\{\widetilde{\theta}_{0},\{\widetilde{D}^{t}_{K}\},\widetilde{D
}^{e}\}$.

## 3 Motivation and Related Work

We can separate the optimizers into two main categories. First category, so called black-box optimizers (Li & Malik, ; Andrychowicz et al., ; Wichrowska et al., ; Lv et al., ; Sandler et al., ; Metz et al., ; ; ), learns an update function $f_{\psi}$ from scratch using a custom inner optimization loop. These methods often have a large number of parameters, making them prone to overfitting and stability issues (Harrison et al., ). While recent work has explored using Transformers as a meta-learner architecture (Chen et al., ; Moudgil et al., ; Jain et al., ), they have not yet shown a significant advantage over traditional optimizers or other learned optimizer architectures.

A notable example of black-box optimizer is VeLO, which uses per-tensor HyperNetworks (Ha et al., ) represented by 512-wide LSTMs to generate parameters of per-parameter multi-layer perceptrons (MLPs). To compute the parameter updates, it passes the per-parameter features through the generated MLP network consisting of 2-hidden layer, 4-hidden MLP. The input to the HyperNetwork and MLP vary slightly, but generally represent the training dynamic of the optimization. VeLO is designed to be general and has a lot of parameters that need to be learned, which makes meta-training very expensive. We envision an ideal optimizer would be lean and ideally fast to train on a narrow task domain.

Other techniques utilize algorithm discovery as a program search (Bello et al., ; Wang et al., ; Zheng et al., ; Chen et al., ), where the symbolic optimizers are found using a tree search of predefined operations (such as gradient and momentum).

The second category of meta-optimizers learns a higher-level meta-component on top of existing hand-designed optimizers. Such approaches include learning to adapt the hyperparameters of an underlying optimizer Daniel et al. (), or learn a schedule for the learning rate (Xu et al., ; ). In addition, these methods can include adapting the optimizer’s parameters directly (Shaban et al., ). Hypergradient methods (Maclaurin et al., ; Baydin et al., ; Grazzi et al., ; Moskovitz et al., ) perform a hyperparameter search on-the-fly during optimization.

Our proposed approach strikes a balance between flexibility of black-box minimizer and the stability of learning hyperparameters of known optimizers. We leverage the benefits of hand-designed optimizers while incorporating the adaptability of learned components using a simple small neural network for learning the per-layer learning rate. Unlike existing methods that learn a fixed set of hyperparameters for a single existing optimizer, L3RS learns to utilize a set of base-optimizers. Moreover, during meta-training we also search over the hyperparameters of the base-optimizers.

Almeida et al. () is probably the closest algorithm to our setting. It uses an LSTM controller to adjust the hyperparameters of a hand-designed inner-optimizer, which uses a collection of hand-designed parts from different known optimizers. A major difference with our method is that L3RS operates per-layer and that our algorithm wraps existing optimizers out-of-the box, without the need to manually add features.

Prémont-Schwarz et al. () also propose an in-between approach, where the learned optimizer falls back to the base-optimizer. However, their goal was to ensure convergence, not necessarily to improve the performance.

The L3RS architecture was also partially motivated by Learning Rate Grafting (Agarwal et al., ), which demonstrates the utility of isolating the direction of an optimizer from its magnitude.

Similar to our paper Landro et al. () also propose an algorithm that combines Adam and SGD as a linear combination of their update direction. However, their method differs significantly in two key aspects: (1) they do not provide a mechanism for learning the mixing coefficient, instead treating it as fixed hyperparameters, and (2) their combination is applied globally across all layers, while our proposed method employs a layer-wise adaptive strategy.

## 4 L3RS : Learned Layer-wise Learning Rate Scheduler

Figure: Figure 2: *Left:* L3RS applied to a single layer of the target model. The MLP receives time features, EMA features of the global loss, layer gradient norm and layer parameter norm, the target layer embedding, and the norm of each direction provided. The MLP outputs the weighting for each direction ($\mu_{p}$) as well as the final update norm ($\lambda$). *Right:* L3RS is applied at every layer of the network independently with the same MLP weights but different layer embeddings.
Refer to caption: x2.png

### 4.1 Model architecture

Given a set of update vectors $d_{p}$ from $P$ input base optimizers (e.g. SGD or Adam) L3RS computes a model parameter update step per layer $l$ using:

$$ $\textstyle{\Delta\theta^{(l)}=\lambda^{(l)}\sum_{p=1}^{P}\mu_{p}^{(l)}\hat{d}_ {p}^{(l)},}$ (2) $$

where $\lambda^{(l)}$ is a layer-specific update $l_{2}$-norm, $\mu_{p}^{(l)}$ are normalized mixing coefficients for the base optimizers’ directions $\sum_{p=1}^{P}\mu_{p}^{(l)}=1$, and $\hat{d}_{p}^{(l)}$ are normalized optimizers’ direction $\hat{d}_{p}^{(l)}=\frac{d_{p}^{(l)}}{\left\lVert d_{p}^{(l)}\right\rVert_{2}}$.

Figure 2 shows the main input and output components of L3RS as well as provides a flow-chart on how the weights of the underlying model $\theta$ are updated.

In order to learn $\lambda^{(l)}$ and $\mu_{p}^{(l)}$, we provide L3RS with a feature vector, comprising the adaptive EMA, time, embedding features as well as the magnitude of the update directions defined below. This input is then processed by two fully connected layers with sizes $32$ and $16$, using ReLU activation. The model outputs logits $z^{(l)}\in\mathbb{R}^{P+1}$, from which we reconstruct $\mu_{p}^{(l)}=\frac{\exp(z_{p}^{(l)})}{\sum_{j=1}^{P}\exp(z_{j}^{(l)})}$ and $\lambda^{(l)}=\exp(z_{P+1}^{(l)})$. The hyperparameters of the input optimizer (e.g. Adam’s $\beta_{1}$ and $\beta_{2})$) are also jointly meta-learned along with the L3RS’s model parameters. This enables co-adaptation of the layer-wise scaling and the underlying optimization algorithm.

#### Adaptive Exponential Moving Averages (EMA).

In order to capture both short-term and log-term performance trends, for each layer $l$, we maintain three EMAs with different smoothing factors $\gamma\in\{0,0.9,0.99\}$. EMAs are updated recursively as $a^{(l)}_{i,k+1}:=\gamma_{i}a^{(l)}_{i,k}+(1-\gamma_{i})\xi^{(l)}$, where $a^{(l)}_{i,k}$ denotes the $i$th EMA for layer $l$ at time step $k$, and $\xi^{(l)}$ represents a layer-specific statistic of interest. Specifically, for a given layer we track the following:

- •
*Log $l_{2}$-norm of weights.* $\log(\left\lVert w^{(l)}\right\rVert_{2})$, where $w^{(l)}$ are the weights for layer $l$.
- •
*Log $l_{2}$-norm of gradients.* $\log(\left\lVert g^{(l)}\right\rVert_{2})$, where $g^{(l)}$ are the gradients for layer $l$.
- •
*Loss.* The loss of the overall model (shared across all layers).

To ensure unbiased estimates, especially during the initial time steps, we apply a bias correction to the EMAs before incorporating them as input features: $\tilde{a}^{(l)}_{k}=\frac{a^{(l)}_{k}}{1-\gamma^{k}}.$

#### Time features.

Inspired by VeLO (Metz et al., ), we propose incorporating the following time features for the explicit modeling of temporal dynamics of the optimization progress. These features enable the optimizer to adapt its strategy based on the stage of the training process. These features are generated using $k$, the current training step, and $K$, the number of total training steps. Similar to some learning rate schedulers, $K$ must be given to the optimizer at the beginning of training so that these time features can be generated.

- •
*Relative time features.* $\tanh\left(10\left(\frac{k}{K}-\alpha_{i}\right)\right)$, where $\alpha_{i}$ are $11$ linearly scaled reference points between $0.0$ and $1.0$, inclusive. These features capture the relative progress through a sequence, with the hyperbolic tangent function providing a smooth, bounded representation. Figure 8 in the Appendix shows the values of these features throughout training.
- •
*Absolute time features.* $\tanh\left(\log\left(K\beta_{i}\right)\right)$, where $\beta_{j}$ are $4$ log-scaled scaling factors between $0.0001$ and $0.1$. These features provide a logarithmic encoding of the total duration, which can be useful for appropriate learning rate scaling. For longer time horizons these features can be extended such that $\max(\beta)>\max(K)$. Figure 8 in the Appendix shows the values of these features for various total training lengths $K$.

#### Embedding features.

A 16-dimensional embedding vector $\psi_{emb}$ is meta-learned for each layer, allowing the optimizer to learn specialized per-layer dynamics.

#### Base-optimizer direction magnitude.

For each base optimizer, we provide the log $l_{2}$-norm of its update: $\log(\left\lVert d_{p}\right\rVert_{2})$.

### 4.2 Comparison with VeLO

**Table 1: Memory and compute overhead for various optimizers using a ResNet-34 model with 25 output classes. Memory Overhead is the ratio of optimizer state size to model parameter size. Compute overhead is the compute cost ratio of the optimizer update step to a full training step for a batch of 64 images. Compute cost is calculated using Jax compilation statistics.**
| Optimizer | Memory Overhead | Compute Overhead |
| --- | --- | --- |
| SGD | $0.0\times$ | $0.003\%$ |
| Adam | $2.0\times$ | $0.019\%$ |
| L3RS | $2.0\times$ | $0.057\%$ |
| VeLO | $4.0\times$ | $0.920\%$ |

VeLO was designed with an efficient hypernetwork-style architecture. The majority of the computation cost is reduced to per-layer LSTM networks, which then generate cheaper per-parameter networks which output the update step direction. L3RS is able to leverage smaller MLP networks for the per-layer operations, and can rely on base optimizers rather than generating the update step direction manually. These design choices result in a learned optimizer which will typically have two to three orders to magnitude fewer parameters compared to VeLO.

## 5 Experiments

To evaluate the performance of our proposed L3RS optimizer, we designed a series of fine-tuning image classification experiments using the ResNet-34 model.

#### Fine-Tuning.

Our experiments focus on fine-tuning pretrained models. In particular, we are interested in optimization horizons on the order of hundreds or thousands of steps. While training can certainly extend further, this training regime is relevant in cases where the amount of training data or high computational costs can be a limiting factor.

#### Model Choices.

While our proposed L3RS can wrap any number of base-optimizers, we found that even a simple version that combines together only Adam and SGD (without momentum) directions performs well. For the remainder of this paper, when we refer to L3RS, we specifically mean this Adam and SGD combination unless we specify otherwise.

During inference, the memory footprint of this L3RS optimizer state is $2\times$ the model parameters (all from Adam). VeLO uses a memory state of $4\times$ the model parameters (plus additional memory from $3$ AdaFactor style accumulators). Table 1 shows a comparison of the memory state requirements for various optimizers, as well as the compute overhead for each.

We derive updates separately for every convolutional, dense or batch normalization layer of the model, including separate updates for kernels and biases. For ResNet-34, this results in $L=111$ components for which we compute learning rate $\lambda^{(l)}$ and mixing coefficients $\mu_{p}^{(l)}$, for $l=1,\dots,L$. With $P=2$ optimizers, L3RS outputs $333$ parameters per iteration.

For NES meta-training, we use a smooth exponential decay for both meta-learning rate $\alpha$ and the Gaussian noise standard deviation $\sigma$ by 0.5 every 500 generations. We use the same antithetic sampling and fitness transformations as in Salimans et al., . We use a population size of $32$, meta-batch size of $4$ and train for $2000$ generations for all our experiments (except the ablation study). We parallelize the evaluation of each candidate in the population, using a total of $32$ A100 GPUs for four days.

#### Task Distribution and Meta-Training.

We trained the learned optimizer using ImageNet dataset in the following way. We partition the ImageNet dataset into three distinct subsets: pretraining, meta-train (ImageNet25), and meta-test (ImageNet25Eval). The first $500$ classes are used to pretrain a model using a conventional off-the-shelf algorithm. The pretrained model serves as an initialization to our method. The remaining $500$ classes are randomly divided into meta-train and meta-test sets. Details on checkpoint pretraining are provided in Appendix A.

A task within our meta-learning framework is constructed by randomly sampling $25$ classes from either the ImageNet25 or ImageNet25Eval set. The selected classes are used to generate $K$ training batches with $64$ samples each and $1$ evaluation batch with $256$ samples, using the train and validation splits respectively. During meta-training, the number of training batches $K$ is uniformly sampled from a range of $10$ to $500$, simulating diverse fine-tuning scenarios.

We meta-train the L3RS optimizer and the VeLO optimizer on the ImageNet25 meta-train task distribution using NES as described above.

#### Meta-Evaluation and Baselines.

To evaluate the performance, we check for in-distribution and out-of-distribution generalization. For *in-distribution evaluation* we use the same initialization as the meta-training ($\widetilde{\theta}_{0}=\theta_{0}$) and use ImageNet25Eval tasks for meta-test. For *out-of-distribution evaluation* we use the Places (López-Cifuentes et al., ) dataset. We create a separate Places initialization by pretraining ResNet-34 on the first $150$ classes of Places. We use the remaining classes as a meta-test set (Places25Eval). Importantly, this dataset is entirely novel and unseen during the meta-training, which allows us to evaluate out-of-distribution performance of the optimizers.

Figure: Figure 3: Meta-evaluation of L3RS meta-trained on ImageNet25 for $10$ to $500$ steps along with various benchmarks. Performance is compared to VeLO (*left*), Adam with cosine learning rate (*center*), and Adam with a constant learning rate (*right*). Each marker represents model evaluation at that number of steps. Solid lines indicate the number of steps for in-distribution evaluation, while dashed lines indicate generalization to more steps than meta-training. A. In-domain Generalization. Both initialization and evaluation are on ImageNet. B. Out-of-Domain Initialization. Initialized on ImageNet, evaluated on Places25Eval. C. Out-of-Domain Evaluation. Initialized on Places, evaluated on ImageNet25Eval. D. Out-of-Domain Init & Eval. Both initialization and evaluation are on Places dataset. E. Random Initialization. Random initialization, evaluated on ImageNet25Eval dataset. F. Speedup of L3RS in iterations. For in-domain generalization, this shows how much faster L3RS achieves a given accuracy compared to the baselines.
Refer to caption: x3.png

We compare L3RS against the following baselines:

- •
VeLO (ft). An instance of VeLO meta-trained using the same method as L3RS.
- •
VeLO (og). The original pretrained VeLO model (Metz et al., ), representing a general-purpose learned optimizer.
- •
VeLO (og, Head). The original pretrained VeLO model applied to only the final model layer (freezing the rest of the model).
- •
Adam (cosine). Adam optimizer with a cosine learning rate schedule.
- •
Adam (cosine, Head). Adam optimizer with a cosine learning rate schedule applied to only the final model layer (freezing the rest of the model).
- •
Adam (const). Adam optimizer with constant learning rate.
- •
Adam (const, Head). Adam optimizer with constant learning rate appled to only the final model layer (freezing the rest of the model).

For optimizers which depend on the number of training steps $K$, such as Adam (cosine), VeLO (ft) and L3RS, we perform separate evaluations across various $K$ values to assess their performance both within and beyond the meta-training regime.

For all meta-evaluations we average the results of 100 tasks sampled from the specified task distribution. The average and standard deviation of eval accuracy across the sampled tasks is reported. All figures are reproduced using loss instead of accuracy in Appendix A.

#### Results.

Figure 3 shows the main result. Our goal is to assess the performance of the learned optimizer under various conditions, for both in and out of distribution scenarios. (A) Represents in-distribution evaluation, where the initialization, evaluation dataset, and the number of steps match the meta-training regime. We also explore out-of-distribution performance by extending the number of evaluation steps beyond the meta-training range (indicated by the dashed line in all plots), changing the evaluation dataset to Places25Eval (B), modifying the initialization to utilize a model pretrained on Places (C), or using both the Places checkpoint and Places25Eval dataset (D).

First, comparing VeLO (ft) to VeLO (og), we observe that fine-tuning VeLO to a specific dataset and number of steps leads to a significant performance improvement compared to the general-purpose VeLO (og). However, VeLO (ft) performance deteriorates sharply when evaluated outside its meta-training regime. We hypothesize this is due to VeLO complexity and parameter count, making it more sensitive to deviations from its training distribution, especially number of steps.

Despite the improvement from fine-tuning, VeLO (ft) still underperforms compared to L3RS. We attribute this to L3RS’s more lightweight architecture and its inherent ability to fallback to the performance of its base optimizers for robust performance. L3RS also performs favorably against Adam for all $K$ values within the training distribution ($10$ to $500$ steps), but their performance becomes comparable when evaluated beyond $K=1\,000$ steps.

Interestingly, changing either the initialization or evaluation dataset to Places has a minimal effect on the relative performance of the optimizers, even though Places was not included in the meta-training data. This highlights the robust generalization capabilities of both L3RS and the fine-tuned VeLO.

We also apply the methods to a randomly initialized Resnet-34 model (E). RandomInit uses a standard initialization method (Klambauer et al., ). In this case, L3RS does not outperform the baseline methods. We believe this is because, while L3RS generalizes well to different checkpoints (C), the randomly initialized model may be too different and far out of distribution for L3RS to perform optimally.

Figure: Figure 4: Visualization of learned mixing coefficients $\mu^{(l)}$ and per-layer learning rates $\lambda^{(l)}$ over 100 steps for a ResNet-34 model. Each layer’s type and component are distinguished by color and line type (see legend). The general trend shows curves moving up and to the left, indicating a transition from Adam ($\mu^{(l)}=0$) to SGD ($\mu^{(l)}=1$) and a decrease in $\lambda^{(l)}$. The initial step is marked with a $\bullet$ and the final step with a $\star$.
Refer to caption: x10.png

Figure: Figure 5: Average learning rate vs. direction mix between Adam and SGD for each of the 100 steps of the L3RS optimizer.
Refer to caption: x13.png

In Figure 3 (F) we show the speedup, in training steps, L3RS achieves over baseline methods to reach a given accuracy. L3RS is able to demonstrate robust 50% speedup over VeLO (ft) and $100\%$ speedup over the best hand-designed optimizer.

The learned parameters of L3RS exhibit interesting dynamics during a 100-step evaluation (Figure 4). Initially, the optimizer strongly favors the Adam direction with a high learning rate for most layers. As training progresses, this preference shifts, transitioning towards SGD and a lower learning rate (represented by a movement towards the top-left of the plot). Figure 5, showing the average parameter movement across all layers, reveals several distinct phases: an initial warm-up period with an increasing learning rate, a period of relatively constant learning rate while transitioning from Adam to SGD, a phase of rapid learning rate decay, and a final convergence to the SGD direction over the last $\sim$10 steps. While the precise
interpretation of these parameter dynamics is challenging, the L3RS parameters are considerably
more interpretable than those of most black-box learned optimizers.

**Table 2: Average and standard deviation of evaluation accuracy for different optimizers and per-layer strategies.**
| Base-optimizers | Embedding | No Embedding | Per-layer MLP | Global |
| --- | --- | --- | --- | --- |
| SGD only | $68.18\pm 4.29$ | $64.48\pm 5.05$ | $64.71\pm 5.00$ | $64.37\pm 4.94$ |
| Adam only | $68.52\pm 4.25$ | $61.44\pm 5.14$ | $67.14\pm 4.57$ | $63.45\pm 4.84$ |
| SGD, Adam | $\mathbf{68.93\pm 4.58}$ | $65.70\pm 4.91$ | $68.52\pm 4.40$ | $66.58\pm 4.93$ |

**Table 3: Average and standard deviation evaluation accuracy given adaptive EMA features.**
| EMA’s Smoothing Factors | Accuracy |
| --- | --- |
| $\gamma\in\{0.99,0.9,0.0\}$ | $68.93\pm 4.34$ |
| $\gamma\in\{0.9,0.0\}$ | $67.43\pm 4.64$ |
| $\gamma\in\{0.0\}$ | $67.78\pm 4.47$ |
| $\gamma\in\varnothing$ | $66.52\pm 4.84$ |

## 6 Ablations and Variants

We perform a methodical ablation and variant study to justify the design choices of the L3RS architecture. We meta-train all models in this section using the ImageNet25 task distribution but set number of steps, $K$, to $10$. We additionally meta-train for $500$ generations of NES and change the meta-learning rate decay rate steps from $500$ to $100$.

#### Ablation of Base Optimizers.

We will show the results of using either SGD or Adam alone as the wrapped optimizer rather than both together.

#### Embedding Variants.

Without layer embeddings, L3RS struggles to distinguish between layers, relying heavily on adaptive EMA features. Alternatively, we can meta-learn a separate MLP for each layer of the target model. This significantly increases the optimizer’s parameter count. We compare this per-layer MLP approach with a single shared MLP (without layer embeddings). We additionally report results for a Global method, which uses a single learning rate ($\lambda$) and direction weights ($\mu_{p}$) for all layers. Results for these different optimizer configurations are presented in Table 2.

#### Ablation of Adaptive EMA Input Features.

We also investigated the impact of different smoothing factors ($\gamma$) for the adaptive exponential moving average (EMA) features. The standard L3RS uses $\gamma\in\{0.99,0.9,0.0\}$. For comparison, we trained models with $\gamma\in\{0.9,0.0\}$, $\gamma=0.0$ (representing raw features without averaging), and no adaptive EMA features ($\gamma=\emptyset$). The results are summarized in Table 3.

**Table 4: Average eval accuracy and standard deviation of L3RS variants, each using a different set of base-optimizers.**
| Base Optimizer Set | Accuracy |
| --- | --- |
| SGD, Adam | $\mathbf{68.93\pm 4.58}$ |
| Adam, Lion, LAMB | $68.61\pm 4.50$ |
| Adamax, SGD, LAMB | $66.95\pm 4.80$ |
| SGD, Adam, Adamax, Lion, LAMB, WeightDecay | $68.41\pm 4.65$ |

#### New Directions.

Our main results are based on L3RS which uses only SGD and Adam as given directions. There is a lot of room for exploration in which optimizers and combinations will result in the most powerful L3RS variant for a given task. As an example of the flexibility of the architecture we explore a few combinations here. We leverage the following optimizers: Lion (Chen et al., ), LAMB (You et al., ), Adamax (Kingma & Ba, ), as well as a WeightDecay direction which simply provides the negative direction of the current parameters. We use the same training/evaluation set up as the rest of the ablations and variants. We report the results in Table 4.

## 7 Conclusions

Learned optimization is a powerful paradigm which has the potential to outperform existing methods. Our results suggest that narrowing the domain that learned optimizers are meta-trained on can provide a reduced meta-training cost, and allow learning domain specific exploitations that can increase their performance, especially on fine-tuning tasks. We provide results repurposing VeLO, an architecture designed for general learned optimization, as well as propose a new architecture L3RS, designed to take advantage of the narrow domain. We demonstrate the robustness of this method on out of distribution tasks. As fine-tuning tasks continue to become more relevant, especially for LLMs, this paradigm and architecture provides a promising direction for future research. Improving this technique further can result in faster model training, better performance, and robust data generalization.

## Appendix A Checkpoint Pretraining

We create the pretrain checkpoints for both ImageNet and Places using the same hyperparameters. We train the models using batch size $128$ and train for $100,000$ steps. Augmentations/preprocessing used include random cropping to size 224, random mirror, resize, and normalization (based on Imagenet train statistics). For the ImageNet checkpoint trained on the first 500 classes, we reach an evaluation loss of $0.9387$ and an evaluation accuracy of $75.68$. For the Places checkpoint trained on the first 150 classes, we reach an evaluation loss of $1.312$ and an evaluation accuracy of $61.72$.

## Appendix B Meta-Training

We provide the meta-training curves, both fitness and accuracy, for the two meta-trained learned optimizers. Figure 6 shows the curves for L3RS and VeLO throughout training. VeLO training is very unstable in this setup, though does converge by the end of training. L3RS reaches a high fitness very rapidly then slowly improves throughout the rest of the training, likely due to the smaller number of parameters in the learned optimizer.

Figures 8 and 8 show the example of the relative and absolute time features used by L3RS.

## Appendix C Learned mixing coefficients dynamics

To better understand the optimization dynamics of L3RS during a 100-evaluation run, we visualize the learning rate and direction mix per layer. Figure 9 provides a more granular view, displaying the same relationship for each individual layer.

## Appendix D Evaluation Tables

We provide the full evaluation tables of accuracy and loss from the main evaluation experiments Figure 5 A-E. We additionally reproduce Figure 5 using the evaluation losses in Figure 10. Table 5 and Table  6 provide the accuracy and loss results respectively from the in-domain evaluation A. Table 7 and Table 8 provide the accuracy and loss results respectively for all head-only fine-tuning for evaluation A. Table 9 and Table 10 provide the accuracy and loss results respectively from the dataset-generalization evaluation B. Table 11 and Table 12 provide the accuracy and loss results respectively from the checkpoint-generalization evaluation C. Table 13 and Table 14 provide the accuracy and loss results respectively from the initialization and dataset-generalization evaluation D. Table 15 and Table 16 provide the accuracy and loss results respectively from the RandomInit evaluation E.

Figure: Figure 6: Meta-training curves for L3RS (Black), and VeLO (Blue). Left: Average population fitness. Right: Average population evaluation accuracy.
Refer to caption: extracted/5903129/figures/l3rs_train_fitness.png

Figure: Figure 7: Relative time features as a function of training progress (k/K). Each of the lines represents a time input feature to the model, generated using $k$, $K$, and $\alpha_{i}$.
Refer to caption: x14.png

Figure: Figure 9: Learned mixing coefficients $\mu_{p}^{(l)}$ and $\lambda^{(l)}$ for each iteration of L3RS for ResNet-34 model. *Top:* average across all the layers. *Bottom:* individual for every learned component.
Refer to caption: x16.png

Figure: Figure 10: Meta-evaluation of the L3RS, meta-trained on ImageNet25 for $10$ to $500$ steps along with various benchmarks. Performance is compared to VeLO (*left*), Adam with cosine learning rate (*center*), and Adam with a constant learning rate (*right*). Each marker represents model loss evaluation at that number of steps. Solid lines indicate the number of steps for in-distribution evaluation, while dashed lines indicate generalization to more steps than meta-training. A. In-domain Generalization. Both initialization and evaluation are on ImageNet. B. Out-of-Domain Initialization. Initialized on ImageNet, evaluated on Places25Eval. C. Out-of-Domain Evaluation. Initialized on Places, evaluated on ImageNet25Eval. D. Out-of-Domain Init & Eval. Both initialization and evaluation are on Places dataset. E. Random Initialization. Random initialization, evaluated on ImageNet25Eval dataset. F. Speedup of L3RS. For in-domain generalization, this shows how much faster L3RS achieves a given loss compared to the baselines.
Refer to caption: x19.png

**Table 5: Average and standard deviation evaluation accuracy for main experiment (A).**
| Optimizer | 10-Step | 25-Step | 50-Step | 100-Step | 250-Step | 500-Step | 1000-Step | 2500-Step |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| L3RS | $68.52\pm 4.46$ | $76.78\pm 3.83$ | $80.22\pm 3.45$ | $83.54\pm 2.99$ | $86.57\pm 2.77$ | $88.47\pm 2.64$ | $88.79\pm 2.57$ | $89.44\pm 2.52$ |
| VeLO (ft) | $62.11\pm 5.51$ | $73.23\pm 4.29$ | $77.41\pm 3.44$ | $81.54\pm 3.46$ | $85.16\pm 2.93$ | $86.64\pm 3.1$ | $73.64\pm 26.26$ | $4.0\pm 0.17$ |
| VeLO (og) | $50.63\pm 4.9$ | $47.48\pm 5.2$ | $53.7\pm 4.6$ | $60.79\pm 4.24$ | $71.02\pm 4.08$ | $78.09\pm 3.62$ | $82.45\pm 3.19$ | $83.39\pm 3.04$ |
| Adam (cosine), 1e-3 | $55.86\pm 4.92$ | $66.38\pm 4.7$ | $71.03\pm 4.11$ | $76.59\pm 4.19$ | $82.53\pm 3.46$ | $85.34\pm 3.17$ | $86.84\pm 2.7$ | $87.25\pm 2.88$ |
| Adam (cosine), 1e-4 | $10.69\pm 2.52$ | $28.22\pm 4.03$ | $52.11\pm 5.04$ | $70.29\pm 4.6$ | $81.22\pm 3.43$ | $85.3\pm 3.22$ | $87.52\pm 2.96$ | $88.88\pm 2.59$ |
| Adam (cosine), 1e-5 | $4.32\pm 1.45$ | $5.09\pm 1.62$ | $6.47\pm 1.81$ | $10.66\pm 2.29$ | $29.4\pm 4.0$ | $54.18\pm 5.05$ | $72.22\pm 4.67$ | $82.36\pm 3.52$ |
| Adam (const), 1e-3 | $60.12\pm 4.62$ | $62.65\pm 4.37$ | $65.82\pm 4.24$ | $70.85\pm 4.23$ | $75.92\pm 3.94$ | $79.11\pm 3.7$ | $80.87\pm 3.58$ | $81.92\pm 3.61$ |
| Adam (const), 1e-4 | $22.09\pm 3.54$ | $51.72\pm 4.98$ | $69.72\pm 4.59$ | $78.58\pm 3.95$ | $84.02\pm 3.07$ | $86.65\pm 2.93$ | $87.72\pm 2.76$ | $87.95\pm 2.78$ |
| Adam (const), 1e-5 | $4.83\pm 1.56$ | $6.47\pm 1.77$ | $10.7\pm 2.29$ | $22.66\pm 3.62$ | $53.91\pm 4.99$ | $72.15\pm 4.6$ | $80.34\pm 3.62$ | $85.62\pm 3.16$ |

**Table 6: Average and standard deviation evaluation loss for main experiment (A).**
| Optimizer | 10-Step | 25-Step | 50-Step | 100-Step | 250-Step | 500-Step | 1000-Step | 2500-Step |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| L3RS | $1.17\pm 0.13$ | $0.82\pm 0.11$ | $0.67\pm 0.10$ | $0.56\pm 0.10$ | $0.46\pm 0.09$ | $0.39\pm 0.08$ | $0.38\pm 0.09$ | $0.43\pm 0.11$ |
| VeLO (ft) | $1.35\pm 0.17$ | $0.90\pm 0.13$ | $0.75\pm 0.11$ | $0.62\pm 0.11$ | $0.50\pm 0.09$ | $0.45\pm 0.10$ | $0.92\pm 0.96$ | $3.23\pm 0.12$ |
| VeLO (og) | $1.75\pm 0.18$ | $1.81\pm 0.16$ | $1.57\pm 0.14$ | $1.33\pm 0.14$ | $0.98\pm 0.13$ | $0.74\pm 0.12$ | $0.61\pm 0.11$ | $0.67\pm 0.14$ |
| Adam (cosine), 1e-3 | $1.67\pm 0.15$ | $1.20\pm 0.14$ | $1.01\pm 0.13$ | $0.82\pm 0.13$ | $0.61\pm 0.11$ | $0.50\pm 0.10$ | $0.45\pm 0.09$ | $0.49\pm 0.13$ |
| Adam (cosine), 1e-4 | $3.18\pm 0.06$ | $2.63\pm 0.09$ | $1.94\pm 0.13$ | $1.23\pm 0.14$ | $0.69\pm 0.11$ | $0.51\pm 0.10$ | $0.42\pm 0.09$ | $0.40\pm 0.10$ |
| Adam (cosine), 1e-5 | $3.56\pm 0.05$ | $3.49\pm 0.05$ | $3.38\pm 0.05$ | $3.17\pm 0.06$ | $2.60\pm 0.09$ | $1.88\pm 0.13$ | $1.17\pm 0.14$ | $0.66\pm 0.11$ |
| Adam (const), 1e-3 | $1.44\pm 0.15$ | $1.34\pm 0.14$ | $1.20\pm 0.15$ | $1.00\pm 0.13$ | $0.83\pm 0.12$ | $0.72\pm 0.12$ | $0.66\pm 0.11$ | $0.70\pm 0.15$ |
| Adam (const), 1e-4 | $2.80\pm 0.08$ | $1.95\pm 0.13$ | $1.24\pm 0.14$ | $0.80\pm 0.12$ | $0.56\pm 0.10$ | $0.46\pm 0.09$ | $0.43\pm 0.09$ | $0.46\pm 0.12$ |
| Adam (const), 1e-5 | $3.51\pm 0.05$ | $3.38\pm 0.05$ | $3.17\pm 0.06$ | $2.78\pm 0.08$ | $1.89\pm 0.13$ | $1.17\pm 0.14$ | $0.74\pm 0.11$ | $0.50\pm 0.10$ |

**Table 7: Average and standard deviation evaluation accuracy for all Head-only fine-tuning from main experiment (A).**
| Optimizer | 10-Step | 25-Step | 50-Step | 100-Step | 250-Step | 500-Step | 1000-Step | 2500-Step |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| VeLO (og) | $42.53\pm 4.73$ | $67.93\pm 4.4$ | $73.68\pm 3.65$ | $76.35\pm 3.58$ | $79.12\pm 3.45$ | $80.71\pm 3.2$ | $81.36\pm 3.11$ | $81.6\pm 3.23$ |
| Adam (cosine), 1e-3 | $7.98\pm 2.16$ | $16.93\pm 3.14$ | $34.58\pm 4.39$ | $54.65\pm 5.05$ | $69.84\pm 4.55$ | $74.95\pm 4.0$ | $78.0\pm 3.68$ | $80.52\pm 3.41$ |
| Adam (cosine), 1e-4 | $4.18\pm 1.47$ | $4.53\pm 1.59$ | $5.32\pm 1.7$ | $7.52\pm 2.06$ | $16.86\pm 3.24$ | $34.34\pm 4.53$ | $54.8\pm 4.99$ | $70.14\pm 4.72$ |
| Adam (cosine), 1e-5 | $3.95\pm 1.38$ | $3.98\pm 1.38$ | $4.05\pm 1.39$ | $4.21\pm 1.45$ | $4.55\pm 1.54$ | $5.24\pm 1.55$ | $7.32\pm 1.96$ | $16.74\pm 3.15$ |
| Adam (const), 1e-3 | $13.31\pm 2.71$ | $33.5\pm 4.31$ | $53.59\pm 4.81$ | $65.91\pm 4.67$ | $74.17\pm 4.07$ | $77.34\pm 3.67$ | $79.53\pm 3.51$ | $81.17\pm 3.45$ |
| Adam (const), 1e-4 | $4.44\pm 1.51$ | $5.22\pm 1.59$ | $7.4\pm 1.97$ | $13.27\pm 3.03$ | $34.17\pm 4.59$ | $54.5\pm 4.94$ | $67.28\pm 4.72$ | $75.17\pm 3.93$ |
| Adam (const), 1e-5 | $3.98\pm 1.38$ | $4.01\pm 1.41$ | $4.21\pm 1.43$ | $4.43\pm 1.52$ | $5.22\pm 1.57$ | $7.32\pm 2.0$ | $13.13\pm 3.05$ | $34.35\pm 4.57$ |

**Table 8: Average and standard deviation evaluation loss for all Head-only fine-tuning from main experiment (A).**
| Optimizer | 10-Step | 25-Step | 50-Step | 100-Step | 250-Step | 500-Step | 1000-Step | 2500-Step |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| VeLO (og) | $2.21\pm 0.11$ | $1.11\pm 0.14$ | $0.92\pm 0.14$ | $0.85\pm 0.13$ | $0.76\pm 0.13$ | $0.68\pm 0.12$ | $0.65\pm 0.12$ | $0.66\pm 0.12$ |
| Adam (cosine), 1e-3 | $3.20\pm 0.05$ | $2.88\pm 0.07$ | $2.45\pm 0.09$ | $1.86\pm 0.11$ | $1.21\pm 0.12$ | $0.93\pm 0.12$ | $0.77\pm 0.11$ | $0.66\pm 0.11$ |
| Adam (cosine), 1e-4 | $3.55\pm 0.05$ | $3.48\pm 0.05$ | $3.38\pm 0.05$ | $3.23\pm 0.05$ | $2.89\pm 0.06$ | $2.46\pm 0.08$ | $1.88\pm 0.11$ | $1.21\pm 0.12$ |
| Adam (cosine), 1e-5 | $3.60\pm 0.05$ | $3.59\pm 0.05$ | $3.58\pm 0.05$ | $3.55\pm 0.05$ | $3.48\pm 0.05$ | $3.39\pm 0.05$ | $3.23\pm 0.05$ | $2.90\pm 0.06$ |
| Adam (const), 1e-3 | $3.00\pm 0.06$ | $2.47\pm 0.09$ | $1.87\pm 0.11$ | $1.35\pm 0.12$ | $0.95\pm 0.12$ | $0.79\pm 0.11$ | $0.70\pm 0.11$ | $0.64\pm 0.11$ |
| Adam (const), 1e-4 | $3.50\pm 0.05$ | $3.38\pm 0.05$ | $3.23\pm 0.05$ | $3.00\pm 0.06$ | $2.47\pm 0.08$ | $1.89\pm 0.11$ | $1.35\pm 0.12$ | $0.93\pm 0.12$ |
| Adam (const), 1e-5 | $3.59\pm 0.05$ | $3.58\pm 0.05$ | $3.55\pm 0.05$ | $3.51\pm 0.05$ | $3.39\pm 0.05$ | $3.24\pm 0.05$ | $3.00\pm 0.06$ | $2.46\pm 0.08$ |

**Table 9: Average and standard deviation evaluation accuracy for main experiment (B).**
| Optimizer | 10-Step | 25-Step | 50-Step | 100-Step | 250-Step | 500-Step | 1000-Step | 2500-Step |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| L3RS | $59.2\pm 3.87$ | $69.16\pm 3.32$ | $73.64\pm 3.23$ | $76.75\pm 3.32$ | $80.67\pm 3.07$ | $82.74\pm 2.92$ | $84.32\pm 2.82$ | $85.51\pm 2.81$ |
| VeLO (ft) | $50.13\pm 7.32$ | $64.78\pm 3.46$ | $70.46\pm 3.29$ | $74.1\pm 7.83$ | $79.12\pm 3.21$ | $76.54\pm 12.9$ | $56.13\pm 30.47$ | $4.02\pm 0.36$ |
| VeLO (og) | $43.13\pm 3.98$ | $44.58\pm 4.18$ | $51.26\pm 4.18$ | $59.02\pm 4.0$ | $68.38\pm 3.83$ | $75.14\pm 3.47$ | $79.97\pm 3.49$ | $83.07\pm 3.13$ |
| Adam (cosine), 1e-3 | $48.85\pm 3.79$ | $60.39\pm 3.34$ | $66.11\pm 3.59$ | $71.81\pm 3.64$ | $77.71\pm 3.44$ | $80.73\pm 3.17$ | $82.99\pm 2.96$ | $84.97\pm 2.76$ |
| Adam (cosine), 1e-4 | $9.87\pm 2.24$ | $23.3\pm 2.97$ | $44.16\pm 3.99$ | $63.11\pm 3.72$ | $74.9\pm 3.65$ | $79.71\pm 3.0$ | $82.68\pm 3.04$ | $85.17\pm 2.98$ |
| Adam (cosine), 1e-5 | $4.38\pm 1.53$ | $4.98\pm 1.55$ | $6.07\pm 1.68$ | $9.06\pm 2.06$ | $22.83\pm 3.06$ | $45.84\pm 3.56$ | $65.3\pm 3.72$ | $76.39\pm 3.54$ |
| Adam (const), 1e-3 | $52.54\pm 4.3$ | $56.19\pm 3.93$ | $61.15\pm 3.92$ | $65.87\pm 3.94$ | $71.71\pm 3.52$ | $74.83\pm 3.42$ | $77.42\pm 3.57$ | $80.51\pm 3.28$ |
| Adam (const), 1e-4 | $17.32\pm 2.81$ | $43.3\pm 3.74$ | $62.03\pm 3.48$ | $71.65\pm 3.48$ | $78.01\pm 3.28$ | $80.57\pm 2.85$ | $82.72\pm 3.01$ | $84.66\pm 2.97$ |
| Adam (const), 1e-5 | $4.73\pm 1.54$ | $6.08\pm 1.66$ | $9.05\pm 2.15$ | $17.72\pm 2.64$ | $45.84\pm 3.79$ | $64.93\pm 3.85$ | $74.3\pm 3.65$ | $80.72\pm 3.3$ |

**Table 10: Average and standard deviation evaluation loss for main experiment (B).**
| Optimizer | 10-Step | 25-Step | 50-Step | 100-Step | 250-Step | 500-Step | 1000-Step | 2500-Step |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| L3RS | $1.44\pm 0.10$ | $1.02\pm 0.09$ | $0.85\pm 0.09$ | $0.73\pm 0.10$ | $0.61\pm 0.09$ | $0.55\pm 0.08$ | $0.49\pm 0.08$ | $0.45\pm 0.08$ |
| VeLO (ft) | $1.73\pm 0.25$ | $1.13\pm 0.10$ | $0.95\pm 0.09$ | $0.82\pm 0.26$ | $0.66\pm 0.09$ | $0.78\pm 0.51$ | $3.65\pm 15.11$ | $1209.28\pm 11410.45$ |
| VeLO (og) | $1.88\pm 0.12$ | $1.80\pm 0.12$ | $1.57\pm 0.12$ | $1.34\pm 0.10$ | $1.01\pm 0.11$ | $0.79\pm 0.10$ | $0.64\pm 0.09$ | $0.53\pm 0.08$ |
| Adam (cosine), 1e-3 | $1.82\pm 0.10$ | $1.33\pm 0.09$ | $1.11\pm 0.10$ | $0.91\pm 0.10$ | $0.71\pm 0.09$ | $0.61\pm 0.08$ | $0.54\pm 0.08$ | $0.47\pm 0.08$ |
| Adam (cosine), 1e-4 | $3.19\pm 0.06$ | $2.75\pm 0.06$ | $2.16\pm 0.08$ | $1.42\pm 0.09$ | $0.85\pm 0.09$ | $0.66\pm 0.09$ | $0.55\pm 0.08$ | $0.46\pm 0.08$ |
| Adam (cosine), 1e-5 | $3.52\pm 0.06$ | $3.46\pm 0.06$ | $3.37\pm 0.06$ | $3.21\pm 0.06$ | $2.75\pm 0.06$ | $2.10\pm 0.09$ | $1.35\pm 0.10$ | $0.80\pm 0.09$ |
| Adam (const), 1e-3 | $1.61\pm 0.12$ | $1.45\pm 0.11$ | $1.26\pm 0.10$ | $1.09\pm 0.11$ | $0.90\pm 0.10$ | $0.80\pm 0.10$ | $0.71\pm 0.10$ | $0.61\pm 0.09$ |
| Adam (const), 1e-4 | $2.91\pm 0.06$ | $2.18\pm 0.08$ | $1.46\pm 0.09$ | $0.98\pm 0.10$ | $0.72\pm 0.09$ | $0.61\pm 0.08$ | $0.55\pm 0.08$ | $0.48\pm 0.08$ |
| Adam (const), 1e-5 | $3.48\pm 0.06$ | $3.38\pm 0.06$ | $3.21\pm 0.06$ | $2.90\pm 0.06$ | $2.10\pm 0.09$ | $1.36\pm 0.10$ | $0.90\pm 0.09$ | $0.63\pm 0.09$ |

**Table 11: Average and standard deviation evaluation accuracy for main experiment (C).**
| Optimizer | 10-Step | 25-Step | 50-Step | 100-Step | 250-Step | 500-Step | 1000-Step | 2500-Step |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| L3RS | $65.65\pm 4.61$ | $73.13\pm 3.99$ | $77.0\pm 4.02$ | $80.33\pm 3.59$ | $83.79\pm 3.06$ | $85.93\pm 2.97$ | $86.91\pm 2.78$ | $87.04\pm 2.68$ |
| VeLO (ft) | $57.85\pm 7.39$ | $69.6\pm 4.0$ | $73.51\pm 4.09$ | $77.96\pm 3.8$ | $82.18\pm 3.22$ | $84.46\pm 2.95$ | $82.67\pm 11.37$ | $4.04\pm 0.27$ |
| VeLO (og) | $52.35\pm 4.67$ | $49.56\pm 4.78$ | $54.52\pm 4.72$ | $61.44\pm 4.58$ | $71.37\pm 4.1$ | $77.87\pm 3.67$ | $81.7\pm 3.15$ | $82.82\pm 3.08$ |
| Adam (cosine), 1e-3 | $55.36\pm 4.8$ | $62.93\pm 4.44$ | $67.45\pm 4.35$ | $72.54\pm 4.18$ | $79.14\pm 3.41$ | $82.61\pm 3.09$ | $84.43\pm 3.19$ | $85.36\pm 3.17$ |
| Adam (cosine), 1e-4 | $13.27\pm 2.7$ | $33.25\pm 4.39$ | $54.11\pm 4.84$ | $68.21\pm 4.46$ | $78.83\pm 3.61$ | $83.49\pm 3.16$ | $85.96\pm 2.98$ | $87.31\pm 2.66$ |
| Adam (cosine), 1e-5 | $4.37\pm 1.52$ | $5.26\pm 1.72$ | $7.14\pm 2.02$ | $12.29\pm 2.7$ | $33.43\pm 4.16$ | $56.37\pm 4.73$ | $70.46\pm 4.56$ | $80.28\pm 3.51$ |
| Adam (const), 1e-3 | $56.2\pm 4.9$ | $58.38\pm 4.31$ | $61.7\pm 4.4$ | $66.41\pm 4.45$ | $72.24\pm 4.05$ | $75.95\pm 3.68$ | $78.45\pm 3.38$ | $80.04\pm 3.6$ |
| Adam (const), 1e-4 | $25.4\pm 3.79$ | $53.72\pm 4.75$ | $67.04\pm 4.56$ | $75.64\pm 4.01$ | $81.56\pm 3.42$ | $84.25\pm 3.09$ | $85.96\pm 2.8$ | $85.92\pm 2.75$ |
| Adam (const), 1e-5 | $4.87\pm 1.58$ | $6.97\pm 2.01$ | $12.38\pm 2.59$ | $26.29\pm 4.03$ | $56.09\pm 4.69$ | $70.48\pm 4.56$ | $78.41\pm 3.75$ | $83.96\pm 3.11$ |

**Table 12: Average and standard deviation evaluation loss for main experiment (C).**
| Optimizer | 10-Step | 25-Step | 50-Step | 100-Step | 250-Step | 500-Step | 1000-Step | 2500-Step |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| L3RS | $1.34\pm 0.13$ | $0.95\pm 0.12$ | $0.78\pm 0.11$ | $0.66\pm 0.11$ | $0.54\pm 0.10$ | $0.47\pm 0.09$ | $0.44\pm 0.09$ | $0.50\pm 0.11$ |
| VeLO (ft) | $1.64\pm 0.27$ | $1.02\pm 0.13$ | $0.88\pm 0.13$ | $0.73\pm 0.12$ | $0.59\pm 0.10$ | $0.52\pm 0.09$ | $0.58\pm 0.40$ | $15.24\pm 39.84$ |
| VeLO (og) | $1.66\pm 0.16$ | $1.73\pm 0.15$ | $1.55\pm 0.15$ | $1.31\pm 0.14$ | $0.97\pm 0.14$ | $0.75\pm 0.12$ | $0.62\pm 0.11$ | $0.71\pm 0.15$ |
| Adam (cosine), 1e-3 | $1.75\pm 0.13$ | $1.33\pm 0.15$ | $1.14\pm 0.13$ | $0.95\pm 0.13$ | $0.72\pm 0.11$ | $0.60\pm 0.10$ | $0.52\pm 0.11$ | $0.55\pm 0.13$ |
| Adam (cosine), 1e-4 | $3.05\pm 0.05$ | $2.63\pm 0.07$ | $2.05\pm 0.10$ | $1.36\pm 0.13$ | $0.79\pm 0.11$ | $0.59\pm 0.10$ | $0.48\pm 0.09$ | $0.46\pm 0.10$ |
| Adam (cosine), 1e-5 | $3.35\pm 0.05$ | $3.30\pm 0.05$ | $3.22\pm 0.05$ | $3.07\pm 0.05$ | $2.63\pm 0.07$ | $1.98\pm 0.11$ | $1.28\pm 0.12$ | $0.74\pm 0.11$ |
| Adam (const), 1e-3 | $1.60\pm 0.16$ | $1.48\pm 0.14$ | $1.33\pm 0.15$ | $1.15\pm 0.13$ | $0.94\pm 0.13$ | $0.81\pm 0.12$ | $0.74\pm 0.12$ | $0.76\pm 0.16$ |
| Adam (const), 1e-4 | $2.79\pm 0.06$ | $2.08\pm 0.10$ | $1.39\pm 0.12$ | $0.93\pm 0.12$ | $0.65\pm 0.10$ | $0.54\pm 0.09$ | $0.49\pm 0.09$ | $0.55\pm 0.12$ |
| Adam (const), 1e-5 | $3.32\pm 0.05$ | $3.22\pm 0.05$ | $3.07\pm 0.05$ | $2.77\pm 0.06$ | $1.99\pm 0.11$ | $1.28\pm 0.12$ | $0.83\pm 0.11$ | $0.56\pm 0.10$ |

**Table 13: Average and standard deviation evaluation accuracy for main experiment (D).**
| Optimizer | 10-Step | 25-Step | 50-Step | 100-Step | 250-Step | 500-Step | 1000-Step | 2500-Step |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| L3RS | $72.15\pm 3.71$ | $77.52\pm 3.11$ | $80.22\pm 3.35$ | $81.76\pm 3.27$ | $83.89\pm 3.0$ | $84.96\pm 2.81$ | $85.95\pm 2.83$ | $87.06\pm 2.79$ |
| VeLO (ft) | $66.93\pm 7.24$ | $75.68\pm 3.35$ | $78.26\pm 3.38$ | $80.88\pm 2.98$ | $83.33\pm 3.06$ | $80.12\pm 12.15$ | $59.12\pm 30.85$ | $4.05\pm 0.37$ |
| VeLO (og) | $58.38\pm 4.31$ | $54.52\pm 3.97$ | $58.73\pm 4.2$ | $64.48\pm 3.83$ | $73.04\pm 3.49$ | $78.45\pm 3.33$ | $81.84\pm 3.41$ | $84.44\pm 2.41$ |
| Adam (cosine), 1e-3 | $60.65\pm 4.08$ | $67.66\pm 3.13$ | $70.77\pm 3.74$ | $74.73\pm 3.69$ | $79.2\pm 3.24$ | $81.41\pm 3.25$ | $83.47\pm 3.13$ | $85.58\pm 3.78$ |
| Adam (cosine), 1e-4 | $15.17\pm 2.9$ | $38.45\pm 4.37$ | $61.08\pm 3.91$ | $74.22\pm 3.33$ | $80.98\pm 3.45$ | $83.73\pm 3.03$ | $85.26\pm 3.0$ | $86.8\pm 2.98$ |
| Adam (cosine), 1e-5 | $4.86\pm 1.84$ | $5.88\pm 2.0$ | $7.95\pm 2.25$ | $13.78\pm 3.02$ | $38.75\pm 4.17$ | $63.86\pm 3.64$ | $76.06\pm 3.29$ | $82.1\pm 2.81$ |
| Adam (const), 1e-3 | $60.29\pm 4.24$ | $60.9\pm 3.63$ | $63.8\pm 3.57$ | $68.15\pm 3.88$ | $72.46\pm 3.92$ | $75.25\pm 3.52$ | $77.68\pm 3.61$ | $80.7\pm 3.14$ |
| Adam (const), 1e-4 | $29.37\pm 4.04$ | $60.46\pm 3.96$ | $73.09\pm 3.48$ | $78.62\pm 3.34$ | $81.95\pm 3.29$ | $83.27\pm 2.93$ | $84.38\pm 2.99$ | $85.86\pm 2.64$ |
| Adam (const), 1e-5 | $5.43\pm 1.9$ | $7.8\pm 2.26$ | $13.8\pm 2.96$ | $30.14\pm 4.13$ | $63.53\pm 3.66$ | $75.85\pm 3.25$ | $80.89\pm 3.24$ | $84.49\pm 3.66$ |

**Table 14: Average and standard deviation evaluation loss for main experiment (D).**
| Optimizer | 10-Step | 25-Step | 50-Step | 100-Step | 250-Step | 500-Step | 1000-Step | 2500-Step |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| L3RS | $2.90\pm 0.08$ | $2.72\pm 0.10$ | $2.59\pm 0.10$ | $2.41\pm 0.12$ | $2.06\pm 0.14$ | $1.79\pm 0.13$ | $1.54\pm 0.14$ | $1.35\pm 0.16$ |
| VeLO (ft) | $2.93\pm 0.13$ | $2.82\pm 0.09$ | $2.91\pm 0.48$ | $14.01\pm 28.82$ | $2552.78\pm 23833.64$ | $14000.60\pm 138939.16$ | $20165.58\pm 184680.17$ | $88.27\pm 713.91$ |
| VeLO (og) | $2.85\pm 0.09$ | $2.72\pm 0.10$ | $2.60\pm 0.11$ | $2.41\pm 0.13$ | $1.95\pm 0.15$ | $1.57\pm 0.14$ | $1.22\pm 0.14$ | $0.95\pm 0.15$ |
| Adam (cosine), 1e-3 | $2.98\pm 0.06$ | $2.81\pm 0.08$ | $2.67\pm 0.09$ | $2.50\pm 0.11$ | $2.13\pm 0.14$ | $1.84\pm 0.14$ | $1.52\pm 0.14$ | $1.13\pm 0.13$ |
| Adam (cosine), 1e-4 | $3.26\pm 0.03$ | $3.18\pm 0.03$ | $3.08\pm 0.04$ | $2.93\pm 0.06$ | $2.68\pm 0.08$ | $2.42\pm 0.11$ | $2.00\pm 0.13$ | $1.49\pm 0.13$ |
| Adam (cosine), 1e-5 | $3.31\pm 0.03$ | $3.30\pm 0.03$ | $3.29\pm 0.03$ | $3.26\pm 0.03$ | $3.18\pm 0.03$ | $3.07\pm 0.04$ | $2.91\pm 0.06$ | $2.65\pm 0.09$ |
| Adam (const), 1e-3 | $2.88\pm 0.07$ | $2.73\pm 0.09$ | $2.61\pm 0.10$ | $2.44\pm 0.12$ | $2.14\pm 0.14$ | $1.86\pm 0.15$ | $1.56\pm 0.14$ | $1.20\pm 0.15$ |
| Adam (const), 1e-4 | $3.21\pm 0.03$ | $3.08\pm 0.04$ | $2.93\pm 0.06$ | $2.76\pm 0.08$ | $2.46\pm 0.11$ | $2.10\pm 0.12$ | $1.76\pm 0.14$ | $1.37\pm 0.14$ |
| Adam (const), 1e-5 | $3.31\pm 0.03$ | $3.29\pm 0.03$ | $3.26\pm 0.03$ | $3.21\pm 0.03$ | $3.07\pm 0.04$ | $2.91\pm 0.06$ | $2.72\pm 0.08$ | $2.36\pm 0.11$ |

**Table 15: Average and standard deviation evaluation accuracy for main experiment (E).**
| Optimizer | 10-Step | 25-Step | 50-Step | 100-Step | 250-Step | 500-Step | 1000-Step | 2500-Step |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| L3RS | $15.77\pm 2.99$ | $21.11\pm 3.52$ | $25.5\pm 3.88$ | $30.46\pm 4.05$ | $40.32\pm 4.33$ | $47.69\pm 4.26$ | $54.74\pm 4.09$ | $60.9\pm 4.04$ |
| VeLO (ft) | $16.18\pm 3.69$ | $18.59\pm 3.27$ | $18.55\pm 6.33$ | $11.59\pm 8.62$ | $5.49\pm 5.06$ | $7.87\pm 7.24$ | $4.38\pm 0.98$ | $3.98\pm 0.27$ |
| VeLO (og) | $17.69\pm 3.1$ | $21.04\pm 3.66$ | $24.83\pm 3.72$ | $30.11\pm 4.28$ | $42.68\pm 4.34$ | $53.73\pm 3.87$ | $64.06\pm 4.14$ | $73.21\pm 3.92$ |
| Adam (cosine), 1e-3 | $14.48\pm 2.49$ | $19.26\pm 3.13$ | $23.45\pm 3.71$ | $28.14\pm 3.71$ | $38.59\pm 4.42$ | $46.64\pm 4.2$ | $55.25\pm 4.19$ | $66.46\pm 3.88$ |
| Adam (cosine), 1e-4 | $5.02\pm 1.43$ | $7.46\pm 1.92$ | $12.07\pm 2.49$ | $17.92\pm 3.09$ | $24.83\pm 3.62$ | $32.58\pm 4.26$ | $43.25\pm 4.41$ | $56.57\pm 3.98$ |
| Adam (cosine), 1e-5 | $4.17\pm 1.18$ | $4.29\pm 1.2$ | $4.46\pm 1.25$ | $4.97\pm 1.35$ | $7.2\pm 1.96$ | $12.18\pm 2.64$ | $18.75\pm 3.21$ | $26.31\pm 3.67$ |
| Adam (const), 1e-3 | $17.19\pm 3.05$ | $20.19\pm 3.34$ | $23.95\pm 3.68$ | $28.74\pm 4.2$ | $37.17\pm 4.62$ | $44.95\pm 4.21$ | $53.75\pm 4.12$ | $65.26\pm 3.64$ |
| Adam (const), 1e-4 | $6.34\pm 1.72$ | $12.18\pm 2.62$ | $17.68\pm 2.95$ | $22.36\pm 3.41$ | $30.64\pm 4.12$ | $40.09\pm 4.14$ | $48.52\pm 4.22$ | $59.35\pm 4.15$ |
| Adam (const), 1e-5 | $4.24\pm 1.23$ | $4.43\pm 1.24$ | $4.95\pm 1.38$ | $6.37\pm 1.64$ | $12.21\pm 2.6$ | $18.84\pm 3.19$ | $24.37\pm 3.61$ | $34.44\pm 4.33$ |

**Table 16: Average and standard deviation evaluation loss for main experiment (E).**
| Optimizer | 10-Step | 25-Step | 50-Step | 100-Step | 250-Step | 500-Step | 1000-Step | 2500-Step |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| L3RS | $1.07\pm 0.09$ | $0.76\pm 0.08$ | $0.65\pm 0.09$ | $0.58\pm 0.09$ | $0.51\pm 0.08$ | $0.47\pm 0.08$ | $0.43\pm 0.07$ | $0.40\pm 0.08$ |
| VeLO (ft) | $1.22\pm 0.25$ | $0.77\pm 0.09$ | $0.69\pm 0.09$ | $0.60\pm 0.09$ | $0.52\pm 0.08$ | $1.29\pm 3.69$ | $3.53\pm 12.36$ | $24563.19\pm 163714.77$ |
| VeLO (og) | $1.38\pm 0.12$ | $1.49\pm 0.12$ | $1.33\pm 0.11$ | $1.14\pm 0.11$ | $0.85\pm 0.10$ | $0.69\pm 0.09$ | $0.57\pm 0.09$ | $0.49\pm 0.08$ |
| Adam (cosine), 1e-3 | $1.52\pm 0.10$ | $1.11\pm 0.08$ | $0.97\pm 0.10$ | $0.82\pm 0.10$ | $0.67\pm 0.09$ | $0.58\pm 0.09$ | $0.52\pm 0.08$ | $0.46\pm 0.08$ |
| Adam (cosine), 1e-4 | $2.98\pm 0.06$ | $2.50\pm 0.07$ | $1.85\pm 0.08$ | $1.12\pm 0.09$ | $0.67\pm 0.08$ | $0.54\pm 0.08$ | $0.47\pm 0.08$ | $0.41\pm 0.07$ |
| Adam (cosine), 1e-5 | $3.34\pm 0.06$ | $3.28\pm 0.06$ | $3.19\pm 0.06$ | $3.01\pm 0.06$ | $2.50\pm 0.07$ | $1.78\pm 0.08$ | $1.05\pm 0.08$ | $0.63\pm 0.08$ |
| Adam (const), 1e-3 | $1.38\pm 0.12$ | $1.30\pm 0.11$ | $1.17\pm 0.10$ | $1.03\pm 0.11$ | $0.88\pm 0.11$ | $0.79\pm 0.09$ | $0.70\pm 0.09$ | $0.61\pm 0.10$ |
| Adam (const), 1e-4 | $2.69\pm 0.07$ | $1.89\pm 0.08$ | $1.15\pm 0.09$ | $0.77\pm 0.08$ | $0.60\pm 0.08$ | $0.53\pm 0.08$ | $0.49\pm 0.08$ | $0.45\pm 0.08$ |
| Adam (const), 1e-5 | $3.30\pm 0.06$ | $3.19\pm 0.06$ | $3.01\pm 0.06$ | $2.67\pm 0.07$ | $1.78\pm 0.08$ | $1.05\pm 0.08$ | $0.69\pm 0.08$ | $0.51\pm 0.08$ |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See weaknesses
soundness: 3
strengths: 1. The presentation is clear and easy to understand; The experiments are comprehensive and well-explained.
2. The proposed method is novel and useful: it eliminates the need for learning rate schedule tuning.
3. The proposed method demonstrates faster convergence and improved performance.
summary: This work introduces L3RS, a learned learning rate scheduler that ensembles updates from base optimizers like Adam and SGD. 

L3RS is parameterized as a shared MLP and layer-wise trainable embeddings. For each layer, L3RS takes its embedding and training status statistics (training time, exponential moving average of loss/gradient norm/weight norm) as input, and predicts coefficients of base optimizer updates. L3RS is trained with natural evolution strategies. In various settings with ImageNet and PLACES dataset, L3RS shows higher classification accuracy and training efficiency on ResNet34 finetuning.
weaknesses: 1. L3RS is not generalized and needs to be trained independently for each model architecture, which limits its application. From ImageNet <-> PLACES experiments it seems a trained L3RS can generalize to other data distributions, and it would be more beneficial to include more benchmarks to showcase that.
2. Experiments are only conducted on ResNet34, while including evaluation on other architectures and more diverse benchmarks (e.g. [VeLO](https://arxiv.org/abs/2211.09760)) would make the results more convincing. 
3. Finetuning for only 1000 steps may not well represent the practical finetuning setting.
4. Lack of a deeper analysis on how SGD/Adam update directions are favored by different layers/learning stages.
5. For practical use of L3RS, there needs to be a L3RS training recipe generalized across different model architectures, or pretrained L3RS checkpoints tuned for popular architectures.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1)	Given that the final performance of L3RS is comparable to Adam and considering the extra computational costs, what are the specific scenarios or benefits that justify the use of learned optimizers?  
2)	Experiments on additional tasks and models are necessary.  
3)	A more in-depth discussion of advanced adaptive learning methods, such as Adabelief, is helpful.
soundness: 2
strengths: 1) The paper studies the important challenge of automating hyperparameter tuning for optimizers, which is a critical aspect of training neural networks efficiently.  
2) By integrating the update directions from multiple optimizers and learning the optimal combination, the proposed method introduces a promising and potentially more adaptable approach to optimizer design.  
3) L3RS shows enhanced performance in the early stages of fine-tuning compared to baseline methods.
summary: This paper explores learned optimizers to refine pretrained models. The proposed method, L3RS, dynamically adjusts the weighting scaler for gradient directions derived from different optimizers like SGD and Adam in a layer-specific manner and tunes hyperparameters such as the learning rate during the meta-learning process. The results demonstrate that L3RS surpasses both traditional and other learned optimizers in terms of performance during the initial fine-tuning steps.
weaknesses: 1) My main concern is the applicability of this approach. According to Figure 3, the proposed learned optimizer only outperforms Adam in the early stages with smaller training steps. For larger training steps, Adam achieves comparable performance to the learned optimizer. Given that the learned optimizer requires extra time and memory to learn additional parameters, using it may not be necessary. Furthermore, the total convergence time, including the meta-learning process, may be slower than Adam.  
2) The evaluation is limited to training a ResNet-34 model on image classification tasks. Expanding the experiments to include diverse tasks such as language understanding or generation and different architectures like transformers would provide a more comprehensive evaluation.  
3) The study does not consider several recent and more effective variants of adaptive learning rate methods, such as Adabelief [1]. Including these in the comparative analysis or integrating their strategies could enhance the performance of learned optimizers and better justify their adoption.  

[1] Zhuang, Juntang, et al. "Adabelief optimizer: Adapting stepsizes by the belief in observed gradients." Advances in neural information processing systems 33 (2020): 18795-18806.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 1
questions: Please refer to the weaknesses section.
soundness: 3
strengths: The experiments on task distribution are comprehensive.
The section on preliminaries is comprehensive.
The ablation studies show the effectiveness of each design choice.
summary: This work proposes a learning to learn method where learned optimizers are used to finetune pretrained models. The work focuses mainly on downstream tasks with the goal of focusing the meta learning process on narrower tasks. Experiments revolve around showing that the learned optimizer can use off-the-shelf optimizers to surpass both off-the-shelf and SotA learned optimizers.
weaknesses: - The abstract is poorly written with wording that replaces necessary definitions. What is meant by “transforms the updates” (missing context), “various statistics” (can be replaced with an actual example) or “hand-designed” (didn’t understand what this could be in relation to)? In general, in terms of writing, the paper seems rushed and incomplete with inconsistencies in writing quality.

- L154: There is a mention of "a lot of parameters". A casual writing style isn't appropriate. Either the number of parameters should be mentioned or a relative figure should be given justifying the issues that the authors take with VELO. Especially since this paper quantitatively addresses this issue in Table 1 and a simple referral to that section would be enough.

- L153: "2-hidden layer, 4-hidden MLP" this part doesn't make sense. 4-hidden MLP?

- L155: The usage of the word "lean" is vague and uncommon as far as I'm aware. Did you mean flexible?

- L189: “is probably the closest”. Again, casual writing style is not appropriate. 

- The structuring of paragraphs is odd. As an example, in the preliminaries section, some paragraphs are only a single sentence.

- One of the problems mentioned in the intro is about the short horizon and how far we need to move from the pretrained model to train it. But then in the next few paragraphs, the authors mention that they want to focus on finetuning pretrained models. Wouldn’t that make the problem of short horizons moot not only for this work but also for the previous works that supposedly had problems with long horizons?

- I didn’t find a comparison to ATMO Landro et al. (2021) in the paper especially since it is mentioned in the paper that ATMO is the closest to this work. Is the ablation at L476 a representation of ATMO?

- The experiments seem to focus on one structure and one variant of that structure. More structures are needed to show that the approach isn't limited to resnet only especially since more complicated mechanisms like attention don't exist in ResNet-34 making them unsuitable for a considerable number of current day problems.

- Section 5, meta-evaluation and baselines: why are those baselines chosen? there is no description of the reasoning as to what the point of comparison to those baselines is. This is referring to the case of cosine, cosine head, const and const head. If the point is to vary the learning rate scheduler, there are many other schedulers. What makes cosine scheduling so special to act as a baseline? Is it the popularity of cosine schedulers? To note, I'm asking for a description for the reasoning of choosing those baselines. The VELO models have descriptions giving reasoning as to why they were chosen for comparison but the other baselines don't.

- Since the work is focused on narrow datasets and finetuning scenarios, I’d expected to see more datasets with varying granularity to showcase the effectiveness of this method.

Nitpicks:
Tables should be changed to booktabs format for readability if the venue rules allow.
Figure 3 caption is hard to read.
L200, comma missing at the start of paragraph.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. How is the proposed approach perform against the other alternative such as Shampoo optimizer?

2. How is the proposed approach determine if the learned optimizer is useful to help the model to be trained to converged rather than reaching certain accuracy?

3. Can you give more back envelop estimation on the computational cost versus the benefits of the proposed approach bringing in in the actual useful applications?
soundness: 3
strengths: 1. L3RS is a unique learning-to-learn optimizer approach that focuses on the fine-tuning cases. This is different from the standard fine-tuning optimizer such as SGD or ADAM, and it is also very different from the general learning-to-learn optimizer such as VELO.
2. The proposed approach is evaluated on various experimental settings such as ImageNet and PLACES datasets. The proposed experimental setting seems to be typical for learning-to-learn related papers.
3. L3RS shows a significantly speed up than the VELO and baseline in the benchmarks.
summary: Typical model training requires an optimizer such as SGD, or ADAM and tune the hyper parameters such as learning rate. One alternative direction is to have meta-learning, where a general-purpose optimizer is trained from multiple tasks and then apply to the specific problem. This meta-learning approach such as VELO is computational expensive. This paper tries a different direction which is to learn a specialized learned optimizer namely L3RS to focus on improving the pre-trained model fine-tuning. Different from the VELO approach, the proposed L3RS uses smaller MLP networks for per-layer operations and relies on base optimizers. This design choice leads to two to three orders magnitude fewer parameters compared to the VELO. The proposed L3RS shows 50% speed up over VELO and 100% speedup over the best hand-designed optimizer on ImageNet benchmark.
weaknesses: 1. Experiments are not comprehensive and convincing. The actual baseline used in the experiments is VELO. However, the field has many other famous approaches, such as Shampoo optimizers. Can the proposed approach show better results than Shampoo as well? I am also not sure about the generalization of the proposed approach. Specifically, how would one know that the layers trained with the proposed approach are not stuck in the local optimal? Can you provide any theoretical or experimental analysis? It seems to be odd to focus only on achieving accuracy. It would be better to see if the model is converged.
2. Resources. The proposed approach is still considerably expensive, and it might not be that attractive to the practitioner if it only helps to optimize the optimizer to adapt to the specific dataset with specific model fine-tuning. It would be great to justify the resources.

Meta-review: The submission introduces L3RS (Learned Layer-wise Learning Rate Scheduler), an optimizer that learns to combine the updates provided by hand-designed optimizers such as SGD and Adam, as well as heuristics such as multi-scale EMA.
After the initial round of reviews, this received scores of 5, 3, 5, 5. 
All reviewers raised the concern that the method was not evaluated on a sufficient number of datasets, architectures, and tasks.

Most importantly, during the rebuttal, the authors uploaded a revision that de-anonymized the submission and revealed their names and affiliations. This was noticed and flagged by the reviewers and AC. This is in violation of the reviewing policy and is sufficient grounds for desk rejection.

Decision: Reject