How far away are truly hyperparameter-free learning algorithms?
Authors: Priya Kasimbeg
Vincent Roulet
Naman Agarwal
Sourabh Medapati
Fabian Pedregosa
Atish Agarwala
George E. Dahl
Venue: Accepted by TMLR ()

Abstract: Despite major advances in methodology, hyperparameter tuning remains a crucial (and expensive) part of the development of machine learning systems. Even ignoring architectural choices, deep neural networks have a large number of optimization and regularization hyperparameters that need to be tuned carefully per workload in order to obtain the best results. In a perfect world, training algorithms would not require workload-specific hyperparameter tuning, but would instead have default settings that performed well across many workloads. Recently, there has been a growing literature on optimization methods which attempt to reduce the number of hyperparameters---particularly the learning rate and its accompanying schedule. Given these developments, how far away is the dream of neural network training algorithms that completely obviate the need for painful tuning?

In this paper, we evaluate the potential of learning-rate-free methods as components of hyperparameter-free methods. We freeze their (non-learning rate) hyperparameters to default values, and score their performance using the recently-proposed AlgoPerf: Training Algorithms benchmark. We found that literature-supplied default settings performed poorly on the benchmark, so we performed a search for hyperparameter configurations that performed well across all workloads simultaneously. The best "algoperf-calibrated" learning-rate-free methods had much improved performance but still lagged slightly behind a similarly calibrated NadamW baseline in overall benchmark score. Our results suggest that there is still much room for improvement for learning-rate-free methods, and that testing against a strong, workload-agnostic baseline is important to improve hyperparameter reduction techniques.

Published in Transactions on Machine Learning Research (6/2025)
How far away are truly hyperparameter-free learning algorithms?
Priya Kasimbeg Google Mind
kusimbeg@googgle.oo
Vincent Roulet Google Deep
oronlet@@google.com
Naman Agarwal*
Sourabh Medapati Google Derp
smedapatildggoogle.com com
Fabian Pedregosa
pedoogosa@@goggee.coo
Google DeepMind
Atish Agarwala Google Mind
thetish@google..om com
George E. Dahl Google DeepMind
gdahl@ com
Reviewed on OpenReview: https: openreview. net/ forum? id= 6B 50C55555
Abstract
Despite major advances in methodology, hyperparameter tuning remains a crucial (and expensive) part of the development of machine learning systems. Even ignoring architectural choices, deep neural networks have large number of optimization and regularization hyperparameters that need to be tuned carefully per workload in order to obtain the best results. In a perfect world, training algorithms would not require worklond-specific hyperparameter tuning, but would instead have default settings that performed well across many workloads. Recently, there has been growing literature on optimization methods which attempt to reduce the number of hyperparameters- particularly the learning rate and its accompanying schedule. Given these developments, how far away is the dream of neural network training algorithms that completely obviate the need for painful tuning?
In this paper, we evaluate the potential of learning-rate-free methods components of syperparameter-free methods. We freeze their (non-learning rate) hyperparameters to default values, and score their performance using the recently pecoool-proposed ALGOPErp: Training Algorithms benchmark. We found that literature-supplidd default settings performed poorly on the benchmark, so we performed a search for hyperparameter configurations that performed well across all workloads simultaneously. The best AALCoPERF-calibrttee learning- rate- free methods had much improved performance but still lagged slightly behind similarly calibrated NADAMW baseline in overall benchmark score. Our results suggest that there is still much room for improvement for learning-rato-free methods, and that testing against strong, workload-agnostic baseline is important to improve hyperparameter reduction techniques.
* Work done while Google Dee

---

Published in Transactions on Machine Learning Research (6/2025)
1 Introduction
Achieving good results with deep learning requires finding reasonable settings for the large number of hyperparameters! that control the model configuration and training pipeline. Finding good hyperparameter settings is far from casy, so rescarchers who are unable to tune well enough often end up with worse results and more diffieult scientifie comparisons. Unfortunately, as models get larger and more costly to train, tuning becomes even more difficult because running large mumbers of experiments is no longer feasible.
The predominant approach to hyperparameter selection in deep learning is mixture of idiosyneratic human judgment and semi-aulomated search methods. Unfortunately, relying on too much subjective judgment both inhibits scientific reproducibility and risks achieving poor results, since there is no reason to think typical researchers should be particularly adept at selecting good hyperparameter values. This is a problem both for solving individual problems but also for scientific progress at large. The difficulty of accounting for hyperparameter search when comparing methods has likely contributed to many contradictions, false conclusions, and dead ends in the machine learning literature.
In practice, researchers often use some (potentially iterative) combination of the following strategies: (1) tuning by hand and selecting points to try based on their own intuition, (2) tuning using low-dimensional sweeps over a few hyperparameters, selecting both which ones to tune and their search ranges based on human judgment, or (3) tuning using blackbox search algorithus, including Bayesian optimization, Although Bayesian optimization (Mockus et al., 1978) and other similar techniques (e.g. Li al. (2018)) can be effective when computational rosources are abundant, they provide no guidance on how to construct the scarch spaces they require as inpul (Ariafar et al., 2022). Furthermore, when computational resources are limited, the success of Bayesian optimization methods strongly depend on the quality of the input search space. This search
issue is exacerbated in popular, more simplistic alternatives such as grid search and (quasi-)randon (Bergstra and Bengio, 2012; Bousquet et al., 2017). In effect, all these mothods convert the problem of finding good hyperparameter settings into the problem of finding a good search space phich which must once again be drawn from the mysterious well of expert intuition.
There are two routes to reducing the amount of human judgment needed in hyperparameter search: the first is to develop more automatic and efficient tuning methods, and the second is to reduce the number of hyperparameters. Although ultimately these approaches are two sides of the same coin, eliminating hyperparameters emphasizes pushing the tuning procedure inside the training algorithm, and is the path we are concerned with in this work. Even if completely eliminating all training algorithm hyperparameters isn't feasible, partial progress would be worthwhile. For example, discovering which hyperparameters are most essential tune can lead to more efficient and robust methods, and help us start hyperparameter search from better points.
The real pain point of using training algorithms that require careful tuning is not the need for hyperparameter tuning per se, but the need for workload-speciftt tuning. For example, if a training algorithm required extensive tuning, but we could perform that tuning once on a small problem and reuse the hyperparameter settings successfully on radically different workloads, then the algorithm might as well be hyperparameter- Then we could tunc, say, a model trained to classify MNIST digits (LeCun et al., 1998), and find good configurations of training algorithm hyperparamelers for, say, training large scale machine translation models. The issue is not so much the number of training algorithm hyperparameters that need to be tuned as how much their optimal settings vary across workloads. The recent excitement in the literature surrounding scaling heuristics (Kaplan et al., 2020; Hoffmann et al., 2022; Wortsman et al., 2023) is fundamentally related to cross-workload hyperparameter transfer. There the goal is to use small models to optimize configurations for larger, more costly models while keeping the data distribution and basic architectural choices the same.
Within the deep learning literature, there growing body of work that attempts to eliminate optimization hyperparameters- most commonly, the base learning rate. The learning rate is perhaps the single most important training algorithm hyperparameter in most popular training algorithms (Bengio, 2012). Several
1 As is common in the deep learning litcrature, we the term "hyperparameter"" mcan any configuration parameter of the training pipeline (e.g. learning rate). Our usage deviates from the original definition of "hyperparameter" parameter of prior distribution in Bayesian machine learning.
2

---

Published in Transactions on Machine Learning Research (6/2025)
heuristics have been proposed to adjust the learning rate during training according to schedule (Loshchilov and Hutter, 2016; Smith, 2017; Li and Arora, 2019) that is a function of some desired number of training iterations, or training horizon, that is assumed to be known in advance. Such schedules typically exhibit two monotonic phases: a warm-up phase where the learning rate slowly increases up to some peak value, and a decay phase where the learning rale decreases to a small value, generally (near) zero (e.g. the schedule used in Goyal et al. (2017)). However, even these methods still require practitioners to identify an overall learning rate scale the base learning rate which must be tuned per workload.
The premise of rate-fre methods is to remove the need for a base learning rate (and ideally, the accompanying schedule) via changes to the training algorithm. Successful establishment of such methods would remove a leading contributor to hyperparameter tuning toil, and go a long way towards making tuning easier in practice. Removing the learning rate and other hyperparameters could enable better "default" optimization and regularization settings, which work well across many workloads.
In this paper, we study how useful recent learning-rate-free optimizers (Section 2.2) are as step lowards the more ambitious goal of removing workload-speciffc hyperparameter tuning from effective training algorithms. The goal of this work is to empirically study these learning-free-laanee free- algorithms. Providing insights into the training dynamics that determine their performance is beyond the scope of this work. We take the position that we must study a diverse set of workloads to shed light on how effective these approaches are at reducing the need for hyperparameter search. The ALGOPERF: Training Algorithms benchmark (Dahl et al., eell-tuungggyypeeaaaneeeee hyperparameter--eeeeee free algorithuss provides a perfect test bed
2023) and, in particular, its track for self-tuning/ to put learning-rate- free optimizers through their paces. We study the cross- moss-worklood performance of single hyperparameter settings and show the following:
Learning-rate-fri methods using default hyperparameter values from the literature perform poorly compared to ADAMW/NADAMYN baselines that use single set of sudence--assed defaults (fixed across all workloads).
Some learning- Searing-rattefinn rate-free methods greatly benefit from calibration of regularization parameters, while others have no settings which generalize well. A simple search reveals configurations which are better across all workloads simultancously.
The best learning-rates fate-free methods using these new ALcoPERRF-aliiratter configurations are competitive with (but not clearly better than) similarly calibrated ADAMW and NADAMW methods
In particular, we find that achieving good performance on ALGOPErF tasks st.ill requires a choice equivalent to a training horizon or learning rate decay schedule, which are workload-dependent Our results highlight the importance of benchmarking algorithms on a diverse set of workloads using well-posed tuning rules. The methods a promising step towards hyperparameter free meth but
current crop of learning-rate- there is still much work to be done to beat a natural baseline algorithm. Our work also suggests that tuning algorithms simultaneously across many workloads can reveal more useful default settings than those provided by papers or codebases.
2 What are hyperparameter-free methods?
For the purposes of this work, we assume that training algorithms operate on fully specified workload a combination of model architecture, loss function, evaluation metric, and dataset. We assume the purpose of training algorithm is to produce set of model parameters which performs well the validation set in accordance with the evaluation metric. This choice of framing implics that workload-agnostic regularization methods, such as weight decay and dropout, are part of the training algorithm, but architectural design choices such as the number of layers or where to place batch normalization (loffe, 2015) layers are part of the workload definition (as are application-specffic methods for improving generalization performance, such as most forms of data augmentation). The best training algorithms may not be the best optimization algorithms, minimizing the training loss is only means to end.
3

---

Published in Transactions on Machine Learning Research (6/2025)
Within this setting, we define a huppepaarmeeter-fre, self-tuning training algorithm as one which works well across variety of workloads without requiring any worklond-specific choices to be made by the user. In principle, such algorithms could be developed using hyperparameter search on a small set of workloads, and would then need to generalize to a much broader set of workloads without any further tuning. In our work, we will use the same set of workloads to discover and test. hyperparameterr-ree methods (to be discussed more concretely Section 3).
One question raised by our operational definition of a hyperparameter--ne training algorithm is how to handle the termination of training. Is the number of training steps a choice made by the user? In our view, user is free to run a training algorithm for as long as they wish, but the duration of training, or training horizon, becomes a meaningful hyperparameter of the training algorithm only when the training algorithm isn't an "interruptible" or "anytime" algorithm. Unfortunately, most practical algorithms use an optimizer with some sort of learning rate schedule which decreases to at the end of training training-deffning a training horizon. Such algorithms can achieve quite different results when trained with different horizons, with optimal training often occuring some intermediate horizon value. We will return to this point in Section 2.2, and will include information about "good" training horizon for standard methods part of our workload specification.
2.1 Hyperparameter-free training algorithms from corkload-agnosse settings
Although by our definition there are no truly hyperparameter-free training algorithms in wide use in deep learning loday, there are methods that might be useful in building hyperparameetrr-free training algorithms, namely rearning-rate-free optimizers. Learning-rate-free methods remove the base learning rate, which already provides some hyperparamcter reduction; they also use information from the specific workload instance and current loss landscape to set the learning rate, which suggests potential for aulomatic workload-specifie adaptation.
A lcarning-rato-finn ratti--- method (or indeod, any optimizor) becomes a hyperparamcten-frcc training algorithm by fixing the remaining optimizer parameters across all workloads. We can then ask the following questions:
Can learning-rate-fin methods perform well across many workloads without workload-specific hyperpa- rameter tuning?
Do learning-rate-fre methods with fixed hyperparameters outperform standard methods with fixed hyperparameters across a diverse set of workloads?
Answering these questions quantitatively on diverse set of benchmark workloads can give us insight into whether or not leamiig-rate-five optimizers are bringing us closer to hyperparameter-free training algorithms.
2.2 Candidate algorithms
the Common Baseline The difficulty of training deep networks with a simple stochastic gradient descent has led to the development of adaptive methods, the most popular of which is Adam (Kingma and Ba, 2014). Adaptivity here amounts to preconditioning the given stochastic gradients by an estimate of their second uncentered moment. In practice, Adam and its main variants have been the main optimizers used in deep learning on broad range of tasks, with small variations (Schmidt et al., 2021). In our work we focus two variants: Adam with decoupled weight decay (ADAMW, (Loshchilov and Hutter, 2017)) and Adam with Nesterov momentum (NADAMW, (Dozat, 2016)). Adam tunes the direction of the update in geomeery-dependentt way, but still requires a base learning rate to set the overall scale of the update-a a separate problem tackled by the learning-rate-fi methods we detail below.
Adaptive Algorithms:
Tuning Learning Rate by Estimating Initial Distance to Minimizer. A first line of learning-rate- free methods, also known as stepsize tuners (DoG, DoWG, D-ADAPT, PRODIGY presented below) are built on top of a theoretical analysis of a gradient descent on a convex non-smooth function. Consider a function f optimized by series of parameter iterates wa of the form w = W - nk0k where 9k is a (sub)gradient of f at wk. If f is convex and G- G-Lipschitz-cconiimuun the optimal constant learning rate for fixed horizon K
4

---

Published in Transactions on Machine Learning Research (6/2025)
is then The = D/(GVK), where D = ||wyy w.l2 is the Euclidean distance between the initial point and a minimizer wi of f (Nesterov, 2018). Adaptive gradient methods such as AdaGrad adapt the learning rate to the unknown quantity G with the help of accumulated gradients.
A more recent line of work takes a step further and estimates the distance D from the initial point wo to any solution w.- Distance over Gradients ( (DoG) (Tvgi et al., 2023) considered simply estimating D by the maximum distance between the iterates and the initial point. Distance over Weighted Gradients (DoWG) ) (Khaled et al., 2023) proposed to refine the adaptive rule used by AdaGrad equipped with DoG to take into account the varying learning rates. D-Adaptation (D-ADAPT) (Defazio and Mishchenko, 2023) exploits the convexity and Lipschitz-continnity of theoretical objective to build lower bound on D. PRODIGY (Mishchenko and Defazio, 2023) considers improving on D-ADAPT (Defazio and Mishchenko, 2023) by correcting the estimates of second moments in adaptive methods. We studied all of these algorithms in our initial experiments in Section 4; as we will see, PRODIGY the most promising in this family due to its compatibility with proconditioners like ADAM and is the focus of experiments in Scction
Continuous Coin Betting. Continuous Coin Betting (CoCoB) (Orabona and Tommasi, 2017) builds upon gambling viewpoint of stochastic optimization to adapt the learning rate. At each step, and for each coordinate, the amount of progress along the subgradient direction is seen as bet whose scale depends on past and current subgradient values and their estimated scales. The optimal betting strategy is derived theoretically and adapted to the problem at hand by online estimates of the coordinate-win wise gradient scales, Contrary to other algorithms like PRODIGY, D-ADAPT, MoMo or MECHANIC, COCOB does not use the same base scaling as Adam and rather defines its own adaptivity recipe to per-coordinate gradients.
Mechanic. The MECHANIC learning rate tuner (Cutkosky et al., 2023) (Algo. 3) is also designed from a study of theoretical convergence rates in a convex optimization framework. Its setup departs slightly from the aforementioned line of work by considering directly an online convex optimization framework as previously done in Adagrad (Duchi et al,, 2011). Most importantly, MECHANIC is designed as a wrapper around any base optimizer, enabling a scamless integration of additional heuristics such as warmups or anncaling schedules. The tuner is designed to guarantee that the produced iterales are, theoretically, not worse than the base optimizer.
Model-Based Momentum. The MoMo (Model-based Momentum) optimizer (Schaipp et al., 2023) (Algo. 4) defines ilerates by minimizing a model of the loss function function based on an a priori lower bound of the loss and past loss values and gradients. For problems where the optimal loss value non-zero, MoMo allows for online estimation of tighter lower bound. In the absence of additional momentum preconditioning techniques, MoMo reduces to Polyuk stepsizes (Polyak, 1964), celebrated technique to set stepsizes from convex optimization that has been adapted to stochastic optimization in deep learning recently (Berrada et al., 2020; Loizou et al., 2021).
2.3 Learning-rate rate--rat--reee free algorithm implementation details
Despite being pitched as learning- D-ADAPT; PRODIGY, MECHANIC, and MoMo still recommend using a predetermined schodule, with a warmup and/or decay phase. We will refer to such schedules as relative learning rale schedules to distinguish them from traditional, absolute learning rate schedules. The most common form this schedule takes is real number, computed a function of the number of steps, that is multiplied into the update provided by the optimizer, just before these updates are applied to the parameters. In other words, Staminn-ratee-ffi free methods remove the base learning rate which sets the overall scale, but not the changes in learning rate over training - including the training horizon over which decay takes the learning rate to 0. These schedules are themselves not exempt from tuning; we will return to this point Section
FFor MoMo (Schaipp al., 2023), the authors arguc that the algorithm naturally tends implement schedule for their experiments on feedforward networks. However, they also use an explicitly applied schedule for their experiments on transformers and diffusion models.
5

---

Published in Transactions on Machine Learning Research (6/2025)
Table Main additional hyperparameters of learning-rates rateiifeertteereee methods.
Algorithm
Base Learning Rate
Initial Estimates
DoG
Schedule- peak 1.
Square distance to minimizer, 10 10 + |woll/y /voll///soll [8ol"
10-8
DoWG
Schedule-based, peak Ir 1.
Square distance to minimizer, 10-4
PRODIGY/D-Ann Schedule-based, peak 1.
Distance to minimizer, 10
MECHANIC
Schedule- based, peak Ir 1.
Initial sealing, 10-4
COCOB
Constaut//schedute-baaes Ir
Gradient scales,
MoMo
Constant/achedullebees 1e-2/1.
Lower bound on objective,
Additionally, all the candidate algorithms work by estimating some unknown global quantity of the problem, such the distance between the initial point and the minimizer. These quantities are eventually estimated using data but require an initialized value. We used values from the literature; the parameters and their values are described in Table 1.
Mechanic also has a decay parameter Amech which used internally by the algorithm, and an integer kmech defining the number of estimates used to approximate the scaling factor. Both of which have been kept to their default values (Amech 10-2 and kmech requires an additional modificatton to incorporate a warmup schedule (called safeguard, warmup in the official implementation), otherwise the algorithm easily diverges. COCOB uses an additional hyperparameter a = 100 that defines the fraction of the current estimate of the gradient magnitude to cap the denominator or the scaling of the updates, see (Orabona and Tommasi, 2017, Section 6).
respectively). In practice, PRODIGY actually also
All algorithms presented above leave aside the tuning of (i) any weight decay, (ii) the momentum or exponential moving overage parameter of the gradients (used SGD with momentun or ADAM), and (iii) the exponential moving average parameter of the second moment estimate in ADAM. We will explore the consquences of tuning these additional parameters in Section 5.
3 Methods
3.1 Workloads and training
We studied the performance of the candidate algorithms by training 8 workloads from the MLCOMMONS ALGoPERf: TRAINING ALGORITHMS benchmark on the self-tuning track (Dahl et al., 2023). The ALGOPERF workloads span a diverse collection of architectures and datascts across image, text, speech, and graph domains. Each workload is specified by a Dataset, Model, Loss, and evaluation metric (see Table for a summary). The workloads this study were trained on TPUv2 2x2 slices (16GB HBM per chip), with the exception of CRITEO TTB DLRM SMALL which was trained on TPUv3 4x4 (32GB HBM per chip). This is in contrast with the original benchmark in which algorithms are trained on 8x Nvidia V100 GPUs.
The interface for the training algorithms is designed to be as workload-agnostic as possible. At each training step the interface exposes the loss function, loss value, gradients, gradient function, model weights, and learning rate. Training algorithms also have access to one additional piece of information: the workload- dependent marimum sleps. We set this quantity by first starting with the per-workload, maximum wall-clock time set by the ALGOPERF self-tuning ruleset (Dahl et al., 2023). We converted this time measurement to a steps measurement by measuring the number of steps ADAMW took in the allotted time on the ALGoPERF competition hardware. This measurement gave an upper bound on the number of steps that an algorithm should take to reach the target on the workload. We note that the algorithms we studied all took about 10% more time per step than ADAMW; therefore the maximum step restriction is generous compared to the equivalent time restriction.
The maximum number of steps plays a key role the training algorithms: we used it define workload- specific training horizons for algorithms with relative schedules. Since in our study we used a single set of hyperparameters across workloads, defined learning rate schedules follows. The learning rate nn at
6

---

Table 2: Summary of the workloads in the AlgoPerf benchmark. The possible losses are the cross- entropy loss (CE), the mean absolute error (L1), and the Connectionist Temporal Classification loss (CTC). The evaluation metrics additionally include the structural similarity index measure (SSTM), the error rate Published in Transactions on Machine Learning Research (6/2025)
step is given by 1/, f(t/thor) Malg,t where f [0, - [0, gives the shape of the schedule, thor is the training horizon, and T is the base learning rate computed each step by the algorithm. The training horizons themselves are defined relative to the maximum steps; that is, for a given workload i, &hor = a,tmax, where imax,i is the maximum steps for that workload, and a; is some fraction from O to 1. In our experiments we fixed ag across workloads to avoid an additional per-workload tuning burden.
Fixed hyperparameter training. In order to explore the potential of current training algorithms hygerparameler-fre methods, we conducted experiments where all hyperparameters were fixed across worklonds. Our experiments give a sense of how these algorithms generalize across variety of workloads without tuning. We specified the optimizer's hyperparameters, rogularization hyperparamcters, and a learning rate schedule (or equivalent), and applied those same settings across all workloads individually. The only workload-specific data were the measurements returned each step by the training interface, as well as the horizon of the learning rate schedule which was defined as fraction of the maximun steps per workload. Our experiments were inspired by the ALGOPERF competition's 's self-tuming track, which similarly scores training algorithms on their ability to generalize across workloads without workload-specific hyperparameter tuning.
(ER), the word error rate (WER), the mean average precision (mAP), and the bilingual evaluation understudy score (BLEU). This table also lists the batch sizes we used for training each of the workloads.
Task
Dataset
Model
Loss
Metric
Target Metric
Max Steps
Batch Size
Clickthrough rate prediction
Carroo 1TB
DLRMSMALL
CE
CE
0.1236
31,998
262,144
MRI reconstruction
PASTMRI
U-NET
L1
SSIM
0.724
108,567
32
Image
IMAGENeT
RESNRT-50
CE
ER
0.226
559,998
1024
classification
VIT
CE
ER
0.227
559 998
1024
Sperch recognition
LIRRSPPRCHH
CONFORMER DIaESPEaCH
CTC
WFR
0.086
240.000
256
CTC
WER
0.120
144,000
256
Molecular property
OGRG
GNN
CE
mAP
0.281
240,000
512
prediction
Translation
WMT
TRANSPORMER
CE
BLEU
30.85
399,999
128
3.2 Evidence based calibration of Marning-rate-Frie methods
One way to frame the search for a good hyperparaaeterr-fre method derived from training algorithm with multiple hyperparameters is a search for calibrated default-a single hyperparameter configuration that shows good results on diverse set of benchmark workloads. We conducted a series of experiments to find such hyperparameter settings for our candidate algorithms. Our setup doesn't 't have any held-out workloads; we used the whole set of ALGOPERF base workloads to find our cross-workload hyperparameter settings. This is a necessary first step towards solving the general problem of using one set of workloads to find evidence basod calibrated defaults which also work well on previously untested workloads.
We performed hyperparameter search on ADAMW and NADAMW baselines, as well as one learning-rate-fres algorithm from each candidate family. While the learning- Bearinng-rate-inn fate-free methods automatically set the base learning rate, finding good values for the remaining hyperparameters that govern the optimizer, regularization, warmup fraction, and training horizon is a nontrivial task.
To find good hyperparameters, we used a search procedure to maximize combined performance across the ALGOPErF tasks. At the end of this process we obtain ALGoPERr-calimmee hyperparameters for cach learning-rato-free method, which represent our best-effort hyperparameter-fie algorithms. The search procedure consisted of the following steps, described in detail below:
7

---

Published in Transactions on Machine Learning Research (6/2025)
1. Set training horizon: specify and calculate thors which determines the mumber of steps of the cosine decay schedule.
2. Quasi-random search: given fixed training horizon, run hyperparameter configurations sampled at random from search space.
3. Hyperparameter selection: given a set of hyperparameter configuration trial results, select the top-3 configurations based on aggregate performance on all workloads.
Setting Training Horizons. The inclusion of training horizon information is perhaps the biggest difference between our experimental sctup and the situation which practitioners face "in the wild." A number of the algorithms that we studied use learning rate decay schedule, which requires to define training horizon-tho the number of steps before the learning rate decays to O. The success of these algorithms is highly sensitive to the choice of training horizon. We chose the training horizon as fractions based on the maximum runtimes provided by ALGOPERF. We converted the ALGOPERF maximum runtimes to a maximum number of steps (as detailed in Section 3.1), which was then used to pick training horizons per workload. We explored performance over a fixed set of horizons corresponding 33%, 50% and 66% of the maximum steps (per-workload). Preliminary experiments suggested that increasing the training horizon beyond 66% does not result in any gains in ttteretttarget metrics, we did not further analyze performance on longer training horizons.
Random Search. For each algorithm, we first perform a quasi-random search (Bousquet el al., 2017) on each of the training horizons specified. To perform the random search we devised a search space over the hyperparameters. This space contains both discrete and continuous sets depending on the hyperparameter. The search space is defined Table 3; base learning rate is only tuned for the ADAMW /NADAMW baselines. Note that the Tabel Smoothing hyperparameter is only relevant to the workloads that support that regular- ization method in the current implementation of the benchmark workloads, namely IMAGENET RESNET-50, IMAGENET VIT, OGBG GNN, WMT TRANSFORMER. From this search space we drew 200 points at random, where each point contains the hyperparameter settings to configure a single set of experiments on the 8 ALGOPERP workloads. We performed this search on all of the algorithms at least on the 50% training horizon. For algorithms that yielded promising results on the 50% training horizon we also performed this search on the 33% and 66% horizons.
Table 3: Search space for bvidence-based search procedure.
Paramoter
Scale
Range
Base Learning Rate Log
[1e-4, 5e-2]
Warmup
Discrete
(0.02, 0.05, 0.1}
Weight Decay
Log
[1e-5, 0.5]
1 A1
Log
[1e-3, 1.0]
1
Log
[1e-3, 1.0]
Dropouts (tied)
Discrete
(0.0, 0.1)
Label Smoothing
Discrete {0.0, 0.2}
Hyperparameter Selection.
In line with Anonymous (2024), we used a cost function based on the gcometric mean of a time-to-targot metric to rank the 200 points. We then took the top 3 configurations under this metric and computed their ALGOPERF benchmark scores. This required re-training with 5 independent random seeds. For each algorithm-training horizon pair, the configuration with the highest benchmark score was chosen for the final comparisons. See Section 3.3 for more details on the two scoring metrics.
Baseline selection. Many of the candidate learning-rate-free algorithms that we consider in this work can be interpreted as wrappers around some base optimizer, which is often ADAMW the default implementation. Therefore it natural to compare algorithms against ADAM baselines. We compared against both ADAMW and NADAMW, there is evidence in recent literature that NADAMW may outperform ADAMW (Dahl
8

---

Published in Transactions on Machine Learning Research (6/2025)
et al., 2023). To configure the ADAMW and NADAMW baseline we used the orideence-based-search procedure detailed above, including the base learning rate as a search dimension. Note that we used the same number of search points, spread over one more dimension of search. The top 3 search candidates for each method (and the eventual baselines chosen) are detailed in Table 7.
3.3 Measuring performance
Given a set of promising algorithins, there are many methods of measuring their performance. In both our hyperparameter search as well as our final analyses we used series of performance measurements that build on each other:
Time to target. We set a validation evaluation metric larget value, and measured how quickly the target is reached (or if it is reached at all within the budget). This metric is the basis for ALGOPERF scoring, and is only useful when large enough fraction of algorithms train successfully (i.c. achieve the validation metric goal within the time limit) a large enough fraction of the time. In that setting, this metric measure training speed and how often training is successful (in terms of achieving relatively competitive validation error). We used the per-workload targets set by the ALGOPERF benchmark. These targets are useful because they provide a notion of a "competitive" value of the validation metrics for these specific workloads based on years of experience from the community. A definition of successful training based on such a well-founded notion of "good" training result helps give us performance measures that are likely to spur useful algorithmic improvements on machine learning lasks.
Performance Profile. Aggregating the performance of training algorithm over a heterogeneous set of workloads poses the challenge of appropriately balancing the different metrics and scales across different workloads. While the time to larget metric alleviates this problem by making the metric of performance uniform, the problem of balanced aggregation remains. An alternative is to present an algorithms x workload sized table of performance metrics; however such tables are often hard to parse as the number of algorithms/workkie /wrkloads grows. The ALGOPERF benchmark adopted performance profiles (Dolan and Moré, 2002), which are convenient measurement, of the performance of a set of algorithms agregated across set of workloads. For every algorithm, the performance profile plots the fraction of workloads where the given method's training time is within some ratio of the best per- worklool training time. Specifically, given a set of training algorithms S and workloads W, let l.ww for any 8 E S, E W be the time taken by the training algorithm 8 on workload w. We can now define the fraction rs,w which is ratio of the given training algorithm s's time on the workload and the training time of the best performing training algorithm on the same workload, i.e.
ts
Ta,w
min'es to
We can now define the performance profile plot. For given training algorithm s, the performance profile (T( for any 7 > 1 is defined as the fraction of workloads w, where the performance of is within 7 factor of the performance of the fastest algorithm on that workload. Mathematically,
(T( W E W Ts. & 7
W
Thus for any training algorithm, a performance profile in a single plot provides us the information of the fraction of workloads where this training algorithm performs nearly as well as any other training algorithm tested on that workload, at various levels of tolerance. Note that the performance profile is by definition a function of the entire set of training algorithms scored.
AlgoPerf Benchmark Score. Having defined the notion of a performance profile Ps(),, we distill single numerical score by considering the area under the performance curve (up to some maximum r). Intuitively, the more area covered by the performance profile of a training algorithm, the closer it performs lo optimal (amongst the comparison set) on a larger number of workloads. The area covered by the performance profile curve is referred as the AlGoOPerF benchmark score, and used to select final hyperparameter configurations (comparing different hyperparameters for a single algorithm) as well as to compare the best hyperparameter settings across algorithns. Since the benchmark score is based on the performance profiles it
9

---

Published in Transactions on Machine Learning Research (6/2025)
is also dependent the set of algorithms being compared. The benchmark score does not get credit for workloads where the target is never reached.
4 Naive learning-rate-free methods
We first considered the performance of the naive learning-rato-free bethods-"out-ofthheee any additional warmup/annealing schedules. The reason for such an initial approach is that the goal of learning-rate-free methods is to leave any tuning of the learning rate (and therefore the schedule) to the algorithms. Namely, we took the default paramcters of the algorithms as implemented by the authors in publicly available libraries, detailed in Table 1. For algorithms that employed additional momentun parameters (D-ADAPT, MECHANIC, PRODIGY, MoMo), we kept these momentum parameters as given in ADAM (3) = 0.9, 32 = 0.999). Moreover, to fully test the ability of these algorithms to remove tuning, we considered no weight decay (A 0) on the parameters.
without
4.1 Naive Gaarning-rate-fre vs. Adam-derived baselines
All of the naive learning-rate-free algorithms performed poorly compared our baselines across all workloads (Table 4). Additionally, only algorithm reached any of the targets set by the ALGOPERF benchmark, which it did for just single workload. Since so few workloads were trained successfully, analyzing the performance of these algorithms is not especially useful in the ALGOPERF framework of time-to-result scores.
We found no single, simple explanation for the poor performance these naive learning-rate setthg-rate-freee methods; detailed training curves can be found in Appendix B. Our results suggest that across a variety dlatast-modd pairs, learning-rato methods have poor performance when using defaults from papers or open sourced codebases.
In contrast, evidence-based defaults allowed the simpler ADAMW and NADAMW baselines to perform well across many workloads. A natural question is whether or not a similar procedure to find better default hyperparameter settings for learning-rate-free algorithms a question we explore in the next section.
Table 4: Best validation metrics over the course of training for baselines and naive learning-ratee rate- free methods. Numbers colored in light gray indicate that the target was not achieved, while number colored in black indicate that the target was achieved. Validation metrics for naive learning-ratte-free methods fail to beat NADAMW baselines on any workload (top row). Up arrow indicates metric is maximized, down arrow minimized.
Criteo 1TB
fastMRI
ImageNet
LibriSpeech
OGBG WMT
DLRMsmall U-Net
ResNet-50
ViT
Conformer DeepSpeech
Transformer
Metric
CE
SSIM
Error Rate
Error Rate
WER
WER.
mAP
BLEU
ADAMW
0.1235
0.722
0.252
0.222
0.085
0.118
0.254
29.64
NADAMW
0.1235
0.723
0.226
0.214
0.079
0.109
0.253
30.07
COCOB
0.1238
0.719
0.305
0.274
0.172
0.149
0.234
26.67
D-APAPT (w. ADAM)
0.1236
0.722
0.095
0.221
DoG
0.1334
0.714
0.317
0.275
0.467
0.226
0.231
26.34
DoWG
0.1259
0.715
0.389
0.998
0.844
0.839
0.205
0.00
MECHANIC (w. Anam)
0.1240
0.722
0.321
0.249
0.098
0.124
0.235
1.82
MECHANIC NADAM)
0.1241
0.721
0.318
0.248
0.099
0.123
0.239
4.50
MECHANIC
0.1361
0.714
0.301
0.203
0.183
0.209
0.045
11.94
MoMo
0.1254
0.716
0.313
0.268
0.116
0.154
0.236
25.78
MoMo (w. ADAM)
0.1240
0.723
0.313
0.999
0.368
0.454
0.221
0.99
PRODICY
0.1241
0.723
0.317
0.251
0.098
0.149
0.212
0.57
10

---

Published in Transactions on Machine Learning Research (6/2025)
5 AlgoPerf-calibratee learning-rate-fres methods
The relative success of the evidence-based basclines over naive learning-rate-fres algorithms suggest that tuning the additional optimization and regualarization hyperparameters is important for good crossworkkload performance. We applied a similar search procedure to calibrate learning-rate- rate-free methods to test their full potential as hyperparameter-free methods.
5.1 Calibrating learning- seniing-atte-ree methods
We first conducted a quasi-random search for hyperparameters while using a training horizon of 50%, using 200 points (as described in Section 3.2). We chose to analyze MECHANIC, PRODIGY, DoG, MoMo and COCOB; for expediency, we omitted D-ADAPT and DoWG, given that PRODIGY is a successor to D-ADAPT, and DoWG is quite similar to DoG. For all algorithms with ADAM/NADAM variants, used the ADAM variant to simplify exploration. For algorithms which accepted relative schedules, used lincar warmup + cosine decay schedule with a decay to the end of the 50% horizon.
In this initial search, only PRODICY and MECHANIC (with AdAM) had any hyperparameter settings which were able to train more than workloads successfully (Appendix D). Therefore, we focused the rest of our analysis on these two methods.
5.2 Analysis of AlgoPerf-calibrate methods
We calibrated PRODIGY and MECHANIC (with AdAM) using the full procedure described in Section 3.2 extending to additional training horizons and carrying out the final selection and benchmarking steps. In addition to the original 50% search, we added 33% and 66% horizons (200 points each). We took the putative top hyperparameter settings (per algorithm per horizon) and subjected them to more rigorous 5-seed evaluation to select the final,
hyperparameters using the ALCOPEre benchmark score. This is exactly the same as the procedure for the baselines, except for the fact that the base learning rate was not tuned. If removing the learning rate is truly successful, by avoiding wasting trials that search bad learning rates, there should be effectively denser coverage of the other hyperparameters in the search as compared to the baselines.
Our calibration procedure greatly improved validation metrics. Table 5 shows the best validation metrics for the 3 hyperparameter points selected for each tuned algorithm at the 33%, 50% and 66% training horizons. AlgoPerf-calibratee versions of MECHANIC and PRODIGY now can train 4 5 workloads successfully at multiple training horizons. The baselines no longer have the best scores on all workloads; in particular, the best validation metrics on IMAGENET and LIBRISPEECH DEEPSPEECH now come from learning-rate-free methods.
Since calibrated versions of PRODIGY and MECHANIC train successfully on more workloads, the ALGOPERR benchmark score for ranking the algorithms based on training speed becomes much more meaningful.
We used the ALGOPERF benchmark score (computed over all candidates collectively) to select final representative for each -lgorin pair (Table 6). On this leaderboard, PRODIGY and MECHANIC rank 3rd and 4th by AlgoPerf score. MECHANIC (with 33% horizon) reached fewer targets, but reached them faster to attain its high ranking, while PRODIGY (with 66% horizon) reached more targets somewhat more slowly to obtain its ranking. This result is reflected in the performance profiles (Figure 1). Note that MECHANIC with longer horizons did not train any additional workloads successfully, and PRODIGY with a smaller training horizon reached the target on fewer workloads. The much-improved cross-workload performance of MECHANIC and PRODIGY using evidence-basee default hyperparameters suggests that it's possible for algorithm designers to provide more useful default settings for their algorithns-ww which may improve uptake within the broader research community.
Our results suggest that PRODIGY and MECHANIC, among all the learning rateing-ratt-feee free methods we tested, are the most promising as a step towards hyperparamcter-free training algorithms. However, they are not demonstrably better than ADAMW and NADAMW at identical offiine tuning budgets. In other words, once selected single hyperparameter configuration to across all workloads, the fact that PRODIGY and
11

---

Published in Transactions on Machine Learning Research (6/2025)
MECHANIC did not need to include default value for the base learning rate in the configuration did not provide a detectable advantage over ADAMW or NADAMW. Although they are designed to adapt the overall learning rate per-workload, in our results, removing the overall base learning rate from the tuning problem did not actually seem to make the search for a configuration of the other non-lcarning rate hyperparameters that worked well across multiple workloads appreciably easier. Any benefits from removing the learning rale were not strong enough take PRODIGY and MECHANIC the top of the leaderboard, let alone get the top spot in convincing manner.
Table Best validation metric values for ALGoPPRRF-calirattod methods achieved over the course of training. Numbers colored in light gray indicate that the target was not achieved, while nunber colored in black indicate that the target was achieved. Validation metries for learning-rate--ree methods with calibrated regularization and "decay schedule" competitive with ADAMW and NADAMW baselines. Validation scores are much closer to baseline for all algorithms, and even beat the bascline on certain workloads.
Critco 1TB fastMRI
ImageNet
LibriSpeech
OGBG WMT
DLRMsmall U-Net
ResNet-50
VIT
Conformer DeepSpench
GNN
Transformer
Metric
CE
SSIM
Error Rate
Error Rate
WER 4
WER
mAP
BLEU
ADaMW 33%
0.1235
0.722
0.085
0.254
29.64
ADAMW 50%
0.1235
0.724
0.277
0.531
0.100
0.118
0.187
24.01
ADAMW 66%
0.1234
0.722
0.234
0.236
0.075
0.108
0.257
29.60
NADAMW 33%
0.1236
0.723
0.252
0,217
0.106
0.116
0:250
29.64
NADAMW 50%
0.1234
0.722
0.251
0.232
0.085
0.113
0.276
30.85
NADAMW 00%
0.1235
0.723
0.226
0.214
0.079
0.109
0,253
30.07
MECHANIC (W. ADAM) 33%
0.1240
0.724
0.238
0.225
0.084
0.113
0.245
29.91
MECHANIC (w. AnAM) 50%
0.1234
0.723
0.245
0.213
0.085
0.1
0:239
2.59
MRCHANIC (w. AnAM) 66%
0.1235
0.723
0.275
0.222
0.082
0.111
0.245
25.53
PRODICY ADAM) 33%
0.1239
0.723
0.082
0.117
0.198
30.11
PRODICY (W. ADAM) 50%
0.1234
0.722
0.223
0.213
0.087
0.107
0.244
29.91
PRODICY (W. ADAM) 68%
0.1236
0.721
0.223
0,208
0.085
0.103
0.220
28.81
Table 6: ALcooPeRR-clliibraeee methods show mixed promise terms of time-to-target metrics. Biggest improvements are on IMAGENET ViT and LIBRISPEECH CONFORMER workloads. reemphasize that the score of these reprosentative examples is function of the entire algorithm pool, including the points that were not selected and suppressed from Table 6. The scores and time-to- target ratios of the complete set of algorithms that was used to compute the scores can be found in Appendix E.
important
A AlroPerf Criteo 1TB fastMRI
ImageNet
LibriSpeach
OCBG
WMT
Score
DLRMsmall
U-Net ResNet-50 VIT Conformer DeepSpeech GNN Transformer
ADAMW 33%
0.303
0.245
NADAMW 66%
0.480
0.429
>1
0.782
0.609
0.435
0.501
1
>
PRODICY (w. ADaM) 66%
0.477
0.470
>1
0.773
0.521
0.491
U.500
>
>
MEXHANIC (w. АnAM) 33%
0.469
0.130
0.335
0.248
0.303
>1
PRODIGY (w. ADAM) 50%
0.455
0.388
0.577
0.404
0.381
NADAMW
0.452
0.335
>
>1
0.369
0.412
>
0.457
MECHANIC (W. AnAM) 50%
0.434
0.382
5
>
0.412
0.375
0.374
>
MECHANIC (w. ADAM) 66%
0.375
0.427
>1
21
0.608
0.436
0.478
31
3
NADAMW 33%
0.369
0.320
21
>1
0.303
0.286
51
31
ADAMW 60%
0.276
0.511
0.419
0.518
PRODICY (w. ADAM) 33%
0.245
>1
21
>1
>1
0.242
0.317
>1
>1
ADAMW 50%
0.212
0.489
0.229
>1
>1
>
0.192
21
>1
12

---

Published in Transactions on Machine Learning Research (6/2025)
100%
AdamW 33%
NadamW 66%
Prodigy AdamW 66%
Percentage of Workloads
Mechanic Adam'W 33%
75%
50%
25%
0%
1.0
1.5
2.0
2.5
3.0
3.5
4.0
T
Figure 1: Porformance profiles of top 4 algorithms from Table 6. These curves represent the best performances, by ALGOPERF benchmark score, for ADAMW, NADAMW, PRODIGY, and MECHANIC over 3 different horizons and 3 hyperparameter settings per moaritmmhhoiioon pair from the broad search. ADAMW and PRODIGY obtain their high scores by hitting targets for 4 workloads relatively quickly. NADAMW and PRODIGY hit 5 targets, but more slowly to obtain their scores.
6 Discussion
Our experiments represent the first comprehensive study of how well learning-rate-fre methods generalize across workloads, and whether they actually provide a practical benefit in terms of simplifying the overall hyperparameter tuning problem. Our experiments address two important measurement pitfalls: comparing learning-rato-fin method to (say) ADAM using carefully tuned lcarning rate that is tuned per-workload provides overly pessimistic assessment, but comparing lo standard optimizer using library defaults is overly optimistic, Instead, we only compared to baselines that also did no workload-specific tuning, which also lets us assess the potential role of learning-ratc-free training algorithms as a component of future hyperparamettrr-free training algorithmss By using the AlgoPerf benchmark to ground our calibration procedure, our characterization of the performance of learning-rate-fi realistic deep learning workloads and a meaningful training success criterion (based on reaching a good validation performance). As result, calibration procedure found evidence-based default hyperparameter settings that dramatically improved upon the library and paper defaults we started with.
methods reflects a diverse set. of
We focused on the "Iyperparameterrfreee"""" comparison setting because learning-rate free methods are the single most studied hyperparamoter reduction technique deep learning. The primary motivation learning-rate rate- free methods that they can select good learning rates per workload, and even per individual step of training: natural followup question, then, whether or not this capability actually leads to better cross- wosssworkload performance without re-tuning. We found that the "default" values reported in the literature, found in standard implementations, did not lead to good performance most workloads.
Indeed, we had to tune over 31 and B2 for ADAM derived methods, as well as the regularization parameters to obtain n.cooPorr-calibrraee mot that hit a majority of the validation largets in the allouted training budget. This result suggests that developers new algorithms would be well-served by finding calibrated, svidenco-based defaults and publishing them papers and open source code. Such a process should include tuning of regularization parameters, which can have optimizer-dependen interactions. Better justified default values could improve adoption of these methods, and could speed up overall research in the field.
13

---

Published in Transactions on Machine Learning Research (6/2025)
The learning-rate-free methods studied here avoided the need to tune the base learning rate only. Another hidden tuning parameter in learning-rate-fres methods is the training horizon, or more generally, the learning rate schedule. All the best performing methods we examined in this work still needed some kind of relative learning rate schedule with warmup and decay. In this sense, they are not truly learning-rate-free algorithm proposed in Defazio et al. (2024) attempts to remove the learning rale schedule; the resulting SCHEDULE- ADAM convincingly won the self-tuning track the recent ALGoPErR self-tuning public competition (Kasimbeg et al., 2025). In Appendix F we make best-effort comparison between the competition version of SCHEDULE-FREE ADAM and find that its estimated benchmark score is better than any method obtained by our evidence based search, in large part because of its ability to hit 7 out of the 8 validation targets (compared to the maximum of from algorithms in our search). This suggests that successfully removing the learning rate schedule may be just as important as removing the need for a base learning rate.
A recent
In the end, even the best performing ALGoPraaF-calibratee learning-rate- algorithms performed slightly worse than the best ADAMW /NADAMW basclines- even though the same number of potential configurations were searched to produce them. This result suggests that the learning-rate-free methods we tested don't 't yel. save time on hyperparameter tuning, and don't provide significant cross- workload generalization over more conventional methods.
One limitation of our study is the selection of workloads. While we believe that ALGOPERR covers a broad, rell-motivated set of problems, some effects can't be tested using the setup. In particular, many practical settings now involve training models at a variety of scales, to extrapolate good hyperparameters for training a final, massive model (Hoffmann et al., 2022). We know theoretically and empirically that most simple training algorithms require scale-specific values for hyperparameters (Lee et al., 2019). In these settings, calibrated hyperparameters as we've defined them do not exist without techniques like dimensionally-awaan parameterizations or scaling heuristics for hyperparameters (Everett et al., 2024). Incorporating these additional considerations could shed more light on the promise of learning-rate-fin rate-free methods as the basis for hyperparameter-fice training algorithms.
Our analysis identified PRODIGY and MECHANIC as the two most, promising methods among those we tested. We believe that our experimental approach should be emulated in future work proposing new methods and provides an example of how to more rigorously test the efficacy of learning-rate-free methods. Though none of the raaeinngraate-ree hods we tried are yet ready to supplant standard optimizers, we believe the future is bright for the development of such adaptive optimizers, now that our benchmarking and evaluation capabilities have improved.
References
Anonymous (2024), 'Training neural networks faster with minimal tuning usingpre-computed lists of hyperpa- rameters for NAdamw', Submitted to Transactions on Machine Learning Research. Under review. URL: htlps://ppenreeieu.a net/ f
Ariafar, S., Gilmer, J., Nado, Z., Snoek, J., Jenatton, R. and Dahl, G. E. (2022), Predicting the utility of search spaces for black-bor box optimization: a simple, budget-aware approach, in G. Camps-Vall Valls, F. J. R. Ruiz and I. Valera, eds, 'Proceedings of The 25th International Conference on Artificial Intelligence and Statistics", Vol. 151 of Proceedings of Machine Learning Research, PMLR, pp. haa151//rraffr22aa.hhm htwped/nwwn..nnpppee///////////a
URL: http:s////oocceigs.ooo wwt.ppres//u111///
Armijo, L. (1966), 'Minimization of functions having lipschitz continuous first partial derivatives', Pacific Journal of mathematics 16(1), 1-3.
Bengio, Y. (2012), Practical recommendations for gradient-based training of deep architectures, in 'Nuura Networks: Tricks of the Trade: Second Edition', Springer, pp. 437-478.
Bergstra, J. and Bengio, Y. (2012), 'Random search for hyper-parameter optimization', Journal of Machine Learning Research 13(10), 281-305 URL: htlp://jmmr: ppppps//u3//brrgge 1 ht.l
14

---

Published in Transactions on Machine Learning Research (6/2025)
Berrada, L., Zisserman, A. and Kumar, M. P. (2020), Training neural networks for and by interpolation, in "International conference on machine learning', PMTR.
Bousquet, O., Gelly, S., Kurach, K., Teytaud, O. and Vincent, D. (2017), 'Critical hyper-paramenn parameters: No random, no cry'.
URL: https:/ https:///rrii...o /trp//..rroo//////// haag///b///// ftabs/170600300
Cutkosky, A., Defazio, A. and Mehta, H. (2023), 'Mechanic: A learning rate tuner', ar preprinl 0rXiv:336600114.
Dahl, G. E., Schneider, F., Nado, Z., Agarwal, N., Sastry, C. S., Hennig, P., Medapati, S., Eschenhagen, R., Kasimbeg, P., Suo, D. et al. (2023), 'Benchmarking neural network training algorithms', arXiv preprint arXiv:2066.07179.
Defazio, A., Cutkosky, A., Mchta, H. and Mishchenko, K. (2023), 'When, why and how much? adaptive learning rate scheduling by refinement', arXiv preprint arXiv:2310.07891
Defazio, A. and Mishchenko, K. (2023), reaerie learning by d-adaptation', arXiv preprinl arXiv:2301.07933.
Defazio, A., Yang, X. A., Mehta, H., Mishchenko, K., Khaled, A. and Cutkosky, A. (2024), 'The road less scheduled', Xr preprint or arxi::4405116682
Dolan, E. D. and Moré, J. J. (2002), 'Benchmarking optimization software with performance profiles', Mathematical programming 91, 201-213.
Dozat, T. (2016), 'Inorpporating nesterov momentum into adam'.
Duchi, J., Hazan, E. and Singer, Y. (2011), 'Adaptive subgradient methods for online learning and stochastic optimization', Journal of machine learning research 12(7).
Everett, K., Xiao, L., Wortsman, M., Alemi, A. A., Novak, R., Liu, J., Gur, I., Sohl-Dicksteii J., Kaelbling, L. P., Lee, J. et al. (2024), 'Scaling exponents across parameterizations and optimizers', arXiv preprint arXiv:2407.08872.
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y. and He, K. (2017), 'Aceurate, large minibatch sgd: Training imagenet in 1 hour', arXiv preprint arXiv: 1066 0200002677...
Hager, W. W. and Zhang, H. (2005), 'A conjugate gradient method with guaranteed descent and efficient line search', SIAM Journal on optimization 16(1), 170-197 192.
Hestenes, M. R., Stiefel, E. et al. (1952), Methods of conjugate gradients for solving linear systems, Vol. 49, NES Washington, DC.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. and Sifre, L. (2022), An empirical analysis of compute-optimal large language model training, in S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho and A. Oh, eds, 'Advances in Neural Information Processing Systems', Vol. 35, Curran Associales, Inc., pp. 30016-30030.
Loffe, S. (2015), 'Batch normalization: Accelerating deep network training by reducing internal covariate shift', arXiv preprint XrXim:1502.08166
Ivgi, M., Hinder, O. and Carmon, Y. (2023), 'Dog is sgd's best friend: A parameter-free dynamic step size schedule', arXiv preprint arXiv:2302. 12022.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J. and Amodei, D. (2020), 'Scaling laws for neural language models', Xix preprint or 20x::2000008800
15

---

Published in Transactions on Machine Learning Research (6/2025)
Kasimbeg, P., Schneider, F., Eschenhagen, R.,, Bac, J., Sastry, C. S., Saroufim, M., Feng, B., Wright, L., Yang, E. 7., Nado, 7.., Medapati, S., Hennig, P., Rabbat, M. and Dahl, G. E. (2025), 'Accelerating neural network training: An analysis of the algoperf competition!.
URL: https://orrri.. orgg org/abs///aareeeoooo ha//aabs/20/ 22//2252.1550 15015
Khaled, A., Mishchenko, K. and Jin, C. (2023), 'Dowg unleashed: An efficient universal parameter-free gradient descent method', Advances in Neural Information Processing Systems 36, 6748-6769.
Kingma, D. P. and Ba, J. (2014), 'Adam: A method for stochastic optimization', arXiv preprint orXiv:1412.695.
LeCun, Y., Bottou, L., Bengio, and Haffner, P. (1998), 'Gradient-based learning applied to document. recognition', Proceedings of the IEEE 86(11), 2278-227788 2324..
Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J. and Pennington, J. (2019), 'Wide neural networks of any depth evolve as linear models under gradient descent', Advances in newral information processing systems 32.
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A. and Talwalkar, A. (2018), 'Hyperband: A novel bandit- based approach to hyperparameter optimization', Journal of Machine Learning Research 18(185), 1 52.
Li, Z. and Arora, S. (2019), 'An exponential learning rate schedule for deep learning', aiv preprint
Loizou, N., Vaswani, S., Laradji, Н. and Lacoste-Julien, S. (2021), Stochastic polyak step-size for sgd: An adaptive learning rate for fast convergence, in 'International Conference Artificial Intelligence and Statistics', PMLR.
Loshchilov, and Hutter, (2016), Sgdr: Stochastic gradient descent with warm restarts, in IInternatinnal Conference on Learning Representations:
Loshchilov, I. and Hutter, F. (2017), 'Decoupled weight decay regularization', arv preprint arXie:1711.005
Mishchenko, K. and Defazio, A. (2023), 'Prodigy: An expeditiously adaptive parameter-free learner', arXiv preprint ar 2tff::2066011000
Mockus, J., Tiešis, V. and Žilinskas, A. (1978), 'The application of Bayosian methods for seeking the extremum',, Towards Global Optimization 2(117-129), 2.
Moré, J. and Thuente, D. (1994), 'Line scarch algorithms with guaranteed sufficient decrcase', ACM Transactions on Malhematical Software (TOMS) 20(3), 286 307.
Nesterov, Y. (2018), Lectures on conver optimization, Vol. 137, Springer.
Nocedal, J. and Wright, S. J. (1999), Numerical optimization, Springer.
Orabona, F. and Tommasi, (2017), 'Training deep networks without learning rates through coin betting', Advances in Neural Information Processing Systems 30.
Polyak, B. T. (1964), 'Some methods of speoding up the convergence of itcration mothods', USSR computational mathematics and mathematical physics 4(5), 17.
Schaipp, F., Ohana, R., Eickenberg, M., Defazio, A. and Gower, R. M. (2023). 'Momo: Momentum models for adaptive learning rates', arXiv preprinl ar Xio::2305.000 07300.
Schmidt, R. M., Schneider, F. and Hennig, P. (2021), Descending through crowded valley-benchmarking deep learning optimizers, in 'International Conference on Machine Learning', PMLR, pp. 9367-937777 9376.
Smith, L. N. (0017), Cyclical learning rates for training neural networks, in '2017 IEEE winter conference on applications of computer vision (WACV)' IEFF, pp. 464-472.
16

---

Published in Transactions on Machine Learning Research (6/2025)
Wolfe, P. (1969), 'Convergence conditions for ascent methods', SIAM review 11(2), 226 235.
Wortsman, M., Liu, P. J., Xiao, L., Everett, K., Alemi, A., Adlam, B., Co-Reyes, J. D., Gur, L., Kumar, A., Novak, R.:; Pennington, J., Sohl-dickstein, J., Xu, K., Lee, J., Gilmer, and Kornblith, S. (2023), 'Small-scale proxies for large-scale transformer training instabilities', arXiv preprint arXiv:2309. 14322.
A How to tune Parameter-Free algorithms?
Unfortunately, the recent proliferation of Learning-rate free methods Mishchenko and Defazio (2023); Defazio and Mishchenko (2023); Defazio et al. (2023); Cutkosky et al. (2023); Ivgi et al. (2023); Orabona and Tommasi (2017) are still far from being parameler-free methods. All of these papers lackle the learning- rate hyperparameter in neural network training, leaving no or defoult prescriptions to be used for other hyperparamoters for eg. first and second moment accumulation hyperparameters (31,82 Adam-like methods), weight decay and other regularization hyperparameters such as label smoothing and dropout. While for well-established and repeatedly trained on workloads in deep learning a usable value for these parameters can be found in literature however those values are unlikely be usable in practice more generally. Further while these algorithms tackle the learning rate scalar, they often employ learning rate schedule to reach good performance in deep learning tasks, thereby not making the learning rate fully parameter-fres well. Learning rate schedules need to be treated as milti-dimenssonal hyperparameter in itself. Even when you fix the rough 'shape' of the learning rate schedule, for example for the commonly used warmup cormuptcosine decay schedule, parameters like how many steps (or what percentage of training) of warmup is performed or when does the decay end still needs be determined.
We consider the following set of recently proposed parameter-free algorithms in this context.
D-ADAPT Defazio and Mishchenko (2023)
MECHaNIC Cutkosky et al. (2023)
PRODIGY Mishchenko and Defazio (2023)
DoG Tvgi et al. (2023)
We perform the following experiments on these algorithms
1. No tuning: We run the presented algorithms without any modification from their default values as found in the Optax implementations. For the regularization hyperparameters, since there are no well-defined values, we choose the default values supplied, which effectively, turns these parameters off (i.e. dropout rates, label smoothing strength, and weight decay of 0). Note that this also means we do not provide an external learning rate schedule and the only schedule applied any maintained by algorithm. To provide a point. of comparison to these algorithms, we ran experiments with ADAM and NADAM with default hyperparameters (and the same default regularization hyperparameters) with an additional sweep over the (constant) learning rate.
Tuning other hyperparameters In this section we use the above algorithms with their default optimization hyperparameters and tune the following regularization and LR schedule parameters on top of it
Weight Decay [1c-1, 1c-2, 1c-3, 1c-4, 1c-5, 1c-6, 1c-7]
Dropout (tied) [0.0, 0.1]
Label Smoothing [0.0, 0.1, 0.2]
Training Horizon [0.33, 0.49, 0.64, 0.79, 1.0]
Warmup Fraction [0.02, 0.05, 0.10]
17

---

Published in Transactions on Machine Learning Research (6/2025)
B
Naive learning-rate- rate-freeeeeeeeeeeeee methods training curves
Criteo TB
Validation CE Loss
0.18
0.16
0.14
0.12
2.5
1.0
2.0
3.0
Train Step
x10
Train Step
x104
NadamW baseline
Mechanic SGD Momentum
NadamW baseline
MoMo
AdamW baseline
Prodigy
AdamW baseline
MoMo AdamW
Target (0.124)
D-Adaptation AdamW
Mechanic NadumW
DoWG
COCDB
Figure 2: Validation CE loss vs training step for 'out-of-the-box' learning-rate-frre methods and baselines on CRITEO 1TB DLRM SMALL.
FastMRI
0.750
Validation SSIM
0.725
0.700
0.675
0.650
0.625
0.600
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Train Step
x10
Train Step
x10$
NadamW baseline
Mechanic SGD Momeotum
NadamW baseline
MoMo
AdamW baseline
Prodigy
AdamW baseline
McMo
Mechanic AdamW
Target SSIM (0.724)
DeG
D-Adaptation AdamW
Mechanic NadumW
DeWG
COCOD
Figure 3: Validation SSTM vs training step for 'out-of-the-box' raariingg-ate--rre-mell and baselines on PASTMRI.
18

---

Published in Transactions on Machine Learning Research (6/2025)
Imagenet ResNet50
10"
Validation Error Rate
6
40-1
5
Train Step
x10
Train Step
x10
Nadam W baseline
Mechanic SGD Momentum
NadamW baseline
MoMo
Adami baseline
Prodigy
Adami
MoMo AdamW
Mechanic AdamW
Target Ervor (0.226)
DoG
D-Adaptation AdamW
Mechanic NadumW
DoWG
COCOB
Figure 4: Validation Error Rate vs train step for 'out-of-the-box' learning-rate-fres methods and baselines on IMAGENET RESNET-50.
Imagenet VIT
100
Validation Error Rate
6 60-1 10
4x-0-1
2 10-1
5
Train Step
x10
Train Step
x10
Nadam
Machanic
NadamW
AdamtW baseline
Prodigy
AdamW baseline
MoMo AdamW
Mechanic AdamW
Target Error (0.227)
DoG
D-Adaptation AdamW
Mechanic NadamW
DoWG
COCOB
Figure 5: Validation Error Rate vs train step for 'out-of-the-box learning-rate-free methods and baselines on IMAGENET VrT.
19

---

Published in Transactions on Machine Learning Research (6/2025)
LibriSpeech Conformer
10a
Validation WER
Train Step
x10
Train Step
x10
NadamW baseline
Mechanic SGD Momentam
NadamW baseline
MoMo
AdamW baseline
Prodigy
AdamW baseline
Mechanic Adam
Target WER (0.06))
DoG
D-Adaptation AdamI
Mechanic Nadam
DuWG
COCOB
Figure Validation WER vs train step for 'out-of-the-box" learning-rate-free methods and baselines on LIBRISPEECH CONFORMER.
LibriSpeech Deepspeech
100
Validation WER
0.50
0.75
1.00
0.25
0.50
0.75
Train Step
x10
Train Step
x10
NadamW
Mochanic
AdamW baseline
Prodigy
Adam baseline
Adam
Mechanic AdamW
Target WER (0.120)
DoG
D-Adaptation AdanI
Mechanic NadamW
DoWG
COCOB
Figure Validation WER vs train step for 'oul-of-the-box" learning-rate-free methods and baselines on LIBRISPEECH DEEPSPEECH.
20

---

Published in Transactions on Machine Learning Research (6/2025)
OGBG
Validation Average Precision
0.25
0.20
0.15
0.10
0.05
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
Train Step
x10
Train Step
x10
NadamW baseline
Machanic Momentum
NadamW baseline
MoMo
AdamW baseline
Prodigy
AdamW baseline
MoMo
Mechanic AdamW
Target Average (0.281)
DeG
D-Adaptation AdamW
Mechanic NadumW
DeWG
COCOB
Figure Validation Average Precision vs train step for 'out-of-the-box" learning-rated mattiing-rtt-fress methods and basclines on OGBG GNN.
WMT
Validation BLEU Score
30
25
20
15
5
0
Train Step
x10
Train Step
x10$
NadamW baseline
Mechanic SGD Momentum
Nadam W baseline
MoMo
AdamW baseline
Prodigy
AdamW basaline
MoMo AdamW
Mechanic AdamW
Turge: BLEU (30.849)
DoG
D-Adaptation AdamW
Mechanic NadamW
DoWG
COCOB
Figure 9: Validation BLEU score vs train step for 'out-of-theebox" learning-rale--ree methods and baselines WMT TRANSFORMER.
21

---

Published in Transactions on Machine Learning Research (6/2025)
C Baseline selection
Table 7: ALGOPERF scores and runtime fractions of the baseline candidate meth We chose the highest. scoring NADAM and ADAMW methods as the baselines. These baseline rows are colored in gray. Note that these methods were scored in a wider population of methods. The scores and runtime fractions of the full population are given in Table 9.
AlgoPerf Criteo 1TB fastMRI
ImageNet
LibriSpeech
OGBG
WMT
Score
DLRMsmall
U-Net ResNet-50 VIT Conformer DeepSpeech GNN Transformer
NADAMW 33%
0.248
>
>
>
0.317
21 0.282
>1
>
NADASW 33%
0.369
0.320
>
>1
0.303
>1
0.286
>1
>1
NADAMIW 33%
0.248
>
>
>
0.306
> 0.296
>1
>
NADAMW 50%
0.369
NADAMW 50%
0.356
>
5
0.566
>1
0.414
0.329
31
NADAMW 50%
0.101
>
>
> 1
> 1
>1
0.112
>1
>
NADAMW 66%
0.125
1
0,084
>1
>
21
>>
>1
>1
NadamW 66%
0.480
0.429
0.782
0.609
0.435
0.501
ADAMW 33%
0.248
0.298
>
>
0.244
>
AdamW 33%
0.498
0.284
> 1
0.303
0.245
0.291
ADAMW 33%
0,125
0.284
1
1
>1
ADAMW 5U%
ADAMW 50%
0:000
>
>
2
> 1
1
1
>1
ADAMW 50%
0.242
0.489
0.229
21
21
21
0.492
21
>1
ADAMW 60%
0.272
21
0,577
0.453
0,478
21
>1
ADAMW 60%
0.276
0.511
0.419
0.518
ADAMW
22

---

Published in Transactions on Machine Learning Research (6/2025)
D Max Targets Hit in Initial Search on 50% Training Horizon
Table 8: Targets hit per algorithm initial search. For each class of algorithms we chose a representative algorithm and train with 100 hyperparameter points sampled from the initial search space on 50% training horizon. Only PRODIGY and MECHANIC (with AdAM) had any hyperparameter settings which were able to train more than 2 workloads successfully.
Algorithm Class Members
Algorithm
Max Workload Targets Hit
MECHANIC
MECHANIC
4
PRODIGY, D-ADAPT
PRODIGY
4
DoG, DoWG
DoG
0
MoMo
MoMo
2
COCOB
COCOB
0
23

---

Published in Transactions on Machine Learning Research (6/2025)
E
AlgoPerf metrics for entire set of methods scored
SCHEDULE-FRE FREE ADAM was intr
Table 9: ALGOPERF scores and runtime fractions of entire set methods that account for the performance profiles and ALGOPERF scores. Methods that were suppressod from table are indicated with gray rows. These methods were suppressed if there exist similar algorithm in the same class of algorithm and training horizon that outperformed the method in terms of ALCOPERF score. Bold numbers indicate the smallest runtime fraction over all submission on given workload.
AlgoPerf
Criteo 1TB fastMRI
ImageNet
LibriSpeech
OGBG
WMT
Score
DLRMsmall
U-Net
ResNet-50
VIT Conformer DeepSpeech
GNN
Transformer
ADAMW 33%
0.498
0.291
NADAMW 66%
0.480
0.429
>
0.782
0.609
0.435
0.501
>
PRODICY (w. ADAM) 66%
0.177
0.470
>
0.773
0.521
0.191
0.500
>
>>
MECHANIC (w. ADAM) 33%
0.469
0.130
1
0.335
0.248
0.303
>
PRODICY (W. ADAM) 50%
0.455
0.577
0.381
NADAXW
MRCHANIC (w. AnAM) 50%
0.434
0.382
0.412
0.375
0.374
MECHANIC (w. ADAM) 50%
0.399
0.513
0.486
0.346
0.448
MECHANIC (W. ADAM) 66%
0.375
0.427
1
0.608
0.436
0.478
>1
NADAMW
0.286
NADAMW 50%
0.356
0.568
0.414
0.329
MECHARIC (w. ADAM) 50%
0.308
0.450
>
0.479
>>
0.405
>
>
MECHARIC (w. ADAM) 66%
0,286
0.533
0.433
0.446
>
ADAMW 66%
0.276
0.511
0.419
0.518
ADAMW 66%
0.272
NADAMW 33%
0.248
>
0.317
0.282
NADAMW 33%
0.248
>
>
>
0.306
> 1
0.296
>
>
ADAMW 33%
0.218
0.298
>
0.244
>1
PRODICY ADAM) 33%
0.215
0.317
ADAMW 30%
0.242
0.489
> 0.229
1
1
>> 1
0.492
>1
MRCHAKIC (w. ADAM) 33%
0.242
0.348
0.292
MECHANIC (w. ADAM) 66%
0.232
0.655
0.494
0.630
>
PRODIGY (W. AnAM) 50%
0.214
0.323
0.429
PRODGG (w. AnaM)
0.204
0.450
PRODICY (W. ADAM) 33%
0.181
>
0.218
>
>
0.303
2
>
NADAMW 66%
0.125
>
0.084
>1
1
>
>
>
ADAMW 33%6
0.125
0.284
>
1 0.303
MECHANIC (w. ADAM) 33%
0.125
ADAMW 50% o
0.120
>
0.093
>
>1
>
NADAMW 50%
0.101
21
>
21
21
3
0,412
>
>
PRODIGY (w. AnAM) 66%
0.098
x
0.502
>1
>
PRODGGY (W. AnAM) 66%
0.091
0.548
>
NADAXW
>
>1
>1
ADAMW 66% 2
0.000
>
>
>1
>1
>
>1
>1
PRODICY (w. ADAM) 33%
0.000
>1
21
>1
>1
51
>1
>1
>1
ADAMW 50%
0.000
21
>1
21
>1
>1
>1
24

---

Published in Transactions on Machine Learning Research (6/2025)
F Adding Schedule-free Adam
SCHEDULE-FREE ADAM introduced by Defazio et al. (2024) achieved first place on the self-tuning track of the inaugural ALGOPERF competition Kasimbeg et al. (2025). At the time of writing, JAX imple- mentation of SCHEDULE-FREE ADAM capable of reproducing the runtime performance observed in the ALCOPERF competition was unavailable. Consequently, we were unable produce SCHEDULE-FREI ADAM algorithm using the evidence-based search procedure described in 3.2.
To provide a good-faith comparison, evaluated the SCHEDULE-FREE ADAM submission against ALGoPERRF-caliratter algorithms by extracting the runtime fractions of workload budgets from the the results of the ALGOPERF competition. Using the runtime fractions, we recomputed the ALGOPERF scores with SCHEDCLE--REF ADAM in the pool of submissions. Since the runtime budgets in our experimental setup are calibrated for the ALGOPERE setup we believe the fractions are good enough approximations for cross-platform comparisons. Note that this remains an approximation and that an ideal selup the SCHEDULE- FREE ADAM algorithm would have been evaluated on the same hardware as the rest of the algorithms in the scoring pool.
The results show that SCHEDULE-FREE ADAM beats MECHANIC, PRODIGY and our baselines by large margin in ALGOPERF score. The SCHEDULE- FREE ADAM reaches the target on 7/8 workloads, whereas the other algorithms reached the targets on a maximum of 5 workloads. The SCHEDULE-FREd ADAM algorithm also achieved the fastest runtime on of the workloads.
AlgoPerf
Criteo 1TB fastMRI
ImageNNt
LibriSpeach
OGBG
WMT
Score
DLRMsmall
U-Net ResNet-50
VIT Conformer DeepSpeech
GNN Transformer
SCHEDULE- FARR
0.225
0.314
ADAMW 33%
0.478
0.284
>>
0.303
0.245
0.291
1
>>
PRODICY (w. ADax) 66%
0.443
0.470
>
0.773
0.521
0.491
0.500
>
>> 1
NADAM 66%
0.442
0.429
>
0.782
0.609
0.433
0.501
>1
PRODIGY (w. ADAM) 50%
0.428
0.388
0.577
0.404
0.381
NADAMW 50%
0.426
0.335
>
>
0.369
0.412
>
0.457
MECHANIC (w. ÁUAM) 33%
0.409
>
0.130
>
0.335
0.248
0.303
>
> 1
MECHANIC (W. ADAM) 50%
0.407
0.382
>1
>
0.412
0.375
0.374
>
NADAMW 33%
0.348
0.320
0.303
0.286
>1
>
ADAM)
21
0.608
ADAMW 60%
0.206
0.511
21
>
>
0.419
0.518
21
>1
PRODICY (w. ADAM) 33%
0.215
>
21
>1
0.242
0.317
21
>1
ADAMW 50%
0.179
0.489
0.229
>1
0.192
21
>1
Table 10: SCHEDULE-FRED ADAM outperforms the ALCoPERP-calibrtted learning-rated rattine--att--ree algorithms and basclines.
G A brief history of learning-rate- free methods
G.1 Linesearches in deterministic setting
The design of hyperparamctor-frec methods naturally depends on the algorithm itself and the objective hand. For simple objectives such as convex quadratics, there exist optimal algorithms that are hyper- parameter free such as the conjugate gradient method (Hestenes et al., 1952). However, even for simple convex objectives, optimization algorithms have generally been designed given known properties of the objective such as the smoothness or the strong convexity of the objective, which translates in the selection of some hyperparameters (Nesterov, 2018).
To illustrate the development of typerparamcter-frec methods, we focus on a simple gradient descent. We consider the non-stochastic case start with. gradient descent requires the selection of learning rate, often called stepsize in the optimization literature. To select the stepsize, several criterions have been proposed.
25

---

Published in Transactions on Machine Learning Research (6/2025)
The first and most common criterion has been to enforce sufficient decrease of the objective (Armijo, 1966). Namely, the stepsize 1y is selected such that f'wwx mpss) & f((wx) + e ((wx) us for -(((w) some small constant (of the order of 10 for example (Nocedal and Wright, 1999)), and wy the current parameters. The stepsize 1) is then found by means of a while loop that decreases the stepsize by some constant factor until the aforementioned criterion satisfied. Equipped with such criterion simple gradient descent has, up to logarithmic factors, the same convergence guarantees as with known smooth and strong convexity constants (Nesterov, 2018).
Ideally, may rather want to select the stepsize to minimize the function along the update direction. Such endeavor is generally too computationally expansive to be undertaken. To circumvent this issue, Wolfe (1969) proposed an additional curvature condition of the form -ug V + muu) & -C24 2 V V((w).. Such a curvature condition ensures that the selected stepsize is close to a local minimum of 17 - f(wy + 174)). Blending both sufficient and curvature conditions an algorithm can be done by some form of dichotomy that carefully accounts for the geometry of the search (Moré and Thuente, 1994; Nocedal and Wright, 1999).
Additional linesearches have been proposed using for example approximate sufficient decrease condi- tion (Hager and Zhang, 2005) alternative measurements of the curvature like Goldstein conditions (Nocedal and Wright, 1999). All aforementioned linesearches can be implemented as long as the update direction uy is a descent. direction (that is (/(()) < 0). In particular, these linesearch methods are an important ingredient for the implementation of quasi-newton methods like LBFGS that precondition the gradient with an estimate of the inverse Hessian (Nocedal and Wright, 1999).
Detailed algorithms
We detail below in pseudocode the algorithms based on an estimation of the distance the solution that we consider in our benchmark. D-ADAPT tailored for the ADAM optimizer is presented Algorithm PRODIGY is presented in Algorithm 2.
26

---

Published in Transactions on Machine Learning Research (6/2025)
Algorithm 1: ADAM with D-ADAPT
Inputs:
Initial parameters wa,
Initial estimate of the distance do (default 10 10-"),
Optional sequence of learning rate (to include e.g. additional schedules) (default 1);
First moment EMA parameter 31 (default 0.9)
Second moment EMA paramcter B2 (default 0.999)
Small positive constant to prevent division with zero e (default 10 10"*)
Total number of iterations K
Initialize 8o 0, mo = 0, vo 0, =
for k 0 to K do
Draw mini-batch is with associated objective f(1e
gk E f(w))
m = 81m= + (1 - 31) 31)d2Tb9
uk+1 = 8r8R + +(1 (1- 32)9k
A&+1 diag( /Ux+1 + c)
A A m++1 Sk+1 = / B258% + (1 v dr
Wk+1
rk+1 V Bark + (1 v da (gk: 8k) A
de+1 (1 (1-V82)lakaa
ds+1 max dx+13
Outputs Last iterate wk
Algorithm 2: PRODIGY algorithm
Inputs:
Initial parameters wo:
Initial estimate of the distance do (default 10-9),
Optional sequence of lcarning rate (to include c.g. additional schedules) (default 1),
First moment EMA parameter 3n (default 0.9)
Second moment EMA parameter 32 (default 0.999)
Small positive constant to prevent division with zero (default 10 10-8)
Total number of iterations K
Initialize ro 0, 8n 0, mo 0, to 0
for & 0 to K do
Draw mini-batch is with associated objective f(ix))
E 2 (wrs
ma+1 = 31mm + (1- 11-3)o1 (:)31)40 drgk
uk+1 = B204 + -1- -B
rk+1 / BBrk + (1 32)) (g
(gk; wh wk)
8k+1 / 1828k + +(1 3/
d.+1 FK+1
|skt1li
de+1 = max(de, dx+1)
W'++1 Wt m. mndmm uk+i + der)
Outputs Last iterate WK
27

---

Published in Transactions on Machine Learning Research (6/2025)
Algorithm 3: MECHANIC: A Learning Rate Tuner.
Default settings: 6, B = (0.9, 0.99, 0.999, 0.9999, 0.99999, 0.999999), À = 0.01, Stait 10-8. Input: Base algorithm base, wjsse Rd 8 E [0, 1)", À E R, Siit E IR, 10-8
Init:00 = E R", 0 IR",, wng 0 Rh, Wree whuse, A 0e IRd 0 R
for k = 1,2..... do
gk - Vf(Wk, 2x)
Send gb: to base, receive update ug-
Ak+1 4 us
St.i
9k
W2
hi
9h
Ab
// Note use of AS rather than A+t1-
m& max
Uk P + K
10
TE Bri-1
81-1/ 11k
TK max(0,
W Sinit my Tkk
We
13
81+1
V
orte
14
wk+1 - Whet + (E s141,) A
Algorithm 4: MoMa: Model-based Momentum method.
Hyperparameters: momentum parameter 3 [o, 1), maximum learning rate scheduler an > 0, objective lower bound approximations f
Default settings: ag 1, 8 0.9, f = 0.
Init:/o = 81),,8)),d = (f(w1181),7 (do, w
Input: Initial iterate 1 Rdd, f = IR
for = 1 to K do
fe = (1 - f) sk) + 3
Tk (1 - (V 8k), wk) 3Tk-1
dę (1 VVVB(w 8te) _dx-1
hE f + (dk, wk) Tk
wh+1 = wk min akk
(hx (xx-1;)) |4
dd
28


Reviews:
[{"review_id": "GhacmbnNj2", "reviewer": "Reviewer_9q3A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- The paper provides a thorough and well-executed empirical study of learning-rate-free methods across a diverse set of workloads. The use of the AlgoPerf benchmark ensures a consistent and relevant evaluation framework. The authors' effort to calibrate these methods using evidence-based defaults is a particularly strong aspect of the work, highlighting the importance of well-justified hyperparameter settings.\n- The paper is well-written and organized, making it accessible to readers familiar with the field. The inclusion of supplementary materials, such as detailed training curves and hyperparameter settings, enhances the reproducibility and transparency of the research.\n- The findings of this study have direct relevance to practitioners seeking to simplify the hyperparameter tuning process for deep learning models. The paper's emphasis on the importance of well-calibrated defaults and rigorous benchmarking is particularly valuable.\n\n## Weaknesses\n- While the empirical evaluation is a significant contribution, the paper lacks substantial algorithmic novelty. The focus is primarily on evaluating existing methods rather than introducing fundamentally new optimization techniques. The “evidence-based defaults” approach is, however, a valuable contribution.\n- The sensitivity of the tested algorithms to the training horizon and the reliance on a relative learning rate schedule (warmup/decay) represent a key limitation. While acknowledged by the authors, this dependence undermines the objective of achieving a truly \"learning-rate-free\" solution. Further investigation into mitigating this dependence within the learning-rate-free optimizers themselves is warranted. Perhaps a dynamic horizon determination based on validation loss would improve the results.\n- The observation that learning-rate-free methods did not demonstrate a discernible advantage over AdamW/NadamW, even with comparable tuning budgets, raises concerns. A deeper analysis of the factors contributing to this lack of efficiency is necessary. The authors should explore whether the hyperparameter search space was adequately tailored to the learning-rate-free methods and whether potential interactions between the eliminated learning rate and remaining hyperparameters were properly addressed.\n- While AlgoPerf provides a valuable benchmark, it is important to acknowledge its limitations. The authors should provide a more explicit discussion of the potential generalizability (or lack thereof) of the results to other types of workloads, datasets, and model architectures. The absence of model-scaling experiments is a notable omission.\n- To strengthen the robustness of the findings, the authors should consider incorporating statistical significance tests to support their comparisons between algorithms.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission presents a valuable empirical investigation into the potential of learning-rate-free optimization methods as a step toward truly hyperparameter-free training algorithms. The authors have undertaken a significant amount of experimentation, employing the AlgoPerf benchmark and a novel \"evidence-based calibration\" procedure to evaluate and compare several state-of-the-art algorithms. The results provide important insights into the strengths and limitations of current approaches, with practical implications for both algorithm design and application.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I recommend that the authors address the points raised above in a revised version of the manuscript. Specifically, I encourage them to:\n- Explore techniques for reducing the dependence on the relative learning rate schedules within the learning-rate-free optimizers.\n- Conduct a more detailed analysis of the factors contributing to the observed tuning inefficiency of the learning-rate-free methods.\n- Provide a more thorough discussion of the generalizability of the results to other types of workloads and datasets.\n- Incorporate statistical significance tests to support the reported performance comparisons.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6eZkCtJ3Od", "reviewer": "Reviewer_ies4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n- The motivation is clear and the presentation is good in terms of the flow and details. \n- Detailed explanations of experimental settings (e.g., why training horizons are set) are included.\n- Experiments are extensive and show convincing evidence that we are still far away from truly hyperparameter-free learning algorithms.\n\n## Weakness\n- Limited novelty and insights. Only the performances are compared, and no in-depth ablations are presented to show further insights.\n- Notations are not carefully introduced, e.g., what are $\\alpha$ and $t_{\\text{hor}}$?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper mainly provides an empirical investigation that reviews the previous learning-rate-free methods and tests their performances on the AlgoPerf benchmark with other hyperparameters fixed to default values. It also explores potential improvements of those learning-rate-free methods by calibrating their configurations per workload.\n\nExtensive experimental result shows that:\n\n- There is still a large space for improvement when comparing previous learning-rate-free methods with AdamW/NAdamW baselines\n- When carefully calibrated, some learning-rate-free methods can be greatly improved\n- The best learning-rate-free method after careful AlgoPerf calibration cannot outperform AdamW/NAdamW\n\nThe paper has considered many learning-rate-free methods, including DoG, DoWG, PRODIGY, CoCoB, Mechanic, and MoMo. Among those, the authors have shown that only PRODIGY and Mechanic have the potential to be comparable to baseline methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- If there are some insights on how far the current methods' (e.g., PRODIGY) scheduled or auto-determined learning rates are from the baselines (Adam) and possibly the optimal, it would strengthen the work and may positively influence my opinion on recommendation for acceptance.", "broader_impact_concerns": "This work mainly aims to advance the field of machine learning, which has no obvious concerns and thus does not necessarily need a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PqShV7ApOx", "reviewer": "Reviewer_PfR8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n* This work has proposed an important and interesting question. The generalization abilities of the existing hyperparameter-free methods have not been studied comprehensively.\n\n* The paper is well-organized and easy to follow.\n\n**Weakness:**\n\n* The technical novelty of this paper is poor. All the empirical studies of this work rely on the AlgoPerf benchmark, including the workloads, tuning method, performance metrics, etc. There is no new hyperparameter-free method, evaluation metric, or benchmark workload proposed. The technical difficulty of evaluating the existing learning-rate-free methods using the AlgoPerf benchmark is not clear.\n\n* The significance of the results is limited. I acknowledge this paper's effort to conduct comprehensive tests of the learning-rate-free methods. However, the results seem to be negative and this work didn't provide any insights into the reason for the poor results or any suggestions to improve the existing algorithms.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a pure empirical paper that studies how well existing hyperparameter-free methods can generalize across workloads without workload-specific tuning. To answer this question, this work tested several learning-rate-free optimization methods on the AlgoPerf benchmark. The results show that (a) The tested learning-rate-free methods using default hyperparameters from the paper perform worse than the adaptive optimization baselines (i.e., AdamW and NadamW); (b) The performances of some tested learning-rate-free algorithms can improve by tuning the regularization parameters; and (c) Even the best learning-rate-free methods are not clearly better than the AdamW and NadamW baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The writing of this paper needs to be polished.\n\n* There are some confusing and contradictory claims. For example, on page 2, it is claimed that there are two routes to reducing the hyperparameter search, \"more automatic tuning method\" and \"eliminating hyperparameters\", and this paper is concerned with the latter one. However, at the end of the same paragraph, it is claimed that \"discovering which hyperparameters are most essential to tune can lead to more efficient and robust methods\". Also, exploring more robust tuning methods seems to be the topic throughout this paper instead of \"eliminating hyperparameters\". The difference between these two routes is not clear to me.\n\n* “Bettting”\n\n* In Section 3.3, the meaning of $r(\\tau)$ is not clear. Why is $r$ a function of $\\tau$?\n\n2. As a research paper, it's better if the author(s) can provide some in-depth analyses about the insights of the (negative) results or inspiring suggestions on improving the hyperparameter-free methods. Otherwise, one may not believe that \"the future is bright\" as claimed in the conclusion from the current results shown in the paper. \n\n3. The paper analyzing the AlgoPerf competition results [1] is highly related to this work. It presents some hyperparameter-free methods (e.g., schedule free AdamW) that perform much better than those tested in this work. A fair comparison and discussion could improve the significance of this paper.\n\n4. In Tables 6&9, it seems that all the scores are lower than the baseline scores of the AlgoPerf (see Table 1 of [1]). What's the reason?  \n\n[1] Accelerating neural network training: An analysis of the AlgoPerf competition. ICLR 2025", "broader_impact_concerns": "There is no ethical concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)