Accelerating neural network training: An analysis of the AlgoPerf competition
Authors: Priya Kasimbeg
Frank Schneider
Runa Eschenhagen
Juhan Bae
Chandramouli Shama Sastry
Mark Saroufim
BOYUAN FENG
Less Wright
Edward Z. Yang
Zachary Nado
Sourabh Medapati
Philipp Hennig
Michael Rabbat
George E. Dahl
Venue: ICLR 2025 Poster (2025)
Keywords: Training algorithms
optimizers
benchmark
competition
neural network
training
Primary area: datasets and benchmarks
TL;DR: Analysis of the AlgoPerf competition results, comparing neural network training algorithms

Abstract: The goal of the AlgoPerf: Training Algorithms competition is to evaluate practical speed-ups in neural network training achieved solely by improving the underlying training algorithms. In the external tuning ruleset, submissions must provide workload-agnostic hyperparameter search spaces, while in the self-tuning ruleset they must be completely hyperparameter-free. In both rulesets, submissions are compared on time-to-result across multiple deep learning workloads, training on fixed hardware. This paper presents the inaugural AlgoPerf competition's results, which drew 18 diverse submissions from 10 teams. Our investigation reveals several key findings: (1) The winning submission in the external tuning ruleset, using Distributed Shampoo, demonstrates the effectiveness of non-diagonal preconditioning over popular methods like Adam, even when compared on wall-clock runtime. (2) The winning submission in the self-tuning ruleset, based on the Schedule Free AdamW algorithm, demonstrates a new level of effectiveness for completely hyperparameter-free training algorithms. (3) The top-scoring submissions were surprisingly robust to workload changes. We also discuss the engineering challenges encountered in ensuring a fair comparison between different training algorithms. These results highlight both the significant progress so far, and the considerable room for further improvements.

Published as a conference paper at ICLR 2025
ACCELERATING NEURAL NETWORK TRAINING: AN ANALYSIS OF THE ALGOPERF COMPETITION
Priya Kasimbeg'
Frank Schneider Runa Eschenhagen' Juhan B
Chandramouli Shama Sastry1.6
Mark Saroufim" Boyuan Feng"
Less Wright?
Edward Z. Yang'
Zachary Nado! Sourabh Medapati
Philipp Hennig
Mike Rabbat" George E. Dahl
'Google DeepMind
Universsity of Tübingen UUniversity of Cambridge
*Vector Institute
University of Toronto
"Dalhousie University "Meta
ABSTRACT
The goal of the ALGOPERF:
TRAINING ALGORITTMS competition is to evaluate practical speed-ups neural network training achieved solely by improving the underlying training algorithms. In the external tuning ruleset, submissions must provide workload-agnostee hyperparameter search spaces, while in the self-tuning ruleset they must be completely hyperparameter-Ine -free. In both rulesets, submissions are compared on time-to-result across multiple deep learning workloads, training on fixed hardware. This paper presents the inaugural ALGOPERF competition' 's results, which drew 18 diverse submissions from 10 teams. Our investigation reveals several key findings: (1) The winning submission in the external tuning ruleset, using DISTRIBUTED SHAMPOO, demonstrates the effectiveness of non- diagonal preconditioning over popular methods like ADAM, when compared on wall-clock runtime. (2) The winning submission in the self-tuning ruleset, based on the SCHEDULE FREE ADAMW algorithm, demonstrates new level of effectiveness for completely hyperparameter-fiee training algorithms. (3) The top- scoring submissions were surprisingly robust to workload changes. We also discuss the engineering challenges encountered in ensuring a fair comparison between different training algorithms. These results highlight both the significant progress far, and the considerable room for further improvements.
INTRODUCTION
Deep neural networks are powerful models that excel in tasks such image recognition, natural lan- guage processing, and speech recognition. However, training these models often requires significant computational resources well as careful (and sometimes brittle) training recipes, including meticu- lous hyperparameter tuning. A big practical issue in the usability of training algorithms, for instance, that many choices are still left to the practitioner. How should the learning rate be tuned? In what range? Using what schedule? These are very crucial decisions that can make or break the training process and, critically, determine which methods perform best. Despite training algorithms being such a fundamental part of the deep learning pipeline, the community has been unable to identify which training methods are the state of the art. Previous empirical comparisons have suffered from a number of issues, e.g., weak baselines, not fully accounting for hyperparameter tuning, failing to properly control for potential confounding factors like model architecture changes. Dahl et al. (2023) proposed the ALGOPERF: TRAINING ALGORITHMS benchmark (Section 2) to measure speed-ups in neural net training due to algorithmic improvements, while addressing the aforementioned issues. This competitive, time-to-He to-result benchmark uses multiple realistic deep learning workloads fixed hardware, allowing submitters to innovate solely on the training algorithms. It allows submissions under two different hyperparameter tuning rulesets: external tuning ruleser that scores submissions based on the best result from handful of hyperparameter configurations, and a self-tuming ruleset without hyperparameters that counts any time a submission spends tuning as part of the training time.
Equal contributions. Corresponding author gdahl @ google.com.

---

Published as a conference paper at ICLR 2025
In this paper, we present the results of the inaugural ALGOPERF: TRAINING ALGORIHTMM com- petition, a competition open to the entire machine learning community, based on the ALGOPERF benchmark proposed in Dahl et al. (2023). Our analysis of Re 4000 training runs reveals several key findings on accelerating neural net training through improved training algorithms:
The winning submission in the external tuning ruleset, based on DISTRIBUTED SHAMPOO (Anil et al., 2020; Shi et al., 2023), demonstrates that non-diagonal preconditioning methods can outperform currently popular diagonal methods, such ADAM (Kingma & Ba, 2015), in terms of wall-clock training time. Across eight deep learning workloads, this submission achieved on average =28 faster model training compared to the baseline, which used NADAMW (Dozat, 2016; Loshchilov & Hutter, 2019) (Section 3).
In the self-tuning ruleset, the winning submission based on SCHEDULE FREE ADAM (Defazio et al., 2024) was the only entry surpassing the baseline, providing 88% faster average training. It was also = 10% faster than the external tuning baseline across the seven base benchmark workloads both algorithms trained successfully, without any hyperparameters or parallel tuning (Section 3)." This result establishes a new state-of-the ahrtth for hyperparameter--fre training algorithms and highlights the exciting potential of fully automated neural network training.
The top-scoring submissions are characterized by their consistent performance across workloads, including robustness to minor workload modifications, e.g. changes to activation functions or normalization layers (Table 3). This suggests that, at least in a competitive evaluation context, achieving consistent performance across workloads is major challenge for training algorithms operating under restricted runtime and tuning budgets.
Building the training harness and software for the ALGOPERF: TRAINING ALGORITTMS competition to enable fair and meaningful comparisons between training algorithms, especially across the deep learning frameworks JAX (Bradbury et al., 2018) and PYTORCH (Paszke et al., 2019), required substantial engineering effort. Section 4 discusses these engineering challenges, while Section highlights lessons learned as well as opportunities to improve future iterations of the benchmark.
2 SUMMARY OF BENCHMARKING METHODOLOGY
The ALGOPERF: TRAINING ALGORITIMMS benchmark evaluates the effectiveness of training algorithms by measuring how quickly they can achieve specific evaluation metric goal values across various realistic deep learning workloads. These per-workload, time-to-resule measurements are performed on fixed hardware configuration, and account for all required workload-speeffic tuning. The final benchmark score aggregates across workloads to identify more efficient general-purpose training algorithms for deep learning. Below, we briefly summarize the AL.GOPERF benchmark by explaining key terms. For more detailed discussion of the benchmark's motivation, description, and justification see Dahl et al. (2023) and the competition rules² & documentation", For the ALGOPERF competition, we solicited submissions from the entire machine learning community, and made several modifications to the benchmark as it was initially proposed by Dahl et al. (2023), which are summarized in Appendix A.2.
Workloads. The benchmark features multiple neural network training tasks, called workloads, each consisting of dataset, model, loss function, target metric, validation target and runtime budget. The submissions' objective is to train these workloads as quickly as possible; if the target is not reached within the runtime budget, the run receives an infinite score. Designed to reflect real-world deep learning training scenarios, the workloads cover several key domains. The benchmark includes two types of workloads: fixed base workloads (Table 2) directly affect the benchmark score, and held-out workload variants (Table 3) discourage submissions to overfit the benchmark's 's fixed workloads and ensure robustness to natural workload changes. While held-out workloads do not contribute directly to the benchmark score, failure to train a held-out workload quickly enough will invalidate the submission's 's score for the corresponding fixed base workload.
Submissions. Submitted training algorithms must adhere to the fixed ALGOPERF API (Dahl et al., 2023, Sec. 4.2) and are limited to four submission functions: (1) update_paaaaaee is responsible for
Note, the two speed-ups for SCHEDULE FREE ADAMW are computed across different sets of workloads. iii tuub...ccmm. com/m1 commons/ aaloonnslllllooo gori i cwwic-efee c- f ci cniency/t (ncy/[..... [...]] 1 COMPET I' ON_OON__UUU BO__RRULA AULLS.. md i hub. com/m1 I commons, / gorri thmi e- f of encenyy [...] /DDCCMDMNNAAAAOO ON md
2

---

Published as a conference paper at ICLR 2025
modifying the network's ' parameters during training and typically involves optimization algorithms, such as SGD, ADAM, or a custom method. (2) cicit_optimieeee imiierr...... _er_stateeeeeeees allows the creating of the training algorithm's 's internal state, e.g. to define learning rate schedules. (3) data_select in allows control of how batches are constructed, to use techniques such as curriculum learning or data echoing (Choi al., 2020). (4) get_bat.ch
ze defines hatch size for each workload, e.g.. participants can predetermine the largest batch size fitting in the competition hardware's 's memory. For the external tuning ruleset (see below), participants can also provide a workload-agnostic search space for hyperparameters. This limited API isolates training speed-ups resulting from improvements to the training algorithm itself, rather than pipeline optimizations or model changes. The API also ensures that submissions can be applied to generic deep learning workloads by being fully specified without any workload -spccfffc behavior (apart from the batch size).
Tuning rulesets. Submissions compete under two distinct rulesets governing hyperparameter tuning. The external tuning ruleset simulates hyperparameter tuning with limited parallel resources. this ruleset, hyperparameters are tuned using five independent trials, with hyperparameter configurations sampled via quasirandom search (Bousquet et al., 2017) from the submission's 's defined search space," and are scored based on the runtime of the trial that achieves the validation target the fastest. To produce a more consistent final score, the tuning process is repeated five times across five different tuning studies, where each study receives different random seed for the workload's model and data initialization. The final workload score used in the benchmark's is scoring procedure is the median of the best training time from each of the five studies. The self-tuning ruleset simulates fully automated hyperparameter tuning during training on a single machine. This includes submissions that use the same hy perparaametesss across all workloads (e.g. ADAMW with defaults for all hyperparameters including regularization) or those that perform inner-loop tuning during the training run. Anticipating that they may require more time to reach the target, self- oolf-tuning submissions have three times the runtime budget of external-tunig submissions (see Table 2). Similar to the external tuning ruleset, five studies are conducted (although each self-tuning study consists of a single trial), and the median runtime across all studies determines the workload score.
Benchmark score. A submission's s benchmark score is based on its individual workload scores relative to those of other submissions, aggregated using performance profiles (Dolan & Moré, 2002; Dahl et al., 2023, Sec. 4.5). A performance profile plots the fraction of workloads where a submission trains successfully (achieves the target performance) within a factor of r of the time required by the per-workload fastest submission, for different values of & T Tmax (see Figures and 3). For instance, the height of submission s's performance profile at T = 1.5 is the fraction of workloads where 8 trains successfully and within 1.5x the wall-clock time the best competitor requires on that workload. The final scalar benchmark scores (Figures la and Ic) are the normalized area under the performance profiles, with 1.0 corresponding to a submission that was faster than all competitors on all workloads.
Computational costs. To score all competition submissions, we conducted 3850 and 420 runs in the external tuning and self self-tuning ruleset respectively. On average, scoring an external submission required R 3469 hours, and x 1847 hours for self-tuning submission, totaling x 49,240 hours on the competition hardware (SXNVIDIA V100 GPUs) (see Appendix A.3).
3 RESULTS
The two ALGOPERF: TRAINING ALGORITHMS competition leaderboards (Figure 1) rank submis- sions in each tnning ruleset by their benchmark score. 5 Although five submissions outperformed the baseline (Table 4) in the external tuning ruleset, only the winning submission was competitive in the self-tuning ruleset. The remaining self-tuning submissions (Table 5) scored significantly lower than the self-tuning baseline, highlighting the challenge of fully automatic training algorithms that must pay the training time cost of any workload-specifee hyperparameter tuning. Although there at least one submission that could successfully train each workload, no submission reached the
"Submissions can alternatively provide list of five hyperparameter contigurations that will be sampled without replacement.
'Note that the external-tunin -oernal-tuning baseline is different from the self-tuning baseline; each was tuned and specialized specifically for its ruleset. Similarly, there are two distinct SCHEDULL E FREE ADAMW submissions, one for each ruleset. We specify which we are referring if the context does not make it clear.
3

---

Published as a conference paper at ICLR 2025
Submission
Line
Score
PYTORCH DISTRIBUTED
0.7784
Percentage of Workloads
100%
SCHEDULE ADAMW
75%
GENERALIZED ADAM
0.6383
CYCLIC LR
0.6301
50%
NADAMP
0.5909
BASELINE
0.5707
AMOS
0,4918
25%
CASPR ADAFTIVE
0,4722
LAWA QUEUE
0.3699
0%
LAWA EMA
0.3384
1.0
1.5
2.0
3.0
3.5
4.0
SCHEDULL FREE PRODICY
0
(a) External tuning leaderboard
External tuning performance profiles
Percentage of Workloads
100%
Submission
75%
SCHEDULE FREE ADAMW
0.8542
BASELINE
NADAMW SEQUENTIAL
0.3308
75
SINV6
0.0903
ADAMG
0%
1.0
1.5
2.0
2.5
3.0
3.5
4.0
T
(c) Self-tuning leaderboard
(d) Self-tuning performance profiles
Figure 1: ALGOPERF competition leaderboard & performance profiles for all submissions to the external (top) and self-tuning (bottom) ruleset. The leaderboards (a, c) are ranked by the submissions' benchmark scores, rounded to four significant digits. Higher scores indicate faster training. Note, scores are not comparable between rulesets. In the performance profiles (b, d), each line represents submission. A step at indicates that, for one workload, this submission reaches the target within times the runtime of the fastest submission for that workload and ruleset.
target every workload (Figure 1; see Section 3.1 for more details). This result simultancously demonstrates the benchmark's feasibility and its difficulty as well as the significant potential for future improvement in training algorithms. There were strong submissions in both PYTORCH & JAX, suggesting that, within the benchmark's codebase, workload implementations in both frameworks are, at least to some degree, sufficiently similar in speed and memory usage (see Section 4).
3.1 FER-WORKLLOAD RUNTIMES
Table I provides detailed overview of the submissions' runtimes across all workloads, normalized by the external tuning runtime budget. Workloads with runtimes deemed infinite for scoring purposes marked in gray, with corresponding symbols explaining the reason (see the caption of Table I). The winning submissions in both rulesets excelled by reliably training most workloads rather than being the fastest on every one. For example, PYTORCH DISTRIBUTED SHAMPOO was the fastest on "only" 2 of the 8 base workloads, while SCHEDULE FREE ADAMW led in of 8 in the self-tuning ruleset. We also observed substantial variations in workload runtimes, even among competitive methods. PYTORCH DISTRIBUTED SHAMPOO, for instance, took over twice long WMT as the fastest submission. This indicates that there is still ample potential to improve training algorithms. In the self-tuning ruleset, if the winning submission successfully trained a workload, it did so within the external tuning runtime budget. This suggests that future benchmarks could significantly reduce the self-tune self-tuning runtime budget to computational costs, perhaps matching the external tuning one.
RESNET workload. The RESNET workload is notable, since only one submission, GENERALIZED ADAM, could reliably train it the target performance within the given budget. This may be
4

---

Published as a conference paper at ICLR 2025
Table Normalized submission runtimes across all workloads. All runtimes are normalized using the external tuning runtime budget, with the fastest submission per workload in each ruleset highlighted bold. Workload runtimes considered infinite for scoring are marked in gray, with (suffix) symbol explaining the reason. inf denotes that submission did not reach the workload target within the allowed runtime budget. NaN indicates an error (such as running out memory) before any evaluation occurred. A f indicates that a held-out score is ignored due to the submission not reaching the target on the corresponding base workload, while a & indicates that base workload score is ignored because the submission did not successfully train the associated held-out workload. Runs that are not within 4x the fastest (valid) workload runtime are marked with
(a) External tuning ruleset
CRITED ITB
FASTMRT
RESNET
VIT
CONFORMER
DEEP SPEECH
OGBG
WMT
Base
H.O.
Base
H.O.
Base
H.O.
Base
Base
H.O.
Base
Base
F.O.
Base
H.O.
AMOS
1nf
1nf
0.33
0.49
inf
0.557
0.65
0.71
0.57
0.57
0.60
0.89*
0.68
0.37
BASELINE
0.94
0.08
0.23
0.51
inf
0.94
0.91
0.90
0.83
0.65
0.427
0.68
0.86
0.35
CASPR
0.13
0.15
0.58
1nf
0.59
0.75
0.12
0.12
0.67
ADAPTIVE
CYCLIC LR
0.67
0.08
0.25
0.44
inf
0.81
0.94
0.92
0.70
0.38
0.51
0.49
0.35
GENERALIZED
0.83
0.05
0.18
0.39
0.97
0.88
0.84
inf
0.83
0.68
0.31
0.64
0.63
0.33
ADAM
LAWA EMA
0.69
0.09
0.29
0.57
int
int
0.80
inf
1nt
int
0.57
0.73*
0.89
0.39
LAWA QUBUB
0.14
0.22
0.55
0.66
inf
inf
int
0.25
0.24
0.56
0.22
0.90'
0.43
0.74'
SCHEDULE FREE
0.67
0.05
0.13
0.41
inf
imf
0.57
0.92
0.57
0.78
0.29*
0.61
0.33
0.12
ADAMW
SCHEDULE
0
0.65
-rf
1nf
1nf
inf
1nf
0.40
PRODIGY
PYTORCH
0.65
0.22
0.93
0.43
0.78
0.68
0.62
0.18
0.80
DISTR. SHAMPOO
(b) Self-tuning ruleset
CONFORMER
OGBG
SPEECH
Base
H.O.
Rase
Rase
Rase
ADAMG
inE
ing
inf
inf
inf
ing
inf
ing
inf
BASELINE
0.75
0.22
0.51
int
int
0.46
0.59
NADAMW
2.96
0.57
0.27
0.44
inf
int
1.58
int
1.16
1.45
0.55
0.96
2.36
1.57
SEQUENTIAL
SCHEDULE FREE
0.75
0.25
0.15
0.58
inf
inf
0.68
0.97
0.61
0.88
0.32
0.56
0.94
0.21
ADAMW
SINV6
3B8
0.49
inf
inf
ing
inf
1.82
1,35"
SINV6 75
Mab
NaN
0.45
0.80
1n1
int
inE
inC
2.55'
2.21
1.50"
inc
1.82
0.44
surprising, as the RESNET target threshold was derived using the same procedure all other workloads. The hyperparameter search space used for target-setting (see Dahl et al., 2023) may have been more suitable for this workload given its well-studied nature. Although only one submission achieved a finite RESNET workload score (recall a finite workload score requires at least 3 out of the repetition studies train successfully). several others met the target in least one study, NADAMP, PyTORCH DISTR. SHAMPOO, and the BASELINE NE hit the target least once but were not consistent, with the remaining studies falling just short (Figure 2). These results support using the median across multiple studies determine submission's 's workload score, ensuring robustness against stochasticity. Across all workloads, including held-out ones, at least one submission reliably reached the target, demonstrating that none of the workloads were impossible handle, rather it difficult for submission to do well on all of them simultaneously.
Speed-up comparisons. In addition benchmark scores, compute speed-ups between submissions. For relative training speed-ups, we focus on the fixed base workloads and impute infinite workload scores with the ruleset's runtime budget, i.e. assuming the submission would have
5

---

Published as a conference paper at ICLR 2025
achieved the target just after the (artificial) cut-off. This is best-case assumption for submissions that do not train successfully (and thus a worst-case assumption for speed-ups over those submissions), which may be reasonable in some cases (e.g. Figure 2) but can significantly underestimate actual training times in others. For speed-up calculations, we do not enforce the held-out workload rules that would invalidate workload runtimes when the corresponding held-out workload was not successfully trained. Table 6 lists the average speed-ups over the baseline across all eight base workloads. PYTORCH DISTRIBUTED SHAMPOO provides significantly accelerated training than the external tuning BASELINE, with an average speed-up of > 28%. It was also = 2% and 19% faster compared to the second- and third-place submissions, respectively. In the self-tuning ruleset, SCHEDULE FREE ADAMW provides x 8% faster hyperparameter- free training than the self-tuning baseline. While speed-up metrics offer a more intuitive measure, they are less meaningful overall, as the penalty for missing target is relatively weak. For instance, CASPR ADAPTIVE achieved a finite workload score on only four out of eight workloads, but ranked third in average speed-up.
3.2 COMPARISON ACROSS HYPERPARAMETER TUNING RULESETS
To compare submissions across rulesets, we evaluated the first-place self-tuning submission as an external tuning submission. This quantifies the performance gap between rulesets and estimates the expected slowdown from the lack of parallel hyperparameter tuning. The self-tuning winner, the (self-tuning) SCHEDULE FREE ADAMW submission, would have scored a 0.4804 under the external tuning ruleset, ranking hypothetically eighth. without affecting the scores of other external tuning submissions. Although there is notable gap compared to its external tuning counterpart, is surprisingly competitive in this ruleset, given its lack of parallel tuning. Across the seven base workloads both submissions trained successfully, PYTORCH DISTRIBUTED SHAMPOO, the winner of SCHEDULE FREE ADAMW. However,
the external tuning ruleset, is 24% faster than (self-tuning) SCHEDULE FREE ADAMW is = 10% faster than the (external tuning) BASELINE across the seven base workloads both trained successfully, In other words, (self-tuning) SCHEDULE FREE ADAMW reduced training time by 10% compared to the baseline, using only single machine instead of five in parallel exploring different hyperparameter configurations. Although there still meaningful gap between the rulesets, the encouraging results of (self-tuning) SCHEDULE FREE ADAMW suggest that even without any free soreefid- tuning, might be possible to create training algorithms that are surprisingly effective.
3.3 RULE CHANGE COUNTERFACTUALE
In order to better understand the effects of the specific ALGoPErr benchmark rules, we can consider how the results would have changed with different rules.
Different ways of determining whether submission receives finite runtime score. There are variety of mechanisms in the rules whereby submission can fail to receive finite runtime score on given workload, and they triggered relatively frequently in the competition. tuning ruleset, out of 154 workload-submission combinations (including both fixed base and held-out workloads), 33 (<= 21%))err classified as infs due to a submission not reaching the target (31 out of 84 or x 37% for the self-tuning ruleset), while 5 (= 3%) and 4 (* 5%), respectively, were classified as NaNs (error before first evaluation). Additionally, 8 held-out workload scores (> 12%) in the external tuning ruleset were ignored due to failure reach the target the base workload, while the reverse occurred for 7 base workload scores (s= 8%). In the self-tuning ruleset, 3 held-out scores (* 8%) and 2 base scores (= 4%) were excluded for the same reasons. Finally, 11 scores (* 7%) in the external and 4 scores (s 5%) in the self-tuning ruleset exceeded the tiitiine-tte---fssttttaaii----ie
the external
threshold and were thus classified as infinite. This is most notable for OGBG, where the fast training times of CASPR ADAPTIVE invalidated significant number of workload scores. Ultimately, requiring training algorithms to be robust to workload variations through the held-out workloads did provide a non-trivial constraint, though the held-out workload targets were not so stringent the effect was that large. Only considering submissions to train a workload successfully when no other submission could train more than four times faster, primarily affected workloads where the overall runtime budget was the most generous (in hindsight). This rule might become redundant if runtime budgets for future iterations are tightened based on the top- poop-erforming submissions. In Figure 3 (Appendix A.5), we investigate the sensitivity of benchmark scores and rankings to changes in Tiax the upper limit of the performance profile.

---

Published as a conference paper at ICLR 2025
Scoring subset of workloads. How would the competition results change if we considered only a subset of the benchmark's 's workloads? Figure shows the performance profiles and benchmark scores when ignoring all held-out workloads. With the exception of CASPR ADAPTIVE and AMOS, which switch positions, held-out workloads have little impact on the leaderboard. However, quantifying the full effect of held-out workloads is challenging. Their presence may have discouraged submissions overfitting to the fixed workloads. While held-out workloads offer some value, their costs likely outweigh their benefits (Section 5). These costs include the additional compute and, perhaps more importantly, the significant human effort involved in designing and implementing them.
Dahl et al. (2023) envisioned ALGOPERF's qualification set as a cheaper way to evaluate training algorithms, allowing for better allocation of compute resources to the more promising submissions. For the competition, we had enough compute to fully score all submitted training algorithms and did not use the qualification set (for a description of all changes between the benchmark and competition see Appendix A.2). Figure shows the submissions' score when using only the qualification set (which also excludes held-out workloads). While the qualification set helps to identify under- performing submissions, rankings can change substantially compared to the full set. We can also assess how scores shift when removing individual workloads. Table 7 presents the benchmark scores and rankings when each workload is excluded. The rankings remain largely stable, with the winning submissions unchanged in all but one case, indicating that the competition results robust to the precise selection of workloads, as long as there is a large and diverse enough set.
4 ENGINEERING CHALLENGES
Scoring submissions based on wall-clock time, especially in a multi-framework setting, introduces non-trivial engincering challenges. Requiring competing submissions to make of shared workload implementations and timing code is an essential requirement for meaningful timing measurements, but also introduces host of issues. Even for standard algorithms,, scemingly minor implementation details can have a dramatic effect on time-per-step or how quickly the loss decreases per step. As much as possible, potential confounding factors for training algorithm comparisons should be removed, while still allowing realistic participation. Perhaps the biggest point of tension between controlling measurements and remaining relevant current practice is due supporting multiple frameworks. Submissions in JAX& & PY have to be compared on a level playing field, despite framework- specific differences in optimization and execution paradigms. To ensure as fair a comparison as possible between algorithms implemented in JAX & PYTORCH, the workload implementations in each framework should be functionally equivalent, high-quality, and realistically performant:
Functional equivalence: workload implementations should perform mathematically identical com- putations across frameworks and use identical implementation strategies. Given the same input (batch, model parameters, random initialization), the outputs of the forward and backward passes should be the same (within numerical precision) regardless of whether the workload is implemented in JAX X or PYTORCH. All parts of the computation not controlled by the submission code (everything from forward & backward passes to weight initialization to timing & logging) should be semantically identical and also be parallelized in the same way on the same devices. This guarantees that any observed performance differences are truly due to algorithmic improvements within the submissions, and not variations in the underlying computations.
Performant in both time and memory: workload implementations shouldn't 't saddle training algo- rithm submissions with time or memory overheads that skilled engineers would easily avoid outside the context of the competition. A Although achieving identical execution times (and memory usage patterns) across frameworks when run on identical competition hardware not required in principle (and is nearly impossible), in practice since both JAX & PYTORCH are mature frameworks and the competition workloads are standard deep learning benchmark problems, any large discrepancy between time or space efficiency across workload implementations in different frameworks is cause for concern. Hypothetically, different frameworks can make different techniques easier or harder to express or make achieving certain memory or runtime requirements more or less difficult. How- ever, only when the workload implementations in both frameworks are as efficient as is realistically possible can one begin to ask how the constraints of the frameworks themselves are playing role. Implementations should realistically capture how the frameworks are used by practitioners while also being as efficient possible.
7

---

Published as a conference paper at ICLR 2025
Addressing these challenges led to the identification and subsequent implementation of various improvements and best practices. Our experience should be useful for researchers attempting to make similar reproducible measurements with as few confounding factors as possible.
4.1 FUNCTIONAL EQUIVALENCE JAX & PYTORCH IMPLEMENTATIONS
Sramework-speciffi defaults. JAX& & PYTORCH provide basic primitives for implementing various components in neural networks. Creating functionally equivalent workloads requires considerable care around framework specific features and defaults. For example, JAX's default Ge u activation function calculates the cumulative density function of the Gaussian with Tanh approximation, whereas in PYTORCH the exact computation is used. Similarly, for layer normalization, the default values for ε differ between the frameworks. Even more insidious, the weights of linear layers in JAX are initialized with lecun norma,, while similar weights in PYTORCH are initialized with kaimi _an_inniiiiiiiiiieiiniiiilliiiee forform. Moreover, PYTORCH does not supply a decun_normen initialization, necessitating manual implementation.
Data pre-processing and augmentation. Differences in data pre rre-processing and augmentation strategies result in data pipelines that are comparable. To mitigate these types of differences, the same TENSOR FL.ow (Abadi et al., 2015) dataset pipelines were used across frameworks where possible (CRITEO ITB, FASTMRI, LIBRISPEECH! OGBG, and WMT). However, the IMAGENET VIT and RESNET workloads in PYTORCH use a custom implementation to match the TENSOR random augmentation strategy the JAX workload.
4.2 PERFORMANCE OF JAX & PYTORCH IMPLEMENTATIONS
Data pipelines. Differences in the data parallelism implementations may also lead to additional overhead for PYTORCH workloads. To match JAX's data parallel paradigm, the PYTORCH workloads implemented with Distr www.rrruueeaaaaa. aaaaaaaaalll maraalle llel (DDP). DDP requires PYTHON process per device, which a naive implementation would lead to replication of TENSORFLOW data pipeline tasks in each process and potentially mitigate this issue, TENSOR FL.OW data pipeline operations are only run in one PYTHON process and batches are broadcasted the remaining processes. This choice results in small additional communication overhead for each batch in the PYTORCH workloads.
much RAM and undesirable number of threads. To
Custom CUDA kernels. The wall-clock time of two functionally equivalent computations can significantly differ due to differences in the underlying GPU implementations. One example for this the torch. 1stm implementation.
PYTORCH, the LSTM implementation uses custom CUDA kernel that results in 2x smaller wall-clock time on the LIBRISPEECH DEEPSPEECH workload compared to the JAX version. This uncovered the opportunity to generate similar CUDA kernel in the JAX I.TTM layer implementation. Using the updated faster JAX LSTM layer implementation significantly sped up the end-to-end wall clock times for the DEEPSPEECH workload.
Adopting PYTORCH 2.0 features Achieving realistically performant workload implementations re- quires adopting novel features and best practices both frameworks. Without these, the performance gaps between PYTORCH & JAX could be large 60%. Over the course of the development of the competition, the performance of PYTORCH workload implementations significantly improved as result of migrating from PYTORCH PYTORCH 2.0 (see Appendix A.6).
PYTORCH 2.0 (Ansel et al., 2024), released in Dec 2022, introduced two major extensions that represent major departure from PYTORCH's (Paszke al., 2019) original eager programming model, where every line of code would dispatch to a CUDA kernel. Specifically, PYTORCH 2.0 introduced TorchDyname, a Python level JIT compiler that enables graph compilation of PYTORCII programs, and Inrrctnndctoo Triton (Tillet et al., 2019) kernels for GPU and C++ for CPUs. The main adjustments that had be made to adopt to PYTORCH 2.0 and close the speed gap to JAX were related to torch. comp le:
compiler backend which translates PYTORCH programs into
Avoiding graph breaks. When TorchDynamo encounters unsupported functionality, it creates a graph break, splitting the model. To maximize performance, graph breaks should be prevented and models compiled with torch. comciccopile (model, (aaggaph-Turee
httpps:// thub. http://tt...../ogggo////////////
8

---

Published as a conference paper at ICLR 2025
Compiling the loss function, not just the model. Most of the tutorials surround- ing torch. compile implied that is only to be applied to the model by running torch. compi 1e (model). However, compiling loss functions had dramatic impact performance for several workloads." 7
Overhead reduction mode. For models where overhead reduction is crucial it is rec- ommended to use CUDA graphs. This can be enabled via torch. compi le (model, for the cost of small memory overhead.
Combining DDP and torch. compile: The compiler also needs to compose with the other subsystems, e.g., there are subtle differences between ddp (torch.. compile (model)) ) and torch. compile le (ddp (model) ). With the latter, the compiler will also trace collectives which may result in further performance optimizations.
Memory allocator settings Older CUDA versions are more likely to cause OOM errors in Py- TORCH. Setting torch. cuda memory. __se____________________________ set__allodee weoottoorseetinneee (' expandable__ segments: : True') can fix OOMs caused by memory fragmentation.
Making adjustments to use modern features and adopting best practices for PYTORCH & JAX helped achieve realistically performant workload implementations in both frameworks and significantly reduced the time gap in between the JAX & PYTORCH workload implementations. While perfor- mance differences still exist, they are mostly within tolerance (12%) and are balanced such that neither framework has overwhelming advantage. Measurements of the performances gaps between PYTORCH & JAX workloads and improvements can be found in Appendix A.6.
LESSONS LEARNED
The improved training algorithms developed for the competition delivered significant speedups in neural network training. The winning submissions achieved 28% and 8% faster model training compared to their respective baselines. The winning submission in the external tuning ruleset, based on DISTRIBUTED SHAMPOO, demonstrates that non-diagonal preconditioning methods can improve runtime over currently popular methods like ADAM. And yet, despite these significant strides made in accelerating neural network training, there is ample room left for algorithmic improvement. No single submission dominated across all workloads; instead, five different submissions achieved least one of the eight base workloads.
the best performance
These algorithmic advances can only be realized reliably by careful benchmarking and en- gineering efforts ensuring fair and meaningful comparisons. Comparisons multiple deep learning workloads are required to isolate a robust signal of a submission's 's performance. Seemingly intuitive aggregate metrics, like average speedup, fail to fully capture pertinent aspects of a training algorithm's 's practical usefulness. Precise engineering work is required to ensure that training algo- rithms are compared fairly. In Section 4, we identified implementation details that dramatically affect the runtime across virtually all training algorithms, and numerous potentially confounding factors that must be accounted for in algorithmic comparisons, in particular across deep learning frameworks.
The competition also underscores the inherent link between hyperparameter tuming and training algorithm performance. To meaningfully compare training algorithms, measurements must properly account for workload
hyperparameter tuning and training algorithms must be fully-specified without leaving free parameters for the user to set. complete training algo- rithm specification includes formal specification of training algorithm's 's hyperparameter defaults and/or search space, and should include regularization choices. The varying performance of many submissions across workloads suggests that hyperparameter tuning remains significant challenge; the top-scoring submissions mostly distinguished themselves by their reliable training across large variety of workloads. Despite promising advances in hyperparameetrr-fre algorithms (as in the self-tuning ruleset), fully-automatic neural network training remains a serious challenge. In light of results, future publications of training algorithms should include clear hyperparameter (search space) recommendations and be paired with a recommended tuning protocol, ideally one that is sensitive to a user-supplied tuning budget. Although a radical change from the current practice of published training algorithms that aren't 't runnable without setting various hyperparameters, publishing
/https://git thub. cwm/m1 cowmons/oooocooooooooo.cooooooo. a "https: / / thub. com / commons/ / geri wtw.iic-effiiieeen///eeeeeeeeeee

---

Published as a conference paper at ICLR 2025
families of update rules and abdicating responsibility for tuning entirely to the user only adds to the community's 's confusion on what to actually use.
5.1 METHODOLOGICAL LESSONS
In our competition, the ALGOPERF: TRAINING ALGORITIMMS benchmark has proven to be quite effective in differentiating algorithms and measuring progress in neural network training, but it results in quite an involved experimental protocol with substantial costs. Most of its features, e.g. (integrated) performance profiles, were useful in order to generate nuanced and robust insights. However, based on our experience, we propose the following modifications to the AL.GOPERF benchmark going forward: (I) Removing held-out workloads. Eliminating the held-out workloads would drastically simplify the evaluation process and reduce the benchmark's 's runtime substantially. Though held-out workloads have the potential to deter overfitting to the base workloads, they require substantial computational, logistical (they can't be generated until submissions are frozen), and engineering effort. Replacing the six held-out workloads with one or two additional base workloads would provide similar benefits, while reducing runtime and allowing additional, practically-relevant workloads to be included, such as autoregressive language models diffusion models. (2) Reducing the runtime budgets to reduce costs, especially for the self-tuning ruleset. The submissions have demonstrated that, for many workloads, significantly less training time is needed. Matching the self-tuning budget the current external tuning budget, and potentially reducing the external tuning budget further, would maintain meaningful comparisons while lowering overall costs. (3) Reducing the number of studies from to 3 would cuts compute costs by an additional 40%. These repetitions with different random seeds ensure robust insights rather than random noise due to the stochastic training process. However, our results indicate fewer studies are sufficient for statistical fidelity. Additionally, modernized hardware setup with a better cost-to- toretto-pprfooaecce ratio could further reduce costs.
6 CONCLUSION
An inescapable limitation of empirical comparisons of (training) algorithms (see also Choi et al., 2019; Sivaprasad et al., 2020; Schmidt et al., 2021) is that one can only directly measure the behavior of specific implementations, not the abstract algorithmic ideas they express. The ALGo TRAINING ALGOITHMM competition embraces this constraint by evaluating submissions that fully specify everything necessary to apply them to any generic learning problem, including all necessary workload-specifie hyperparameter tuning. The competition conditions provide realistic simulation of applying a generic training algorithm to new problem, with some important caveats. First and foremost, the maximum runtime budgets and evaluation metric targets for each workload provide a lot of information about what training horizon is appropriate and what validation eиror is achievable, unlike in novel learning task where such information isn't an available. Second, by scoring on time to reach particular validation error goals and not using results on separate per-workload test sets, the competition conditions do not capture the challenges of train/test skew or distribution shift. Additionally, while this competition reveals how well training algorithms perform, deeper analysis, such as Kunstner et al. (2024), is needed to explain why.
Nevertheless, despite its limitations, the ALGOPERF: TRAINING ALGORITHMS competition has produced insights about the current training algorithm landscape. In particular, the competition results have validated two exciting directions for improvements: non-diagonal preconditioning (PYTORCH DISTRIBUTED SHAMPOO) and hyperparameter reduction strategies (SCHeDULE FREE ADAMW). These insights are only possible because they are the result of an open and competitive process. This is in contrast to existing works that, due to the lack of a suitable existing benchmark, have each been forced to introduce their own training algorithm evaluation protocols. As result, much of existing research on training algorithms has focused on isolating and improving individual ideas and components, but hasn't 't always produced complete and usable training algorithms that incorporate viable hyperparameter tuning protocols, as would be necessary to perform well in competition.
Our end goal is not the specific insights generated by the inaugural round of comparisons, as interesting as they might be. Ideally, competitions like ALGOPERF will affect training algorithms research in two main ways. First, they should provide a sieve to filter the most useful ideas out of the literature by separating truly practical methods from interesting ideas that aren't 't yet useful. Second, they should provide a valuable signal to guide the design process for new training algorithms.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
FS was supported by funds from the Cyber Valley Research Fund. PH and FS gratefully acknowledge co-funding by the European Union (ERC, ANUBIS, 101 123955), and by the DFG through Project HE 7114/5-1 SPP2298/1: PH member of the Machine Learning Cluster of Excellence, funded by the DFG under Germany's s Excellence Strategy - EXC number 2064/1 Project number 390727645; PH and FS also acknowledge the German Federal Ministry of Education and Research (BMBF) through the Tübingen Al Center (FKZ:01IS) (1K2:))1S180339)) and funds from the Ministry of Science, Research and Arts of the State Baden- Württemberg.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: http://wttnssseee twnsorrfrrrooooo ow org/. Software available from tensorflow.org.
Olah,
Rohan Amil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable Second Order Optimization for Deep Learning. 2020. URL https: aariiviiiiv.///aa//aaaaaaaaaavva org/abss/s ab//bbs//200 2002. 09018. 8.
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary Vito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, K. Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Shunting Zhang, Michael Suo, Phil Tillet, Xu Zhao, Eikan Wang, Keren Zhou, Richard Zou, Xiaodong Wang. Ajit Mathews, William Wen, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In Proceedings of the 29th ACM International Conference Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS '24, pp. 929-947, New York, NY, USA, 2024.
Olivier Bousquet. Sylvain Gelly, Karol Kurach, Olivier Teytaud, and Damien Vincent. Critical Hyper- Parameters: No Random. No Cry, 2017. URL htttt:///axxii org/abs///// 03200.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, and Skye Wanderman-Milne, JAX: composable transformations of Python+NumPy programs, 2018. URL http: / thub. wtt.ccom///ooooooo e/ jax.
Dami Choi, Christopher Shallue, Zachary Nado, Jaehoon Lee, Chris J. Maddison, and George Dahl. On Empirical Comparisons of Optimizers for Deep Learning, 2019. URL https:/ / arxiv. org/ abs /wt.//91..... 05446.
Dami Choi, Alexandre Passos, Christopher J. Shallue, and George E. Dahl. Faster Neural Network Training with Data Echoing, 2020. URL https: /awxiv.iiivv/vv/////i//aaa/aaiia org. abs/ 05550.
George E. Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, Juhan Bae, Justin Gilmer, Abel L. Peirson, Bilal Khan, Rohan Anil, Mike Rabbat, Shankar Krishnan, Daniel Snider, Ehsan Amid, Kongtao Chen. Chris Maddison, Rakshith Vasudev, Michal Badura, Ankush Garg, and Peter Mattson. Benchmarking Neural Network Training Algorithms, 2023. URL https: //tti//..../////////e//eee/aaav wro.oo////aoooo abs 6.066.3.....0 07179.
Aaron Defazio, Xingyu Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The Road Less Scheduled, 2024. URL https:/ http://awvvvv..ogaaa//a /245///b./25...... 2405. 1682.
11

---

Published as a conference paper at ICLR 2025
Elizabeth D. Dolan and Jorge J. Moré. Benchmarking optimization software with performance profiles. Mathematical Programming, 2002.
Timothy Dozat. Incorporating Nesterov Momentum into Adam. In 4th International Conference on Learning Representations, ICLR, 2016.
Sai Surya Duvvuri, Fnu Devvrit, Rohan Anil, Cho-Jui Hsieh, and Inderjit S. Dhillon. Combining Axes Preconditioners through Kronecker Approximation for Deep Learning. In /2th International Conference on Learning Representations, ICLR, 2024.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging Weights Leads to Wider Optima and Better Generalization. In Ricardo Silva, Amir Globerson, and Amir Globerson (eds.), 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp. 876-885. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.
Jean Kaddour. Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging. In Has it Trained Yet? NeurIPS 2022 Workshop, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 3rd Interna- tional Conference on Learning Representations, ICLR, 2015.
Frederik Kunstner, Robin Yadav, Alan Milligan, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why Adam outperforms gradient descent language models. In NeurlPS 2024 Workshop on Mathematics of Modern Machine Learning, 2024. URL http:: //opnreveee......... net/ forum? ? d
Iya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts, In 5th International Conference on Learning Representations, ICLR, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR, 2019.
Konstantin Mishchenko and Aaron Defazio. Prodigy: An Expeditiously Adaptive Parameter-Free Learner. In 41 International Conference on Machine Learning, ICMI., Proceedings of Machine Learning Research. PMLR, 21-27 Jul 2024.
Yijiang Pang, Shuyang Yu, Bao Hoang, and Jiayu Zhou Towards Stability Parameter-free Optimization, 2024. URL wwtpthttp//// /ttpp::://aaxi....////
04376.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, NeurlPS, 2019.
Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Mathemati- cal Statistics, 1951.
Robin M. Schmidt, Frank Schneider, and Philipp Hennig. Descending through Crowded Valley- Benchmarking Deep Learning Optimizers. In 38th International Conference on Machine Learning, ICML, 2021.
Hao-Jun Michael Shi, Tsung-Hsien Lee, Shintaro Iwasaki, Jose Gallego-Posada, Zhijing Li, Kaushik Rangadurai, Dheevatsa Mudigere, and Michael Rabbat. A Distributed Data- Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale, 2023. URL https: : /tap::::///raaee /ttwxiiivvv//////////aa////aaava //tpa//i..../////////..../// 06497.
Prabhu T. Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, and Francois Fleuret. Optimizer Benchmarking Needs to Account for Hyperparameter Tuning. In 37th International Conference on Machine Learning, ICML, 2020.
Leslie N. Smith. Cyclical Learning Rates for Training Neural Networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464 464-472,77 2017.
12

---

Published as a conference paper at ICLR 2025
Ran Tian and Ankur P. Parikh. Amos: An Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scale, 2022. URL https: http:://wwww...o..e.o.eeoo wttpg/gaa////ao../////////// http://...rr/////22.111333
Philippe Tillet, H. T. Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp. 10-19, New York, NY, USA, 2019. Association for Computing Machinery.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ALGOPERF DETAILS
In this section, we provide additional details about the ALGOPERF: TRAINING ALGORITHMS benchmark (Dahl et al., 2023) and our competition based on it. Tables 2 and summarize the fixed base and held-out workloads the ALGOPERF benchmark as they are used in the competition.
Table 2: Summary fixed base workloads in the ALGOPERF benchmark. Losses include cross- entropy (CE), mean absolute error (L.1), and Connectionist Temporal Classification loss (CTC). Additional evaluation metrics are structural similarity index measure (SSIM), (word) error rate (ER WER), mean average precision (mAP), and bilingual evaluation understudy score (BLEU). Note: Some workloads have minor changes (see Appendix A.2) to the runtime budgets and validation targets compared the ALGOPERF benchmark publication (Dahl et al., 2023). The runtime budget is that of the external tuning ruleset, the self-tuning ruleset allows 3x longer training.
Validation Target
Runtime Budget
Task
Dataset
Model
Loss
Metric
Clickthrough rate prediction
CRITEO 1TB
DIRMSMALL
CE
CE
O. 0.123335
7703
MRI reconstruction
FASTMRI
U-NET
LI
SSIM
0.7344
8859
Image
IMAGENET
RESNET-50
CE
ER
0,22569
63,008
classification
VIT
CE
ER
0.22691
77,520
Speech recognition
LIBRISPEECH
CONFORMER DEEPSPEECH
CTC
WER
0,085884
61,068
CTC
WER
0.119936
55,506
Molecular property
OGBG
GNN
CE
MAP
0.28098
18,477
prediction
Translation
WMT
TRANSFORMER
CE
BLEU
30.8491
48,151
Table 3: Summary of held-out workloads sampled for this iteration of the ALGOPERF competi- tion. One held-out workload was sampled from random workload variants (see (Dahl et al., 2023, Table 11)) for each dataset used in the fixed workloads. Note, for the IMAGENET and LIBRISPEECI datasets only a single held-out workload was sampled, according to the modified competition rules (see Appendix A.2). The loss function and performance metrics are identical to the corresponding fixed base workload (Table 2).
Validation Target
Runtime Budget
Task
Dataset
Variant
Clickthrough rate prediction
CRITEO 1TB
EMBED INIT SCALE
0. 0.129657
7703
MRT reconstruction
FASTMRI
TANH
0.717840
8859
Image classification
IMAGENET
RESNET BN INIT SCALE
0.23474
63,008
Speech recognition
LIBRISPEECH
CONFORMER LAYERNORM
0.09731
61,068
Molccular property prediction
OGBG
ALTERED LAYERS
0.269446
18,477
Translation
WMT
GLU & TANH
29.5779
48,151
A.2 MODIFICATIONS TO THE ALGOPERF BENCHMARK FOR THIS COMPETITION
For the purposes of this competition, we made several modifications to the benchmark from how it was initially proposed by Dahl et al. (2023), mainly to reduce the competition's 's overall compute costs. All modifications were done before the competition's submission deadline. Most notably, the number of tuning trials in the extemnl tuning ruleset reduced from 20 to 5. This change significantly lowers the computational cost the competition while also mitigating the risk of submissions overfitting the benchmark's workload selection. To further reduce computational
14

---

Published as a conference paper at ICLR 2025
costs, also decreased the runtime budgets for both LIBRISPEECH workloads significantly from 101, to 61, 068 seconds and from 92, 509 to 55, 506 seconds with only a slight impact on target performance (the WER changed from 0.078477 to 0.085884 and from 0.1162 to 0.119936). Additionally, small bug fixes led to slight changes in two other workload targets: the target loss on CRITEO ITB increased from 0. 0.123649 to 0.123735, and the target SSIM FASTMRI decreased from 0.7344 to 0.723653. Furthermore, only a single held-out workload was sampled for each dataset, not each workload. This means that for both the IMAGENET and LIBRISPEECH datasets, only one, instead of two, held-out workloads were sampled from the combined set of workload variants, further reducing the competition's 's overall runtime significantly.
Finally, we decided to use only the validation targets, excluding the test set targets. Including both validation and test set targets, as proposed by Dahl et al. (2023), complicates the evaluation process by involving the practitioner's task of selecting hyperparameters for an unknown test set based on validation performance. We chose to omit this aspect of the training process in the current iteration of the benchmark.
A.3 COMPUTATIONAL COSTS
The AL.GOPERF benchmark requires a substantial number of training runs and thus significant computational resources to yield meaningful results. To evaluate a single submission, it needs be evaluated on eight fixed-and six held-out-workloaade are performed, and in the external tuning ruleset, each study comprises five tuning trials. In this iteration of the competition, we conducted 3850 runs in the external tuning ruleset (due to eleven training algorithms evaluated on 14 workloads, with 5-5 trials) and 420 runs in the self-tuning ruleset (six training algorithms evaluated five times on 14 workloads, with a runtime budget 3x larger). The actual computational cost of these 4000+ training runs varies significantly based on the performance of the submission. Training can terminate early if the validation (and test targets) are met or if errors occur, such as out-of-memory (OOM) issues. The runtime budget in Table reflects only the compute time spent by the submissions themselves, excluding time for free evaluations other operations outside the submission functions. It is thus not an upper bound on the total computational cost. On average, we required approximately 1847 h to fully run self-tuning submission and 3469 h per external tuning submission, amounting to a total of roughly 49,240 h on the competition hardware NVIII V100 GPUs). This is considerably less than what is reported in Dahl et al. (2023, Table 15), primarily due to fewer tuning trials and a reduced runtime budget for certain workloads (see Appendix A.2), as well as early termination of some submissions. In theory, the number of required runs could be reduced, for example, by skipping a held-out workload if a submission fails the corresponding fixed workload. However, this would limit the ability to parallelize different runs and reduce opportunities for further analysis beyond calculating benchmark scores.
For each workload, five repetition studies
A.4 SUBMISSION DETAILS
In this section, we provide details on the submissions received in this iteration of the competition. Table 4 lists the submissions the external tuning ruleset, while Table 5 details submissions received for the self-tuning ruleset. Notably, no submission in this iteration significantly modified the data_- selection function, indicating potential area for future exploration.
A.5 ADDITIONAL COMPETITION RESULTS
In this section, we provide additional analysis of the ALGOPERF competition results. In Figure 2, we investigate submissions could achieve the target performance on the RESNET workload at least once, but not reliable enough to receive a finite score. Table 6 compares the training time speed-ups of all submission relative to the BASELINE in their respective ruleset. We investigated the sensitivity of leaderboard rankings and benchmark scores to changes in Tmax: the upper limit of the performance profile and integration for the benchmark score. As shown in Figure 3, rankings remain relatively stable for most submissions across different Tmax values. With Figures 4 and 5 and Table 7 we explore how hypothetical rules changes would affect the competition results.
A Ithoug decided not to use test targets for the purpose of our competition, wanted to log when submissions reached the test targets for potential future analysis.
15

---

Published as a conference paper at ICLR 2025
Table 4: External tuning submissions. Details of all submissions to the external tuning ruleset.
Submission
Authors
Institutions Framework
Description
PYTORCH
Hao-Jun Shi, Tsung-Hsien Lee. Anna Cai, Shintaro Iwasaki, Wenyin Fu, Yuchen Hao, Mike Rabbat
Meta Platforms
PYTORCH
Based on the Distributed Shampoo algorithm of Anil al. (2020) with implementation tailored to leverage PyTorch performance optimizations. See Shi et al. (2023) for details. The submission uses list of five hyperparameter settings.
DISTR.
SUBMISSION
SCHEDULE
Alice Yang, Aaron
Meta Samsung AI
PyToRCH
externally tuned version of SCHEDULLE FREE ADAMW (Defazio et al., 2024) with list of five hyperparameter contigurations.
FREE
Defazio, Konstantin
ADAMW
Mishchenko
GENERAL
George Dahl,
Google
JAX
Submission with an ADAM-style update rule, tuning over the of Nesterov acceleration and preconditioning. Essentially tuning over ADAMW (Kingma & Ba, 2015), NADAM and SGD (Robhins & Monro, 1951) with or without momentum.
IZED
Sourabh Medapati,
ADAM
Zack Nado, Rohan
Anil, Shankar Krishnan, Naman
Agarwal, Priya
Kasimbeg, Vlad
Feinberg
CYCLIC LR
Niccolò Ajroldi,
MPI-IS, ELLIS
PYTORCH
Revisits the work of Loshchilov Hutter (2017) and Smith (2017), coupling
Antonio Orvieto,
Jonas Geiping
Institute
NADAMW (Dozat, 2016; Loshchilov & Hutter, 2019) with cyclic learning rate scheduler. Each cycle involves linear warmup phase for the LR. followed by cosine annealing.
Tübingen
NADAMP
George Dahl,
Google
JAX
Uses NADAMW with extra tunable hyperparameter p enabling pth of denominator inside NADAMW update rule instead of the default of 2.
Sourabh Medapati,
Zack Nado, Rohan
Anil, Shankar
Krishnan, Naman
Agarwal, Priya
Kasimbeg. Vlad
Feinherg
BASELINE
JAX
Baseline using NADAMW (Dozat, 2016; Loshchilov & Hutter, 2019) and linear learning rate warmup followed by cosine decay (Dahl al., 2023).
AMOS
Ran Tian
Google
JAX
Submission based on the AMOS optimizer (Tian & Parikh, 2022) with list of five hyperparameter settings.
CASPR
Sai Surya Duvvuri,
UT Austin, UCLA,
JAX
submission based on (Duvvuri al., 2024) with a list five hyperparameter configurations.
ADAPTIVE
Inderjit Dhillon,
Cho-Jui Hsieh
Google
LAWA QUEUE
Niccold Ajroldi,
MPI-IS, ELLIS Institute
PYTORCH
Employs Latest Weight Averaging (lzmailov al., 2018: Kaddour, 2022) top of NADAMW (Dozat, 2016; Loshchilov Hutter, 2019), maintaining a queue of previous model weights. The queue is periodically updated during training and passed to the competition API for evaluation.
Antonio Orvicto,
Jonas Geiping
Tübingen
LAWA EMA
Niccolò Ajroldi, Antonio Orvicto, Jonas Geiping
MPI-IS, ELLIS Institute
PYTORCH
Similar to LAWA QUEUE maintaining an exponential moving average of the model weights, which is updated periodically during training and returned to the competition API for evaluation.
Tübingen
SCHEDULE
Alice Yang. Aaron Defazio, Konstantin Mishchenko
Meta Samsung
PYTORCH
Combining Schedule-free (Defazio al.. 2024) with the PRODIGY optimizer (Mishchenko & Defazio, 2024).
FREE
PRODIGY
Al
16

---

indicated with Published as a conference paper at ICLR 2025
Snady Thiil
Study 1, Thal
Study Trial
Accuracy
Snudy Trial
Study Thial
0.73
50000
52000
(a) BASELINE
Accuracy
Snady 0, Trial
Study 1, Trial 2
Study 2, Trial 2
Study 3, Trial 2
Study 4, Trial 2
30000
32000
60000
62000
(b) NADAMP
0.77
Snady Tmil
Study Trial
Study Trial 5
Accuracy
0.76
Snudy Trial
Study Trial
0.74
0.73
50000
52000
54000
56000
58000
60000
62000
Runtime in seconds
(c) PYTORCH DISTRIBUTED SHAMPOO
Figure 2: Validation accuracy vs. runtime on the RESNET workload. The BASELINE (Figure 2a), NADAMP (Figure 2b), and PYTORCH DISTRIBUTED SHAMPOO (Figure 2c) all reach the validation target on the RESNET workload for at least one study but not reliably enough to get a finite score. Shown the best trials from each of the five studies, where "best" is either the fastest trial to achieve the target performance or the trial whose best performance closest to the target. Trials that reach the target marked with solid line, while studies that do not reach the target dashed line. The gray dashed horizontal and vertical lines indicate the target performance and runtime budget respectively. Additionally, both AMOS and CYCLIC LR came close but missed the target in all studies.
17

---

Published as a conference paper at ICLR 2025
0.8
0.6
Benchmark Score
0.4
0.2
0.0
2.0
2.5
3.0
4.0
Time
PyTorh Distr. Shampoo
NadamP
Lawa Queue
Schedule Free AdamW
Baseline
Lawa EMA
Generalized Adam
Amos
Schedule Free Prodigy
Cyclic I.R
CASPR Adaptive
(a) External tuning ruleset
0.8
Benehmark Score
0.1
0.2
2.0
2.5
3.0
4.0
Tiaax
Schedule Free Adam
NadamW Sequential
Sinv6
Baseline
Sinv6 75
AdamG
(b) Self-tuning ruleset
Figure 3: Benchmark score as function of Tmax- The upper limit of the performance profile and upper integration limit for the benchmark score, Tmax, determines which workload scores are treated as finite and influences the penalty for infinite scores. We observe that rankings remain stable for most submissions across different values of Timax
18

---

Published as a conference paper at ICLR 2025
Py Torch Distr. Shampon (0.7784)
Schodule Free AdamW (0.7740)
Porcestigge WWorkoooss
Generalized Adam (0.69444)
Cyclie L (0.6648)
Nadami (0.6067)
50%
Baseling (0.5915)
CASPR Aduptive (0:5549)
259
(0.4918)
Queae
Lawn EMA (0.3384)
Schodule Free Prodigy (0.0966)
(a) Performance profiles for the external tuning rulesct ignoring held-out workloads
100%
Purcentage Worklends
Schooulle Adami (0.8542)
Haseline
NadamW Sequental (0.3826)
Sinv6 (0.1420)
Sinv6 (0.05I33
AdamG (0.0000)
08
1.0
1.5
2.0
2.5
3.0
3.5
400
(b) Performance profiles for the self tufitunnng ruleset ignoring held-out workloads
Figure 4: Performance profiles of all ALGOPERE submissions when ignoring held-out work- loads. Structurally the same as Figure but here we ignore all benchmark rules involving the held-out workloads.
19

---

Published as a conference paper at ICLR 2025
Schedule Free Adam
Torch Distr. Shampoo (0.7845)
Cyclic LR (0,7054)
69%
Generalized Adam (0.6886)
Percentge of
CASPR Adaptive (0.5539)
Nadomil (0.5269)
33%6
Baseline (0.5000)
Lawa EMA (0:4747)
Lawa Queae (0.4680)
(0.2172)
1.0
1.5
2.0
2.5
3.0
3.5
Schodule Free Prodigy (0.0000)
(a) Performance profiles for the external tuning ruleset on the qualification workloads
Schoollle Free Adkm' (0.9848)
Baseline (0.9529)
NadamW Sequential
(0.2037)
33%
Sinv6 1364)
AdamGi (0.0000)
09
(b) Performance profiles for the self-tuning ruleset on the qualification workloads
Figure 5: Performance profiles of all ALGOPErF submissions on the qualification workloads. Structurally the same as Figure and Figure but only considering the three workloads that are part of the qualification set, i.c. CRITEO ITB, WMT, and OGBG. In the qualification set, no held-out workloads are used.
20

---

Published as a conference paper at ICLR 2025
Table S: Self-tuning submissions. Details of all submissions to the external tuning ruleset.
Submission
Authors
Institutions
Framework Description
SCHEDULE
Alice Yang, Aaron
Meta AL,
PYTORCH
A self-tuning version of SCHEDULE FREE ADAMW (Defazio al., 2024) using single hyperparameter configuration.
FREE
Defazio, Konstantin
Samsung AI
ADAMW
Mishchenko
BASELINE
JAX
Baseline using NADAMW, linear learning rate warmup followed by cosine decay, and a single hyperparameter point (Dahl et al., 2023).
NADAMW
George Dahl,
Google
JAX
Uses NADAMW update rule runs fixed hyperparameter points sequentially. The intention was for these to be the top hyperparameter points found the self-tuning rulesct step budgets.
SEQUENTIAL
Sourabh Medapati,
Zack Nado, Rohan
Anil, Shankar
third
Krishnan, Naman
Agarwal, Priya
Kasimbeg, Vlad
Feinberg
SINV6 75
Abhinav Moudgil
Mila,
JAX
A submission for task-invarian learned optimizer meta-trained small tasks. Uses 75% of number of steps as target in Icarned optimizer initialization.
Concordia
University
SINV6
Abhinav Moudgil
Mila,
JAX
A submission for a dusk-invariant learned optimizer meta-trained small tasks.
Concordia
University
ADAMG
Yijiang Pang
Michigan
PYTORCH
submission based on the ADAMG optimizer (Pang et al., 2024).
State
University
Table 6: Speed-ups vs. the baseline. To compute the speed-up over the baseline, we first compute the geometrie mean of the workload runtimes relative the runtimes the baseline. Only fixed base workloads are considered. If submission did not reach the target workload, its runtime is imputed with the runtime budget, i.e. assuming that the submission would have reached the target just after the cut-off. This is the best-case assumption for the submissions. We do not set runtimes to infinity, e.g. because the corresponding held-out workload was not trained successfully. The geometric mean is then expressed as relative speed-up over the baseline, with positive numbers representing faster training.
(a) External tuning
(b) Self-tuning
Submission
Speed-up
Submission
Speed-up
PYTORCH DISTR. SHAMPOO
27.87%
SCHEDULE FREE ADAMW
7.76%
SCHEDULE FREE ADAMW
26.60%
BASELINE
0.00%
CASPR ADAPTIVE
24.33%
NADAMW SEQUENTIAL
-92.44%
GENERALIZED ADAM
10.89%
SINV6 75
-157.67%
CYCLIC LR
10.67%
SINV6
-168.63%
LAWA QUEUE
7.98%
ADAMG
-294.1699
NADAMP
3.37%
AMOS
1.46%
BASELINE
0.00%
LAWA EMA
-9.17%
SCHEDULE FREE PRODIGY
-15.68%
21

---

Published as a conference paper at ICLR 2025
Table 7: Submission benchmark scores and ranking when removing individual workloads. Shown are the benchmark scores (S.) and leaderboard rankings (R.) of all external tuning (Table 7a) and self-tuning (Table 7b) submissions, when dropping specific workloads. For example, the last column reports the rankings of the submissions if we consider all workloads (including held-out workloads) except the WMT workloads. The "Full" columns show the scores and ranks when considering all workloads.
(a) External tuning ruleset
Full
CRITBO 1TB
FASTMRI
RISNET
VIT
CON- FORMER
DEEP SPEECH
OGBG
WMT
Score
Rank
R.
S.
S.
R
S.
S.
R.
PYTORCH DISTR.
0.78
0.75
0.75
1
0.89
0.75
0.75
0.75
0.77
0.81
SHAMPOO
SCHEDULE FREE
0.71
2
0.67
0.67
2
0.81
2
0.68
0.68
0.68
2
0.81
0.67
GENERALIZED
0.64
0.61
4
0.59
6
0.63
0.73
0.59
0.73
ADAM
NADAMP
0.59
0.54
0.57
5
0.58
0.55
0.53
5
0.68
BASELINE
0.57
0.53
0.55
6
0.65
5
0.56
0.52
0.52
6
0.65
0,58
AMOS
0.49
0.56
0.50
7
0.56
7
0.44
0,42
0.42
8
0.56
0.47
CASPR
0.47
0.40
8
0.54
0.41
0.41
0.40
ADAPTIVE
LAWA QUEUE
0.37
0.42
0.32
9
0.42
0.31
0.42
0.42
7
0.33
10
0.31
10
LAWA EMA
0.34
0.25
0.31
0.39
0.28
10
0.39
10
0.39
0.39
0.32
SCHEDULE
PRODIGY
(b) Self-tuning ruleset
Full
CRITBO ITB
FASTMRI
RESNET
CON-
DEEP SPEECH
OGBG
WMT
FORMER
Score
Rank
R.
S.
R.
R.
S.
S.
R
S.
R.
R.
S.
SCHEDULE FREE
0.85
0.83
0.98
0.83
0.85
0.84
ADAMW
NADAMW
0.33
0.38
0.27
3
0.38
0.30
0.38
0.29
3
0.27
0.38
SEQUENTIAL
SINV6
0.14
0.12
0.08
SINV6
0.09
0.07
0.10
0.10
0.09
5
0.10
0.04
ADAMG
0.00
0.00
0.00
6
0.00
6
0.00
0.00
0.00
6
0.00
6
0.00
6
22

---

Published as a conference paper at ICLR 2025
A.6 ALGOPERF WORKLOAD WALL-CLOCK TIME COMPARISON BETWEEN JAX & PYTORCH
To measure the wall-clock time performance of JAX & PYTORCH workload implementations, we estimate the submission times for ADAMW baseline training algorithm to of the workloads on the 8x NVIDIA V100 GPUs competition system. The workloads support the same training batch sizes for this ADAMW baseline across frameworks.
train to target for each
The submission time accumulates the wall-clock times of the update_ parans and the data__ select ion calls and excludes any time spent logging and checkpointing (which disabled during scoring runs anyway). During the first few steps of training, there may be some additional overhead in these calls resulting from cache warm-ups and compilation costs of the model and update code. An accurate estimate would be based enough steps that any especially slow initial steps play only small role in the average step time. We found that running for 20% of the step hint allowed to estimate the equilibrium step time well. We then calculated the full submission time by extrapolating the submission time over the 20% step hint to the full step hint.
We initially performed this measurement when the workloads were complete in the sense that they were functionally equivalent. After upgrading the JAX PYTORCH packages, changing the CUDA driver versions for the hardware configuration, and implementing various improvements and best practices, we then repeated the measurement to capture the final state of the workloads performance across frameworks. The projected submission times of the JAX & PYTORCH workload presented in Table 8.
Table 8: Projected submission times for ADAMW baseline JAX & PYTORCH workloads before and after workload performance adjustments. All submission times are in minutes. pos- itive difference indicates that JAX implementation is faster than our PYTORCH implementation.
Workload
JAX
PYTORCH
Difference
Before
After
Before
After
Before
After
CRITEO ITB
127
136
213
122
68%
-10%
FASTMRI
148
145
163
163
10%
12%
RESNET
1047
1063
1174
1135
12%
7%
VIT
1290
1378
1260
1253
-2%
-9%
CONFORMER
1689
1445
1755
1407
4%
-3%
DEEPSPEECH
1047
875
1229
967
17%
10%
OGBG
306
399
398
445
30%
12%
WMT
804
782
972
792
21%
1%
23


Reviews:
[{"review_id": "hvDQdUE1mf", "reviewer": "Reviewer_3MCC", "summary": "This paper presents an analysis of the results of the recent AlgoPerf Training benchmark, in which a variety of community-submitted algorithms were evaluated on multiple workloads and in multiple settings to identify those which yield improved training algorithms. A variety of details from the benchmark results are presented, leading to some broad trends (e.g., the best optimizers are those that are \"consistently reasonable\" as no one approach dominated all workloads) as well as suggestions for future directions. The paper also includes lessons learned and commentary on the benchmark itself, and on the engineering efforts involved.", "questions": "n/a", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper summarizes and analyzes the results of the AlgoPerf Training benchmark, providing a valuable focal point to the community for driving future progress in training algorithms and setting the agenda for research. The current advances and limitations of training algorithms are highlighted, helping to clearly identify areas for improvement in the community.\n\nEqually valuable, the paper has a detailed discussion of lessons learned and suggestions from the process of running the competition. These are details that are often not widely disseminated, and are valuable for others seeking to build similar benchmarks. This includes a discussion of engineering challenges involved in ensuring fair and reasonable comparisons across submissions and frameworks.\n\nOverall the paper is clear, well-written, and likely to help drive progress in the ML community.", "weaknesses": "I have no notable concerns about the paper.\n\nVery minor typo: L382, \"framekworks\" -> \"frameworks\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6k9WexolE2", "reviewer": "Reviewer_Ed5X", "summary": "This paper details the experience of the \"AlgoPerf Competition: Training Algorithms\". The goal of the competition is to evaluate neural network training speeds by improving the training algorithms. The competition evaluated submissions under two rulesets: external tuning (using predefined hyperparameter search spaces) and self-tuning (completely hyperparameter-free). The competition also demonstrated that the top-scoring algorithms generalized across workloads. For the former, distributed shampoo outperformed other techniques, and for the latter, Schedule Free AdamW demonstrated superior performance.\n\nThe paper also describes future training algorithm developments -- emphasizing the importance of fair benchmarking, providing complete algorithm specifications, and different hyperparameter tuning strategies. The paper is written like an experience paper, demonstrating methods and techniques that help with neural network speedups, as well as conducting a fair evaluation of different methods.", "questions": "1) Please describe a fit with ICLR, and how publishing this paper helps the broader ICLR community.\n2) Can you please provide more insights into the specific reasons behind the significant compute costs, and are there suggestions for optimizing the evaluation process without compromising the robustness of the benchmark?\n3) Can you comment on increase in compute costs if test sets and LLMs are considered for model evaluation?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The papers' winners (Dist. Shampoo, and Adam W) are interesting to note, and offer strong baselines for the workloads used in the paper.\n- The paper describes engineering effort needed to bring parity between Jax and Pytorch, which can be useful in understanding   accuracy/performance differences between the two frameworks related to specific features/API calls that were used in the competition.\n- The paper details the engineering and compute needed in hosting a systematic model evaluation framework/process.\n- The paper is well-written, and describes the methodology, results and lessons clearly.", "weaknesses": "- Weak conclusions:  The authors are encouraged to draw stronger conclusions from the experience. While it is acknowledged that these types of papers are difficult to write, the broad applicability or lessons can be difficult to grasp for the reviewer. The specific nuances in performance evaluation is interesting. But can these results be made more general or useful to improve the paper?  E.g. can you claim that Pytorch/JAX parity is impossible to achieve for specific workloads?\n- Unclear fit with ICLR: The paper reads more like an experience report (e.g. Kaggle summaries), rather than a research paper. While the experiences are interesting, the novel contributions/lessons are limited. The lack of a test set and lack of common workloads also limit the applicability. The paper would be likely be a better fit for a software engineering conference both in terms of fit and conference attendee interests.\n- Challenges with methodology: The competition evaluation is resource intensive and uses a validation test. Most competitions are evaluated on test sets, and a note describing how the results/methodology can be extended to include test sets would help improve the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TAvYTPlVll", "reviewer": "Reviewer_9ctZ", "summary": "This paper analyses the results of the AlgoPerf competition. For that, it presents a summary of the methodology, a detailed description of the winning submissions, how the evaluation was carried out, the implementation details of the competition itself, and the engineering challenges they faced. The main goal of the competition is to evaluate the effectiveness of training algorithms. This is done by measuring how long submissions take to achieve some evaluation goal on some defined workload with restricted runtime and budget. In general, results highlight the competitiveness of the benchmark, as few of the submissions were able to do well on all the different workloads, indicating lots of room for improvement.", "questions": "- \"Although a radical change from the current practice of published training algorithms that aren’t runnable without setting various hyperparameters, publishing families of update rules and abdicating responsibility for tuning entirely to the user only adds to the community’s confusion on what to actually use\". This is an interesting comment that is hidden in the bulk of the text. It would be interesting if the authors clarified this comment, and expanded on it further.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Authors provide a detailed analysis of their competition, including the methodology, the best results, and lessons learnt.\n - Originality of the work lies in having the initiative to setup the competition (organisation, dissemination, infrastructure set up), and reporting the results obtained. \n - Analysis is extensive. Authors provide tables and graphics to showcase the results of the competition.\n - Authors provide low level details and lessons learnt also on the implementation and mainteinance of the benchmark, comparing Pytorch and JAX.", "weaknesses": "- Novelty: Besides the competition results and the insights obtained, novelty is not high. Contributions are mainly the insights extracted from the submissions. It feels more like a report (summarising results obtained from a competition). I would encourage authors to further highlight the contributions they make, clearly stating that this benchmark is solving a gap, and backing up the claims. In addition, I believe that some of the lessons learnt highlighted in bold are not novel but already established practises (e.g. \"having fair comparisons in a competition\" is something widely known and established).\n - Clarity: Narrative can be improved in some sections. E.g. Section 3 is specially dense to read, and is not always clear what the authors want to convey. I encourage authors to include, at the start of each paragraph in section 3, a sentence that summarises the main findings of that paragraph. (Eg. ResNet workload subsection. Then, main takaway sentence. Then, the rest of the details, numbers, statistics, etc.)\n - I believe authors could improve the significance of the work by better motivating the need of this benchmark. Why is this benchmark important and needed? Is it the first benchmark to allow evaluation of training algorithms? What makes it different from other benchmarks?  The current paper is lacking a strong motivation background and more evidence. For example, that there is a real need for self-tuning algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pWrYuv3MbP", "reviewer": "Reviewer_QEMB", "summary": "The paper describes the methodology and results of the \"AlgoPerf: Training Algorithms\" competition, which aims to evaluate the speed-ups of neural network training by modifying the underlying training algorithm.\n\nThe competition covers two rulesets: \"external tuning\", which requires a hyperparameter search space that is workload agnostic, and \"self-tuning\", which is hyperparameter-free. This paper details the winners of both rulesets \"Distributed Shampoo\" and \"Schedule Free AdamW\".\n\nFinally, the authors detail the issues they encountered while developing this competition, highlighting compatibility and performance issues between different frameworks and improving the respective implementation by copying the better-performing one.", "questions": "I would like the authors to address W1.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: Ensuring a fair comparison with JAX and PyTorch is likely impossible, but the authors did a good job of tracing most issues (mathematical correctness, comparing kernel runtimes, etc.). They stopped at memory allocation patterns, which is arguably impossible to get outside of creating an intermediate translation layer between NVIDIA GPU drivers and whatever TPUs are using. This decision showed improvement potential in both frameworks, as the direct comparison between them showcased performance gaps that could be easily closed by copying the better-performing implementation.\n\nS2: The lessons learned are very interesting for practitioners and future competition creators, outlining gaps in current algorithmic development and the dependency on hyperparameter tuning to get the best results.", "weaknesses": "W1: I am missing a more detailed analysis of why the winners of the respective rulesets came first. While this paper is more about the competition itself, I would love for it to be slightly more useful for practitioners questioning whether they should drop AdamW for Distributed Shampoo in their experiments. Other questions like whether the current on-trend LLM training will see significant changes due to the results from AlgoPerf (due to the significant cost to training these models) might provide a slightly better outlook and highlight the impact of this competition.\n\nMinor issues:\n- Typo in Line 382: \"framekworks\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CSoPGS3EV6", "reviewer": "Reviewer_XF5p", "summary": "This paper investigates the AlgoPerf competition and provides insights into model training acceleration. Some findings are reasonable and could potentially guide the design of training algorithms. As this is a survey rather than a technical paper, I am not sure whether it is qualified to be published in ICLR. Therefore, I seek to consider the opinions of other reviewers.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Some interesting findings are provided and they may help to design more efficient training algorithms.", "weaknesses": "I am not an expert in evaluating this kind of survey paper, and I am curious about how the findings can be applied to refine existing algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)