Benchopt: Reproducible, efficient and collaborative optimization benchmarks
Authors: Thomas Moreau
Mathurin Massias
Alexandre Gramfort
Pierre Ablin
Pierre-Antoine Bannier
Benjamin Charlier
Mathieu Dagréou
Tom Dupre la Tour
Ghislain Durif
Cássio Fraga Dantas
Quentin Klopfenstein
Johan Larsson
En Lai
Tanguy Lefort
Benoît Malézieux
Badr Moufad
Binh Nguyen
Alain Rakotomamonjy
Zaccharie Ramzi
Joseph Salmon
Samuel Vaiter
Venue: NeurIPS 2022 Accept (2022)
Keywords: reproducibility
optimization
lasso
resnet
logistic regression
open source software
benchmark

Abstract: Numerical validation is at the core of machine learning research as it allows us to assess the actual impact of new methods, and to confirm the agreement between theory and practice. Yet, the rapid development of the field poses several challenges: researchers are confronted with a profusion of methods to compare, limited transparency and consensus on best practices, as well as tedious re-implementation work. As a result, validation is often very partial, which can lead to wrong conclusions that slow down the progress of research. We propose Benchopt, a collaborative framework to automatize, publish and reproduce optimization benchmarks in machine learning across programming languages and hardware architectures. Benchopt simplifies benchmarking for the community by providing an off-the-shelf tool for running, sharing and extending experiments. To demonstrate its broad usability, we showcase benchmarks on three standard ML tasks: $\ell_2$-regularized logistic regression, Lasso and ResNet18 training for image classification. These benchmarks highlight key practical findings that give a more nuanced view of state-of-the-art for these problems, showing that for practical evaluation, the devil is in the details.

Benchopt: Reproducible, efficient and collaborative optimization benchmarks
Thomas Moreau !.", Mathurin Massias Alexandre Gramfort Pierre Ablin', Pverre-Antiin Bannier, Benjamin Charlier', Mathieu Dagréou', Tom Dupré la Tours, Ghislain Durif', Cassio F. Dantas", Quentin Klopfenstein' Johan Larsson', En Lai', Tanguy Lefort', Benoit Malézieux², Badr Moufad Binh T. Nguyen Alain Rakotomamoniy";
Zaccharie Ramzi Joseph Salmon 4.5, Samuel Vaiter
Université Paris-Saclay, Inria, CEA, 91120 Palaiseau, France
2 Univ Lyon, Inria, CNRS, ENS de Lyon, UCB Lyon 1, LIP UMR 5668, F-69342, Lyon, France
3 Université Paris-Dauphine, PSL University, CNRS, 75016, Paris, France IMAG, Univ Montpellier, CNRS, Montpellier, France 5 Institut Universitaire de France (IUF)
University of California, Berkeley, CA 94720, USA 7 TETIS, Univ Montpellier, INRAF, Montpellier, France
University of Luxembourg, LCSB, Esch-sur-Alzette, Luxembourg
The Department of Statistics, Lund University
LTCI, Télécom Paris, 91120 Palaiseau, France
11 Criteo Lab, Paris, France 12 ENS Ulm, CNRS. UMR 8553, Paris, France
13 CNRS & Université Côte d' Azur, Laboratoire J.A. Dieudonné, CNRS, Nice, France
Abstract
Numerical validation is at the core of machine learning research as it allows to assess the actual impact of new methods, and confirm the agreement between theory and practice. Yet, the rapid development of the field poses several challenges: researchers are confronted with a profusion of methods compare, limited trans- parency and consensus on best practices, as well as tedious re-implementation work. As a result, validation is often very partial, which can lead to wrong conclusions that slow down the progress of research. We propose Benchopt, a collaborative frame- work to automate, reproduce and publish optimization benchmarks machine learning across programming languages and hardware architectures. Benchopt simplifies benchmarking for the community by providing an off-the-shelf tool for running, sharing and extending experiments. To demonstrate its broad usability, we showcase benchmarks on three standard learning tasks: Ez-regularized logistic re- gression, Lasso, and ResNet18 8 training for image classification. These benchmarks highlight key practical findings that give a more nuanced view of the state-of of-te--f-hh-art for these problems, showing that for practical evaluation, the devil is in the details. We hope that Benchopt will foster collaborative work in the community hence improving the reproducibility of research findings.
Introduction
Numerical experiments have become an essential part of statistics and machine learning (ML). is now commonly accepted that every new method needs to be validated through comparisons with existing approaches on standard problems. Such validation provides insight into the method's benefits and limitations and thus adds depth to the results. While research aims at advancing knowledge and not just improving the state of the art, experiments ensure that results are reliable and support theoretical claims (Sculley et al., 2018). Practical validation also helps the ever-increasing number of MI. users applied sciences to choose the right method for their task. Performing rigorous and extensive experiments is, however, time-consuming (Raff, 2019), particularly because comparisons
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Benchmark
Objectives Datasita Solvers
oniqut camera ready paper
.PDF
Objortives:
beachopt ruc
benchopt plot
Datazets:
saved
Dynamic output untercctive exploration
in log file
JTTHL
Solvers
beachopt publish
julia
Reproducible rosults online
Figure 1: A visual summary of Benchopt. Each Solver run (in parallel) on each Dataset and each variant of the Objective. Results are exported as a CSV file that is easily shared and can be automatically plotted as interactive HTML visualizations PDF figures.
against existing methods in new settings often requires reimplementing baseline methods from the literature. In addition, ingredients necessary for a proper reimplementation may be missing, such as important algorithmic details, hyperparameter choices, and preprocessing steps (Pineau et al., 2019).
In the past years, the MI. community has actively sought to overcome this "reproducibility crisis" (Hutson, 2018) through collaborative initiatives such open datasets (OpenML, Vanschoren al. 2013), standardized code sharing (Forde et al., 2018), benchmarks (MLPerf, Mattson et al. 2020), the NeurIPS and ICLR reproducibility challenges (Pineau et al., 2019; Pineau al., 2021) and new journals (e.g., Rougier and Hinsen 2018). As useful as these endeavors may be, they do not, however, fully address the problems in optimization for ML since, in this area, there are no clear community guidelines on how to perform, share, and publish benchmarks.
Optimization algorithms pervade almost every of ML. from empirical risk minimization, varia- tional inference to reinforcement learning (Sra et al., 2012). It is thus crucial to know which methods the task and setting Beitt-Beeelltenn et al., 2020). While
to use depending optimization for MI. provide extensive validations (Lueckmann et al., 2021), many others fall short in this regard due to lack of time and resources, and in turn feature results that are hard reproduce by other researchers. In addition, both performance and hardware evolve over time, which eventually makes static benchmarks obsolete. An illustration of this the recent work by Schmidt al. (2021), which extensively evaluates the performances of 15 optimizers across 8 deep-learning tasks. While their benchmark gives overall assessment of the considered solvers, this assessment is bound to become out-of-date if it is not updated with new solvers and new architectures. Moreover, the benchmark does not reproduce oftte-of-hhearte results on the different datasets, potentially indicating that the considered architectures and optimizers could be improved.
papers in
We firmly believe that this critical task of maintaining an up-to-date benchmark in field cannot solved without collective effort. We want to empower the community to take up this challenge and build a living, reproducible and standardized state of the art that can serve as a foundation for future research.
Benchopt provides the tools to structure the optimization for machine learning (Opt-ML) community around standardized benchmarks, and aggregate individual efforts for reproducibility and results sharing. Benchopt can handle algorithms written in Python, R. Julia or C/C++ via binaries. It offers built-in functionalities to ease the exccution of benchmarks: parallel runs, caching, and automatic results archiving. Benchmarks are meant to evolve over time, which is why Benchopt offers modular structure through which a benchmark can be easily extended with new objective functions, datasets, and solvers by the addition of a single file of code.
The paper is organized as follows. We first detail the design and usage of Benchopt, before presenting results on three canonical problems:
f Ez-regularized logistic regression: a convex and smooth problem which is central to the evaluation of many algorithms in the Opt-ML community, and remains of high relevance for practitioners;
the Lasso: the prototypical example of non-smooth convex problem in MI.;
training of ResNet18 architecture for image classification: a large scale non-convex deep learning problem central in the field of computer vision.

---

The reported benchmarks, involving dozens of implementations and datasets, shed light on the current state-of-the-an solvers for each problem, across various settings, highlighting that the best algorithm largely depends on the dataset properties (e.g., size, sparsity), the hyperparameters, as well as hardware. A variety of other benchmarks (e.g. MCP, TV1D, etc.) are also presented Appendix, with the goal to facilitate contributions from the community.
By the open source and collaborative design of Benchopt (BSD 3-clause license), we aim to open the way towards community-endorsee and peer-reviewed benchmarks that will improve the tracking of progress in optimization for ML.
2 The Benchopt library
The Benchopt library aims to provide standard toolset and structure to implement benchmarks for optimization in ML, where the problems depend on some input dataset DD The considered problems are of the form
0*
E arg min ((A)
liee
where f is the objective function, A are its hyperparameters, and θ is the feasible set for The scope of the library is to evaluate optimization methods in their wide sense by considering the sequence {8'}, produced to approximate 8*, We emphasize than Benchopt does not provide fixed set of benchmarks, but framework to create, extend and share benchmarks on any problem of the form (1). To provide a flexible and extendable coding standard, benchmarks are defined as the association of three types of object classes:
Objective: It defines the function f to be minimized well as the hy- perparameters A or the set θ, and the metrics to track along the iterations (e.g.. objective value, gradient norm for smooth problems, validation loss). Multiple metrics can be registered for each 0'.
benchmark/
objective.py
datasets/
dataset1 py
Datasets: The Dataset objects provide the data D to be passed to the Objective class. They control how data is loaded and preprocessed. Datasets are separated from the Objective, making it easy to add new ones, provided they coherent with the Objective.
dataset2.pp py
solvers/
solver1.py
solver2.py
The Solver objects define how run the algorithm. They are provided with the Objective and Dataset objects and output a sequence be obtained using single run of the method, or with multiple runs in case the method only returns its final iterate.
Solvers:
Figure 2: Standard
benchmark structure
{0'). This sequence
Each of these objects can have parameters that change their behavior, e.g. the regularization parameters for the Objective, the choice of preprocessing for the Datasets, or the step size for the Solvers. By exposing these parameters in the different objects, Benchopt influence on the benchmark results. The Benchopt library defines an application programming interface (API) for each of these concepts and provides command line interface (CLI) to make them work together. A benchmark is defined as a folder that contains an Objective as well as subfolders containing the Sol vers and Datasets. Appendix B presents concrete example on Ridge regression of how to construct a Benchopt benchmark while additional design design choices of Benchopt discussed in Appendix C.
evaluate their
For each Dataset and Solver, and for each set of parameters, Benchopt retrieves a sequence {0'). and evaluates the metrics defined in the Objective for each 0¹. To ensure fair evaluation, the computation these metrics is done off-line. The metrics are gathered in a CSV file that can be used to display the benchmark results, either locally the benchmarks run with Benchopt. This workflow is described in Figure 1.
HTML files published on a website that reference
This modular and standardized organization for benchmarks empowers the optimization community by making numerical experiments easily reproducible, shareable, flexible and extendable. The benchmark can be shared as a git repository or a folder containing the different definitions for the Objective, Datasets and Sol vers and it can be run with the Benchopt CLI, hence becoming a convenient reference for future comparisons. This ensures fair evaluation of baselines in follow-up experiments, as implementations validated by the community are available. Moreover, benchmarks can be extended casily as one can add a Dataset or Solver to the comparison by adding single
3

---

file. Finally, by supporting multiple metrics e.g.. training and testing losses, error on parameter estimates, sparsity of the estimate the Objective class offers the flexibility to define the concurrent evaluation, which can be extended to track extra metrics on per benchmark basis, depending on the problem at hand.
As one of the goal of Benchopt is to make benchmarks as simple as possible, it also provides a set of features to make them easy to develop and run. Benchopt written in Python, but Solvers run with implementations in different languages (e.g., R and Julia, as in Section 4) and frameworks (e.g. PyTorch and TensorFlow as Section 5). Moreover, benchmarks can be run in parallel with checkpointing of the results, enabling large scale evaluations many CPU or GPU nodes. Benchopt also makes it possible to run solvers with many different hyperparameters' values, allowing assess their sensitivity on the method performance. Benchmark results are also automatically exported as interactive visualizations, helping with the exploration of the many different settings.
Benchmarks All presented benchmarks are run on 10 cores of an Intel Xeon Gold 6248 CPUs @ 2.50GHz and NVIDIA V100 GPUs (16GB). The results' interactive plots and data are available at https://benchoott. gi thub.
ts. html.
3 First example: lz-regularized logistic regression
Logistic regression is very popular method for binary classification. From a design matrix X E IR"XP with rows X; and a vector of labels E {- with corresponding element Vr+ (z-regularized logistic regression provides generalized linear model indexed by 0" IR? discrim- inate the classes by solving
A log (1 exp(-xxXX 0)) + + 1012
0" = arg mii
(2)
DERP
where À >> is the regularization hyperparameter. Thanks the regularization part, Problem (2) is strongly convex with a Lipschitz gradient, and thus its solution can be estimated efficiently using many iterative optimization schemes.
The most classical methods to solve this problem take inspiration from Newton's 's method (Wright and Nocedal, 1999). On the one hand, quasi-Newton methods aim at approximating the Hessian of the cost function with cheap to compute operators. Among these methods, L-BFGS (Liu and Nocedal, 1989) stands out for its small memory footprint, its robustness and fast convergence in variety of settings. On the other hand, truncated Newton methods (Dembo et al., 1982) try to directly approximate Newton's 's direction by using e.g., the conjugate gradient method (Fletcher and Reeves, 1964) and Hessian-vector products to solve the associated linear system. Yet, these methods suffer when n is large: each iteration requires a pass the whole dataset.
In this context, methods based on stochastic estimates of the gradient have become standard (Bottou, 2010), with Stochastic Gradient Descent (SGD) as main instance. The core idea is to use cheap and noisy estimates of the gradient (Robbins and Monro, 1951; Kiefer and Wolfowitz, 1952). While SGD generally converges either slowly due to decreasing step sizes, for constant step sizes,
neighborhood of the solution adaptations such as SAG (Schmidt et al., 2017), SAGA (Defazio et al., 2014) and SVRG (Johnson and Zhang, 2013) make it possible to solve the problem more efficiently and are often considered to be state-of-the-art for large scale problems.
Finally, methods based on coordinate descent (Bertsekas, 1999) have also been proposed to solve Problem (2). While these methods are usually less popular, they can be efficient in the context of sparse datasets, where only few samples have non-zero values for a given feature, when accelerated on distributed systems or GPU (Dünner et al., 2018).
The code for the benchmark is available at https: //tp::://ggi //gttbbb.. 12/. To reflect the diversity of solvers available, we showcase Benchopt benchmark with datasets, 10 optimization strategies implemented in 5 packages, leveraging GPU hardware when possible. We also consider different scenarios for the objective function: (i) scaling (or not) the features, a recommended data preprocessing step, crucial in practice to have comparable regularization strength on all variables; (ii) fitting (or not) an unregularized intercept term, important in practice and making optimization harder when omitted from the regularization term (Koh et al., 2007); (iii working (or
ahhenchopt/benchhmrkke

---

acikib-lcamjiien lcam|liblinerr
snapMLispu
lightning/sas)
sclkit (carn
scikit-eammlliig [b[aa]]
(aapM
hightning/saaa tick/ovag
scikit- learnss
aclkit.lcam/nnwtorre
lightning(cd]
solkit lcarnaaaaaa
cuML]ssp]]
Raw
Scaled
Intercept
104
ijcnnl
109
10
10-
10
10a
10
10
10
10"
10
22
10
10
10a
madelon
10
10-4
10
3
10a
10a
10a
10
10
2
10a
10
10a
3
10a
10
10a
10a
news20 binary
101
109
10
10
102
10a
10
10
10a
10
10a
10a
10
criteo
109
10 4
10-8
10P
10
10a
10a
10"
10:
10a
10a
10a
10
10a
10a
Time (s)
Time (s)
Time (s)
Benchmark for the fz-regularized logistic regression, on 13 solvers, datasets (rows), and 3 variants of the Objective (columms) with X = 1. The curves display the suboptimality of the iterates, f(0)) f(0), as function of time. The first column corresponds the objective function detailed in Problem (2). In the second column, datasets were preprocessed by normalizing each feature unit standard deviation. The third column is for an objective function which includes an unregularized intercept.
Figure 3:
not) with sparse features, which prevent explicit centering during preprocessing keep memory usage limited. Details on packages, datasets and additional scenarios are available in Appendix D.
Results Figure D.1 presents the results of the benchmarks, in terms of suboptimality of the iterates, f(0') - ((0",, for three datasets and three scenarios. Here, because the problem is convex, 0" is approximated by the best iterate across all runs (see Section C. Overall, the benchmark shows the benefit of using GPU solvers (cuML and snapML), even for small scale tasks such as ijennl. Note that these two accelerated solvers converge higher suboptimality level compared to other solvers, due to operating with 32-bit float precision. Another observation is that data scaling can drastically change the picture. In the case of madelon, most solvers have a hard time converging for the scaled data. For the solvers that converge, we note that the convergence time one order of magnitude smaller with the scaled dataset compared the raw one. This stems from the fact that in this case, the scaling improves the conditioning of the dataset.' For news20.binan the stochastic solvers such as SAG and SAGA have degraded performances scaled data. Here, the scaling makes the problem harder?
I TThe condition number the dataset divided by 5.9 after scaling.
The condition number is multiplied by 407 after scaling.

---

On CPU, quasi-Newton solvers are often the most efficient ones, and provide a reasonable choice in most situations. For large scale News20.binary stochastic solvers such as SAG, SAGA or SVRG -that are often considered as state of the art for such problem- have worst performances for the presented datasets. While this dataset often used as a test bed for benchmarking new stochastic solvers, we fail to see an improvement over non-stochastic ones for this experimental setup. In contrast, the last row in Figure D.I displays an experiment with the larger scale criteo dataset, which demonstrates a regime where variance-reduced stochastic gradient methods outperform quasi-Newton methods. For future benchmarking of stochastic solvers, we therefore recommend using such a large dataset.
Finally, the third column in Figure D.I illustrates classical problem when benchmarking different solvers: their specific (and incompatible) definition and resolution of the corresponding optimization problem. Here, the objective function is modified to account for an intercept (bias) in the linear model. In most situations, this intercept is not regularized when it is fitted. However, snapML and liblinear solvers do regularize it, leading to incomparable losses.
4 Second example: The Lasso
The Lasso, (Tibshirani, 1996; Chen et al., 1998), is an archetype of non-smooth ML problems, whose impact on ML, statistics and signal processing in the last three decades has been considerable (Bühlmann and van de Geer, 2011; Hastie et al., 2015), It consists of solving
0 E arg min 1 Ily --00 A 1|01
(3)
PERP
where X E RPXP is a design matrix containing p features as columns, y E R is the target vector, and À 0 regularization hyperparameter, The Lasso estimator was popularized for variable selection: when is high enough, many entries in 0* are exactly equal to 0. This leads to more interpretable models and reduces overfitting compared the least-squares estimator.
Solvers for Problem (3) have evolved since its introduction by Tibshirani (1996). After generic quadratic program solvers, new dedicated solvers squares (IRLS) (Grandvalet, 1998), followed by LARS (Efron et al., 2004), a homotopy method computing the full Lasso path The LARS solver helped popularize the Lasso, yet the algorithm suffers from stability issues and can be very slow for worst case situations (Mairal and Yu, 201 2). General purpose solvers became popular for Lasso-type problems with the introduction of the iterative soft thresholding algorithm (ISTA, Daubechies et al. 2004), an instance of bowarrd-bcckwee splitting (Combettes and Wajs, 2005). These algorithms became standard in signal and image processing, especially when accelerated (FISTA, Beck and Teboulle 2009).
proposed based iterative reweighted least-
In parallel, proximal coordinate descent has proven particularly relevant for the Lasso in statistics. Early theoretical results were proved by Tseng (1993) and Sardy et al. (2000), before it became the standard solver of the widely distributed packages glmnet t in R and scikit-learn in Python. For further improvements, some solvers exploit the sparsity of 8*, trying to identify its support to reduce the problem size. Best performing variants of this scheme are screening rules (e.g., El Ghaoui et al., 2012; Bonnefoy et al., 2015; Ndiaye et al., 2017) and working/active sets (e.g., Johnson and Guestrin 2015; Massias et al. 2018), including strong rules (Tibshirani et al., 2012).
While reviews of Lasso solvers have already been performed (Bach et al., 2012, Sec. 8.1), they are limited certain implementation and design choices, but also naturally lack comparisons with more recent solvers and modern hardware, hence drawing biased conclusions.
The code for the benchmark is available at https: //g thub. com/ beenchopt/benchmarkk _aasoo// Results obtained on 4 datasets, with standard packages and some custom reimplementations, possi- bly leveraging GPU hardware, and 17 different solvers written in Python/numba/Cython R, Julia or C++ (Table 1) are presented in Figure 4. All solvers use efficient numerical implementations, pos- sibly leveraging calls to BLAS, precompiled code in Cho or just-in-time compilation with numba.
The different parameters influencing the setup are
the regularization strength X, controlling the sparsity of the solution, parameterized as a fraction ||x vll.c (the minimal hyperparameter such that 0* 0).
of Amax
'The Lasso path is the set of solutions of Problem (3) À varies in (0, oo).

---

the dataset dimensions: MEG has small and medium revl.binary has medium n and p; Nows20.binary has medium and very large p while MillionSong has very large and small p (Table F.2).
blitz
celer
coordinate descent
cuML]ccd]
cuML[an]]
FISTA
FISTA(adaptive-I]
FISTA[greedy]
glmnet
ISTA
LARS
lasso jl
noncwx-pro pro
scikit-learn
skglm
snapML (cpu)
snapML[ggu]
0. 0.1ax
0.01 0.n0A
0.001 Amax
10
MEG
10-2
10-5
10-8
10-2
10
10P
10
10a
10a
10
10
10
10
10a
10
10
10
10
10
10
10
107
MillionSong
10a
10
10-2
10-5
10-2
10
100
10a
10
10
109
10 2
10
109
10
10a
rcv1 binary
10a
10-3
10
10
10
10
10
10a
10
10
10
10
10a
10
10
10
10
10a
news20 binary
10a
10a
10-3
10-6
10
10P
10a
10a
10a 10
10"
10a
10a
10a
10
10
10a
10a
10a
Time (s)
Time (s)
Time (s)
Figure 4: Benchmark for the Lasso, 17 solvers, 4 datasets (rows), and 3 variants of the Objective (columns) with decreasing regularization X. The curves display the suboptimality the objective function, f(0t) f(0),, as function of time.
Results Figure 4 presents the result of the benchmark on the Lasso, in terms of objective subopti- mality f(0') (
function of time.
Similarly to Section 3, the GPU solvers obtain good performances in most settings, but their advantage is less clear. A consistent finding across all settings is that coordinate descent-based methods outperform full gradient ones (ISTA and FISTA, even restarted), and are improved by the use of working set strategies (blitz, celer, skglm, glmnet). This observation is even more pronounced when the regularization parameter is large, as the solution sparser.
When observing the influence of the dataset dimensions, we observe regimes. When n is small (MEG), the support the solution is small and coordinate descent, LARS and noncvx-pro perform the best. When n is much larger than (MillionSong), noncvx-po -pro clearly outperforms other solvers, and working set methods prove useless. Finally, when n and p are large (rev). aevllbinary, (ewx20.binary), CD and working sets vastly outperforms the rest while noncvx-pro fails, as it requires solving linear system of size min(N, p). We note that this setting was not tested the original experiment of Poon and Peyré (2021), which highlights the need for extensive, standard experimental setups.
7

---

When the support of the solution is small (either small X, either small n since the Lasso solution has at most n nonzero coefficients). LARS a competitive algorithm. We expect this to degrade when increases, but as the LARS solver in scikit-learn does not support sparse design matrices we could not include for news20. news2O.binary and rev aev..ry
This benchmark is the first to evaluate solvers across languages, showing the competitive behavior of lasso. and glmnet compared Python solvers. Both solvers have large initialization time, and then converge very fast. To ensure that the benchmark is fair, even though the Benchopt library implemented in Python, we made sure to ignore conversion overhead, as well as just-in-time compilation cost. We also checked the timing's consistency with native calls to the libraries.
Since the Lasso is massively used for it feature selection properties, the speed at which the solvers identify the support of the solution is also an important performance measure. Monitoring this with Benchopt is straightforward, and figure reporting this benchmark is in Appendix E.
Third example: How standard is a benchmark on ResNet18?
As early successes of deep learning have been focused on computer vision tasks (Krizhevsky et al., 2012), image classification has become a de facto standard to validate novel methods in the field. Among the different network architectures, ResNets (He et al., 2016) are extensively used in the community as they provide strong and versatile baselines (Xie et al., 2017; Tan and Le, 2019; Dosovitskiy et al., 2021; Brock et al., 2021; Liu et al., 2022). While many papers present results with such model on classical datasets, with sometimes extensive ablation studies (He et al., 2019; Wightman et al., 2021; Bello et al., 2021; Schmidt et al., 2021), the lack of standardized codebase and missing implementation details makes it hard to replicate their results.
The code for the benchmark is available at https: //g gi thub. classif/. We provide a cross-dataset -SVNN, Netzer al. (2011); MNIST, LeCun et al. (2010) and CIFAR-1O, Krizhevsky (2009)- and cross- framewrr -TensorF low/Keras, Abadi et al. (2015) and Chollet et al. (2015); PyTorch, Paszke et al. (2019)- evaluation of the training strategies for image classification with ResNet18 8 (see Appendix F for details on architecture and datasets). We train the network by minimizing the cross entropy loss relatively to the weights of the model. Contrary to logistic regression and the Lasso, this problem is non-convex due to the non-linearity of the model fø- Another notable difference is that we report the evolution of the test error rather than the training loss.
Because we chose to monitor the test loss, the Solvers are defined as the combination of an optimization algorithm, its hyperparameters, the learning rate (LR) and weight decay schedules, and the data augmentation strategy. This is in contrast to case where we would monitor the train loss, and therefore make the LR and weight decay schedules, as well as the data augmentation policy, part of the objective. We focus on 2 standard methods: stochastic gradient descent (SGD) with momentum and Adam (Kingma and Ba, 2015), as well as a more recently published one: Lookahead (Zhang et al., 2019). The LR schedules are chosen among fixed LR, step LR and cosine annealing (Loshchilov and Hutter, 2017). We also consider decoupled weight decay for Adam (Loshchilov and Hutter, 2019), and coupled weight decay (i.e., fz-regularization) for SGD. Regarding data augmentation, use random cropping for all datasets and add horizontal flipping only for CIFAR- 10, as the digits datasets do not exhibit mirror symmetry. We detail the remaining hyperparameters in Table F.2, and discuss their selection as well as their sensitivity in Appendix F.
Aligning cross-framewwrr implementations Due to some design choices, components with the same name in the different frameworks do not have the same behavior. For instance, when comes to applying weight decay, PyTorch's 's SGD uses coupled weight decay, while in Flos/erlow/ersss weight decay always refers to decoupled weight decay. These two methods lead to significantly different performance and it is not straightforward to apply coupled weight decay in a post-hoc manner in Tensor low/Keras (see further details in Section F.3). We conducted an extensive effort to align the networks implementation in different frameworks using unit testing to make the conclusions of our benchmarks independent the chosen framework. We found additional significant differences (reported in Table F.3) in the initialization, the batch normalization, the convolutional layers and the weight decay scaling.
'decreasing the learning rate by factor 10 at mid aid-training. and again at 3/4 of the training
8

---

Vanilla SGD
SGD + data aug. + momentum + cosine LR sched.
Best Adam
SGD data aug.
Best SGD
Lookahead
SGD + data aug. + momentum
Best SGD (TF/Keras)
CIFAR-10
SVHN
MNIST
Test error (%)
15-
2-
10-
4-
1000
2000
3000
2000
4000
1000
2000
Time (s)
Time (s)
Time (s)
The best SGD configuration features data augmentation, momentum, cosine learning rate schedule and weight decay. In dashed black is the state of the art for the corresponding datasets with ResNet18 measured by Zhang et (2019) for CIFAR- 10, by Zheng al. (2021) ) for SVHN with a PreAct ResNet18, by Papers WithCode for MNIST with all networks considered. Off-the-shelf ResNet implementations in TensorFlower LowKeras do not support images smaller than 32 32 and is hence not shown for MNIST. Curves are exponentially smoothed.
Figure 5: ResNet18 image classification benchmark with PyTorch Solvers.
Results The results of the benchmark are reported in Figure 5. Each graph reports the test error relative to time, with ablation study on the solvers' parameters. Note that only report selected settings for clarity but that we run every possible combination."
Firstly, reaching the state of the art for a vanilla ResNet18 is not straightforward. On the popular website Papers with code it has been so underestimated. It can achieve 4.45% and 2.65% test error rates on CIFAR-/0 10 and SVHN respectively (compared to 4.73% and 2.95% for a PreAct ResNet18 before that). Our ablation study shows that variety of techniques is required to reach it. The most significant one is appropriate data augmentation strategy, which lowers the error rate on CIFAR-10 from about 18% to about 8%. The second most important one is weight decay, but it has to be used in combination with proper LR schedule, well as momentum. While these techniques not novel, they are regularly overlooked in baselines, resulting in underestimation of their performance level.
This reproducible benchmark not only allows a researcher to get clear understanding of how to achieve the best performances for this model and datasets, but also provides a way to reproduce and extend these performances. In particular, we also include in this benchmark the original implementa- tion of Lookahead (Zhang al., 2019). We confirm that slightly accelerates the convergence the Best SGD, even with a cosine LR schedule a setting that had not been studied in the original paper.
Our benchmark also evaluates the relative computational performances of the different frameworks. We observe that TiTforc--Lighning is significantly slower than the other frameworks we tested, in large part due their callbacks API. We also notice that our TensorFlow/Keras implementation is significantly slower (= 28%) than the PyTorch ones, despite following the best practices and our profiling efforts. Note that we do not imply that TensorFlow is intrinsically slower than PyTorch, but a community effort is needed to ensure that the benchmark performances are hammewwrk-agnosstee
recurrent criticism of such benchmarks is that only the best test error is reported. In Figure 6, measure the effect of using a vaiinvalidation--ess split, by keeping a fraction of the training set as validation set. The splits ablation study do not change significantly when using such procedure, even though their validity is reinforced by the use of multiple trainings. Yet, possible limitation of our findings that some of the hyperparameters we used for our study, coming from the Py Torch-CIFAR GitHub repository, may have been tuned while looking at the test set.
detailed in Table F.1. Our finding is that the results of the
'The results are available online user-friendly interactive HTML file.

---

Best SGD with step LR schedule (no val test)
Best SGD with cosine LR schedule (no val test)
Best SGD with step LR schedule (val_test)
Best SGD with cosine LR schedule (val_test)
CIFAR-10
SVHN
MNIST
10
Test/Val error (%)
8-
6-
1000
1500
2000
2500
1000
2000
3000
1000
2000
Time (s)
Time (s)
Time (s)
Figure 6: ResNet18 image classification benchmark with validation split. In dashed black is the state of the art (see caption of Figure 5 for more details). In addition, we show in colored horizontal dashed lines, the test results for early stopping on the validation and on the test set for the different solvers, the square mark indicating the moment this stopping would happen. The curves for the train-val splits show the exponentially smoothed median results for five different random seeds.
6 Conclusion and future work
We have introduced Benchopt, a library that makes it easy to collaboratively develop fair and extensive benchmarks of optimization algorithms, which can then be scamlessly published. reproduced, and extended. In the future, we plan on supporting the creation of new benchmarks, that could become the standards the community builds on. This work is part of wider effort improve reproducibility of machine learning results. aims to contribute to raising the standard of numerical validation for optimization, which is pervasive in the statistics and ML community well as for the experimental sciences that rely more and more on these tools for research.
7 Acknowledgements
can not be stressed enough how much the Benchopt library relies on contributions from the community and in particular the Python open source ecosystem. In particular, it could not exist without the libraries mentioned in Appendix A.
This work was granted access to the HPC resources of IDRIS under the allocation 2022- AD01101 117222 and 2022-AD01 1013570 made by GENCI, which used to run all the bench- marks. MM also gratefully acknowledges the support of the Centre Blaise Pascal's IT test platform ENS de Lyon (Lyon, France) for Machine Learning facilities. The platform operates the SIDUS solution (Quemener and Corvellec, 2013).
TL, CFD and JS contributions
supported by the Chaire IA CaMeLOt (ANR- 108--20CHHAAL 1-HA--0011-001 AG, EL and TM contributions were supported by the Chaire IA ANR BrAIN (ANR SANR-20-CHII 30-CHIA-0016) BMa contributions were supported by grant from Digiteo France. MD contributions were supported by public grant overseen by the French National Research Agency (ANR) through the program UDOPIA, project funded by the ANR-20-THIA-001344 0013-01 and DATAIA convergence institute (ANR- 17-CONY-0003). BN work was supported by the Télécom Paris's Chaire DSAIDIS (Data Science & Artificial Intelligence for Digitalized Industry Services). BMo contributions supported by grant from the Labex MILYON.
References
Abadi, M.. A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L.. Kaiser, M. Kudlur, J. Levenberg. D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, E. Viégas,
O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng (2015). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Software available from tensorflow.org.
10

---

Akiba, T., S. Sano, T. Yanase, T. Ohta, and M. Koyama (2019). Optuma: A Next-generatien Hyperpa- rameter Optimization Framework.
Bach, F., R. Jenatton, J. Mairal, and G. Obozinski (2012). "Optimization with sparsity-inducing
penalties". In: Foundations and Trends Machine Learning 4.1, pp. 1-106.
Bacry, E., M. Bompaire, S. Gaïffas, and S. Poulsen (2017). "Tick: A Python Library for Statistical Learning, with a Particular Emphasis on Time- -eeenDeenddent Modeling". In: ArXiv e-prints.
Barbero, A. and S. Sra (2018). "Modular Proximal Optimization for Multidimensional Regularization" In: Journal of Machine Learning Research 19.56, pp. 1-82.
Barlow, R. E. and H. D. Brunk (1972). "The Isotonic Regression Problem and Its Dual". In: Journal of the American Statistical Association 67.337, pp. 140-147.
Bartz-Beielstein, T., C. Doerr, D. van den Berg, J. Bossek, S. Chandrasekaran, T. Eftimov, A. Fis- chbach, P. Kerschke, W. La Cava, M. Lopez-Ibanez, et al. (2020). "Benchmarking in optimization: Best practice and open issues". In: arXiv preprint arXiv:2007.03488.
Beck, A. and M. Teboulle (2009). "A fast iterative shrinkage-thresholding algorithm for linear inverse problems". In: SIAM J. Imaging Sci. 2.1, pp. 183-202.
Bello, I., W. Fedus, X. Du, E, D. Cubuk, A. Srinivas, T.-Y. Lin, Shlens, and B. Zoph (2021). "Re- visiting ResNets: Improved Training and Scaling Strategies". In: Advances in Neural Information Processing Systems.
Bergstra, J. and Y. Bengio (2012). "Random search for hyper-parameter optimization". In: J. Mach. Learn. Res. 13.2.
Bergstra, J., D. Yamins, and D. Cox (2013). "Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures.. In: ICML. Vol. 28. 1, pp. 115- 123.
Bertin-Mahieux, T., D. P. Ellis, B. Whitman, and P. Lamere (2011). "The Million Song Dataset". In: Proceediiggs of the 12th International Conference on Music Information Retrieval (ISMIR 2011), l).
Bertrand, Q., Q. Klopfenstein, P.-A. Bannier, G. Gidel, and M. Massias (2022). "Beyond L1: Faster and Better Sparse Models with skglm". In: arXiv preprint arXiv:2204.07826.
Bertsekas, D. (1999). Nonlinear programming. Athena Scientific.
Bezanson, J., A. Edelman, S. Karpinski, and V. B. Shah (2017). "Julia: A fresh approach to numerical computing". In: SIAM Review 59.1, pp. 65-98.
Bleakley, K. and J.-P. Vert (2011). "The group fused Lasso for multiple change-point detection".. In.
Blondel, M. and F. Pedregosa (2016). Lightning: Large-Scale Linear Classification, Regression and Ranking in Python.
Boisbunon, A., R. Flamary, and A. Rukotomamonjy (2014). "Active set strategy for high-dimensional non-convex sparse optimization problems". In: ICASSP. IEEE, pp. 1517-1521.
Bolte, J., S. Sabach, and M. Teboulle (2014). "Proximal alternating linearized minimization for nonconvex and nonsmooth problems". In: Mathematical Programming 146.1, pp. 459-494.
Bonnefoy, A., V. Emiya, L. Ralaivola, and R. Gribonval (2015). "Dynamic screening: accelerating first-order algorithms for the Lasso and Group-Lasso". In: IEEE Trans. Signal Process. 63.19, p. 20.
Bottou, L. (2010). "Large- Scale Machine Learning with Stochastic Gradient Descent". In: COMP- STAT. Physica-Veriee pp. 177-186.
Boyd, S., N. Parikh, E. Chu, B. Peleato, and J. Eckstein (2011). "Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers". In: Foundations and Trends in Machine Learning 3.1.
Boyd, S. and L. Vandenberghe (2004). Convex Optimization. Cambridge, UK New York: Cam- bridge University Press.

---

Boykov, Y., O. Veksler, and R. Zabih (2001). "Fast approximate energy minimization via graph cuts".
In: IEEE Transactions on Pattern Analysis and Machine Intelligence 23.11. pp. 1222-1239.
Brandl, G. (2010). "Sphinx documentation".. In: URL http://sphinx-doc org/sphinx. pdf.
Bredies, K., K. Kunisch, and T. Pock (2010). "Total generalized variation". In: SIAM J. Imaging Sci.
3.3, pp. 492-526.
Breheny, and Huang (2011). "Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection". In: Ann. Appl. Stat. 5.1, p. 232.
Brock, A., S. De. L. Smith, and K. Simonyan (2021). "High-performance large-scale image recognition without normalization". In: ICML. PMLR, pp. 1059-1071.
Bühlmann, P. and S. van de Geer (2011). Statistics for high-dimensional data. Springer Series in Statistics. Methods, theory and applications. Heidelberg; Springer.
Candès, E. J., M. B. Wakin, and S. P. Boyd (2008). "Enhancing Sparsity by Reweighted Minimiza- tion". In: J. Fourier Anal. Applicat. 14.5-6, pp. 877-905.
Chambolle, A. and P.-L. Lions (1997). "Image recovery via total variation minimization and related problems". In: Numerische Mathematik 76.2, pp. 167-188.
Chambolle, A. and T. Pock (2011). "A first-order primal-dual algorithm for convex problems with applications to imaging". In: Journal of Mathematical Imaging and Vision 40.1.
Chen, S. S., D. L. Donoho, and M. A. Saunders (1998). "Atomic decomposition by basis pursuit", In:
SIAM J. Sci. Comput. 20.1, 1, pp. 33-61.
Cherkaoui, H., T. Moreau, A. Halimi, and P. Ciuciu (2019). "Sparsity-Based Semi-Blind Decon- volution of Neural Activation Signal in fMRI". In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Brighton, UK.
Chollet, F. et al. (2015). Keras.
Clark, A. (2015). Pillow (PIL Fork) Documentation.
Combettes, P. L. and V. Wajs (2005). "Signal recovery by proximal forward-backward splitting". In: Multiscale modeling & simulation 4.4, pp. 1168 1200.
Combettes, P. L. and L. E. Glaudin (2021). "Solving Composite Fixed Point Problems with Block Updates". In: Advances in Nonlinear Analysis.
Condat, L. (2013a). "A Direct Algorithm for 1-D Total Variation Denoising". In: IEEE SIGNAL PROCESSING LETTERS 20.12.
Condat, L. (2013b). "A primal-dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms". In: Journal of Optimization Theory and Applications, Springer Verlag.
Criteo-Labs (2015). "Criteo releases industry's largest-ever dataset for machine learning to academic community". In.
Daubechies, I. M. Defrise, and C. De Mol (2004). "An iterative thresholding algorithm for linear inverse problems with a sparsity constraint". In: Commun. Pure Appl. Math. 57.11, pp. 1413-1457.
Defazio, A., F. Bach, and S. Lacoste-Julies Julien (2014), "SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives".. In: Advances in Neural Information Processing Systems. Vol. 28, pp. 1646-1654.
Dembo, R. S., S. C. Eisenstat, and T. Steihaug (1982). "Inexact Newton Methods". In: SIAM J. Numer.
Anal. 19.2, pp. 400-408.
Dosovitskiy, A., L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021). "An Image is Worth Transformers for Image Recognition Scale". In: ICLR.
16x 16 Words:
12

---

Dünner, C., T. Parnell, D. Sarigiannis, N. Ioannou, A. Anghel, G. Ravi, M. Kandasamy, and H. Pozidis (2018). "Snap ML: A hierarchical framework for machine learning.. In: Advances in Neural Information Processing Systems. Vol. 31.
Efron, B., J. J. Hastie, I. M. Johnstone, and R. Tibshirani (2004). "Least angle regression". In: Ann. Statist. 32.2. With discussion, and a rejoinder by the authors, pp. 407-499.
EI Ghaoui, L., V. Viallon, and Rabbani (2012). "Safe feature elimination in sparse supervised learning". In: J. Pacific Optim. 8.4, pp. 667-698.
Elad, M., P. Milanfar, and R. Rubinstein (2006). "Analysis versus synthesis in signal priors", In: 2006 14th European Signal Processing Conference.
Falcon, W., Borovec, A. Wälchli, N. Eggent, Schock, J. Jordan, N. Skafte, IrldXD, V. Bereznyuk, E. Harris, T. Murrell, P. Yu, S. Præsius, T. Addair, J. Zhong, D. Lipin, S. Uchida, S. Bapat, H. Schröter, B. Dayma, A. Karnachev, A. Kulkarmi, Komatsu, Martin.B, J.-B. SCHIRATTI, H. Mary, D. Byrne, C. Eyzaguirre, cinjon, and A. Bakhtin (May 2020). PyTorchLighhiing/ytorrhh lightning: 0.7.6 release. Version 0.7.6.
Fan, J. and R. Li (2001). "Variable selection via nonconcave penalized likelihood and its oracle properties". In: J. Amer. Statist. Assoc. 96.456, pp. 1348-1360.
Fan, R.-E., K.-W. Chang. C.-J. Hsieh, X.-R. Wang, and C.-J. Lin (2008). "LIBLINEAR: A Library for Large Linear Classification". In: Mach. Learn. Res. 9.
Farrens, S., A. Grigis, L. EI Gueddari, Z. Ramzi, G. Chaithya, S. Starek, B. Sarthou, H. Cherkaoui,
P. Ciuciu, and J.-L. Starck (2020). "PySAP: Python Sparse Data Analysis Package for multidisci- plinary image processing". In: Astronomy and Computing 32. p. 100402.
Fletcher, R. and C. M. Reeves (Jan. 1964). "Function Minimization by Conjugate Gradients", In: The Computer Journal 7.2, pp. 149 154.
Forde, J. T. Head, C. Holdgraf, Y. Panda, G. Nalvarete, B. Ragan Kelley, and E. Sundell (2018). Reproducible research environments with repo2docker. Tech. rep.
Ragan-Kelley,
Friedman, J., T. Hastic, and Tibshirani (2010). "Regularization paths for generalized linear models via coordinate descent". In: J. Stat. Softw. 33.1, pp. 1-22.
Gao, F. and L. Han (2012). "Implementing the Nelder-Mead simplex algorithm with adaptive parameters". In: Computational Optimization and Applications 51.1, pp. 259-277.
Glorot, X. and Y. Bengio (2010). "Understanding the difficulty of training deep feedforward neural networks" In: AISTATS. Vol. 9, pp. 249-256.
Golub, T. R., D. K. Slonim, P. Tamayo, C. Huard, M. Gaasenbeek, J. P. Mesirov, H. Coller, M. L.. Loh, J. R. Downing, M. A. Caligiuri, et al. (1999). "Molecular classification of cancer: class discovery and class prediction by gene expression monitoring". In: science 286.5439, pp. 531-537.
Gong, P., C. Zhang, Z. Lu, J. Huang, and J. Ye (2013). "A general iterative shrinkage and thresholding algorithm for non-convex regularized optimization problems". In: ICML, pp. 37-45.
Gramfort, A., M. Luessi, F. Larson, D. A. Engemann, D. Strohmeier, C. Brodbeck, I.. Parkkonen, and
M. S. Hämäläänen (2014). "MNE software for processing MEG and EEG data". In: Neuroimage 86, pp. 446-460.
Grandvalet, Y. (1998). "Least absolute shrinkage is equivalent to quadratic penalization". In: Interna- tional Conference on Artificial Neural Networks. Springer, pp. 201-206.
Guyon, I., S. Gunn, A. Ben-Hur, and G. Dror (2004). "Result analysis of the nips 2003 feature selection challenge". In: Advances in neural information processing systems 17.
Hansen, N., A. Auger, R. Ros, O. Mersmann, T. Tušar, and D. Brockhoff (2021). "COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting". In: Optimization Methods and Software 36.1. ArXiv e-prints, arXiv:1603.08785 pp. 114-144.
Hansen, N. and A. Ostermeier (2001). "Completely derandomized self-adaptation in evolution strategies". In: Evolutionary computation 9.2, pp. 159-195.
13

---

Harris, C. R., K. J. Millman, S. J. Van Der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, Taylor, Berg, N. Smith, et al. (2020). "Array programming with NumPy". In: Nature pp- 357-362.
Hastie, J.J., Tibshirani, and Wainwright (2015). Statistical Learning with Sparsity: The Lasso and Generalizations, CRC Press.
He, K., X. Zhang, Ren, and J. Sun (2015). "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification" In: CVPR, pp. 1034-1
He, K., X. Zhang. Ren, and Sun (2016). "Deep residual learning for image recognition". In: CVPR.
He, T., Z. Zhang, H. Zhang, Z. Zhang. J. Xie, and M. Li (2019). "Bag of tricks for image classification with convolutional neural networks". In: CVPR. Vol. 2019-June, pp.
Hunter, J. (2007). "Matplotlib: 2D graphics environment". In: Computing Science Engi- neering 9.3, pp. 90-95.
Hutson, M. (2018). "Artificial intelligence faces reproducibility crisis", In: Science 359.6377, pp. 725- 726.
Inc., P. T. (2015). Collaborative data science. URL: https://plot. 1y.
loffe, S. and C. Szegedy (2015). "Batch normalization: Accelerating deep network training by reducing internal covariate shift". In: ICMI, pp. 448-456.
Johnson, R. and T. Zhang (2013). "Accelerating Stochastic Gradient Descent Using Predictive
Variance Reduction". In: Advances in Neural Information Processing Systems. Vol. 26.
Johnson, T. B. and C. Guestrin (2015). "Blitz: Principled Meta-Algorithm for Scaling Sparse Optimization". In: ICML. Vol. 37, pp. 1171-1179.
Karahanoglu, I., C. Caballero Caballero-Gaudes, Lazeyras, and D. Van De Ville (June 2013). "Total Activation: fMRI Deconvolution through Spatio-Temporal Regularization". In: Neurolmage 73, pp. 121-134.
Keerthi, S. S., D. DeCoste, and T. Joachims (2005). "A modified finite Newton method for fast solution of large scale linear SVMs." In: Journal of Machine Learning Research 6.3.
Kiefer, J. and J. Wolfowitz (1952). "Stochastic estimation of the maximum of regression function". In: Ann. Math. Stat., pp. 462-466.
Kingma, D. P. and J. Ba (2015). "Adam: A Method for Stochastic Optimization", In: ICLR, pp. 1-10.
Koh, K., J. Kim, and Boyd (2007). "An interior-point method for large-scale I1-regularized logistic regression." " In: J. Mach. Learn. Res. 8.8, pp. 1519-1555.
Kolmogorov, V. and R. Zabin (2004). "What energy functions can be minimized via graph cuts?" In: IEEE Transactions Pattern Analysis and Machine Intelligence 26.2. pp. 147-159.
Komodakis, N. and J.-C. Pesquet (2015). "Playing with Duality: An overview of recent primal-dual approaches for solving large-scale optimization problems". In: IEEE Signal Processing Magazine, Institute of Electrical and Electronics Engineers.
Komblith, S. (Oct. 28, 2021). Lasso.jl. Version 0.6.3. JuliaStats.
Krizhevsky, A. (2009). Learning multiple layers of features from tiny images. Tech. rep.
Krizhevsky, A., I. Sutskever, and G. F. Hinton (2012). "Imagenet Classification with Deep Convo- lutional Neural Networks". In: Advances Neural Information Processing Systems. South Lake Tahoe, NV, USA, pp. 1097-1105.
Lalanne, C., M. Rateaux, Oudre, M. P. Robert, and Moreau (July 2020). "Extraction of Nystagmus Patterns from Eye-Tracker Data with Convolutional Sparse Coding". In: Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Montreal, QC, Canada: IEEE, pp. 928-931.
14

---

LeCun, Y., C. Cortes, and C. Burges (2010). "MNIST handwritten digit database". In: ATT Labs http:///ww..cccm.co/////bbiiiaee
[Online]. Available:
Lewis, D. D., Y. Yang. T. Russell-Rose, and F. Li (2004). "Revl: A new benchmark collection for text categorization research". In: Journal of machine learming research 5.Apr, pp. 361-397.
Liang, J. T. Luo, and C.-B. Schöenlieb (2022). "Improving "Fast Iterative Shrinkage-Thresholoeee Thresholding Algorithm": Faster, Smarter, and Greedier". In: SIAM J. Sci. Comput. 44.3, 11069-A1099
Liu. D. C. and J. Nocedal (Aug. 1989). "On the Limited Memory BFGS Method for Large Scale Optimization". In: Math. Program. 45.1-3, pp. 503-528.
Liu, Z., H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie (2022). "A ConvNet for the 2020s". In: CVPR.
Loshchilov, I. and F. Hutter (2017). "SGDR: Stochastic Gradient Descent with Warm Restarts". In: ICLR.
Loshchilov, I. and F. Hutter (2019). "Decoupled Weight Decay Regularization". In: ICLR.
Lueckmann, J.-M., J. Boelts, D. S. Greenberg, P. J. Gonçalves, and J. H. Macke (2021). "Benchmark- ing Simulation- Inference". In: AISTATS. Vol. 130. PMLR, pp. 343-351.
Mairal, J. and B. Yu (2012). "Complexity analysis of the Lasso regularization path". In: ICML, pp. 353-360.
Mairal, J. (2019). "Cyanure: An open-source toolbox for empirical risk minimization for python, c++, and soon more". In: arXiv preprint 1912.08
Massias, M., A. Gramfort, and J. Salmon (2018). "Celer: fast solver for the lasso with dual extrapolation". In: ICML, pp. 3315-3324.
Mattson, P., V. Reddi, C. Cheng, C. Coleman, G. Diamos, D. Kanter, P. Micikevicius, D. Patterson, G. Schmuelling, H. Tang, et al. (2020). "MLPerf: An industry standard benchmark suite for machine learning performance". In: IEEE Micro 40.2, pp. 8-16.
Mazumder, R., J. H. Friedman, and T. Hastie (2011), "Sparsenet: Coordinate descent with nonconvex penalties". In: Amer. Statist. Assoc. 106.495, pp. 1125-1138.
McKinney, W. et al. (2010). "Data structures for statistical computing in python". In: Proceedings of the 9th Python in Science Conference. Vol. 445. Austin, TX. pp. 51-56.
Moreau, T., P. Glaser, R. Yurchak, and O. Grisel (June 2017). Loky. Version 3.0. URL: https: //github. com/jobbib/loky.
Nájera, Ó., E. Larson, L. Estève, G. Varoquaux, L. Liu, J. Grobler, E. S. de Andrade, C. Holdgraf, A. Gramfort, M. Jas, J. Nothman, O. Grisel, N. Varoquaux, E. Gouillart, M. Luessi, A. Lee, Vanderplas, T. Hoffmann, T. A. Caswell, B. Sullivan, A. Batula, jaeilepp, T. Robitaille, S. Appelhoff. P. Kunzmann, M. Geier, Lars, K. Sunden, D. Stanezak, and A. Y. Shih (May 2020).
saliinegellleysshhin-ggller Release v0.7.0. Version v0.7.0.
Ndiaye, E., O. Fercoq, A. Gramfort, and J. Salmon (2017). "Gap Safe screening rules for sparsity enforcing penalties". In: J. Mach. Learn. Res. 18.128, pp. 1-33.
Nesterov, Y. (1983). "A method for solving the convex programming problem with convergence rate O(1/k2)". In: Dokl. akad. nauk Sesr. Vol. 269, pp. 543 547.
Netzer, Y., T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng (2011). "Reading Digits in Natural Images with Unsupervised Feature Learning.. In: Advances in Neural Information Processing Systems.
Paszke, A., S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. Vetooo, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala (2019). "PyTorch: An Imperative Style, High-Performance Deep Learning Library". In: Advances in Neural Information Processing Systems, pp. 8024-8035.
Pedregosa, F., G. Negiar, and G. Dresdner (2020). "copt: composite optimization in Python". In.
15

---

Pedregosa, F., G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, et al. (2011). "Scikit-leam: Machine learning in Python". In: J. Mach. Learn. Res. 12, pp. 2825-2830.
Pineau, J., Vincent-Lamarre, Sinha, V. Larivière, A. Beygelzimer, d' A'AlhhhéBBu, E. Fox, and H. Larochelle (2021). "Improving reproducibility in machine learning research: report from the NeurIPS 2019 reproducibility program". In: J. Mach. Learn. Res. 22.
Pineau, J., K. Sinha, G. Fried, R. N. Ke, and H. Larochelle (2019). "ICLR reproducibility challenge 2019". In: ReScience C 5.2, p.
Poon, C. and G. Peyré (2021). "Smooth Bilevel Programming for Sparse Regularization". In: Ad-
vances Neural Information Processing Systems. Vol. 34, pp. 1543-1555.
Powell, M. J. (1964). "An efficient method for finding the minimum of a function of several variables without calculating derivatives". In: The computer journal 7.2, pp. 155-162.
Press, W. H., S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery (2007). Numerical recipes 3rd edition: The art of scientific computing. Cambridge university press.
Prokhorov, D. (2001). "UCNN 2001 neural network competition". In: Slide presentation in IJCNN 1.97, 38.
Quemener, E. and M. Corvellec (2013). "SIDUS-thh solution for extreme deduplication of an operating system". In: Linux Journal 2013.235, p.
Core Team (2017). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. Vienna, Austria.
Raff, E. (2019). "A step toward quantifying independently reproducible machine learning research".
In: Advances in Neural Information Processing Systems. Vol. 32, pp. 5486-5496.
Rapin, J. and O. Teytaud (2018). Nevergrad A gradient-free optimization platform. https: /
Gi Git. www.cccm////eeeee http/FFceboooeeeeeec//eeeeeeee
Raschka, S., J. Patterson, and C. Nolet (2020). "Machine "Machine Learning in Python: Main Developments and Technology Trends in Data Science, Machine Learning, and Artificial Intelligence". In: Information-an International Interdisciplinary Journal 11.4, 193.
Robbins, H. and S. Monro (1951). "A stochastic approximation method". In: Ann. Math. Stat., pp. 400-407.
Rodola, G. (2016). "Psutil package: cross-platform library for retrieving information running processes and system utilization". In: Google Scholar.
Rougier, N. P. and K. Hinsen (2018). "ReScience C: a journal for reproducible replications in compu- tational science". In: International Workshop on Reproducible Research in Pattern Recognition. Springer, pp. 150-156.
Rudin, L. I. S. Osher, and E. Fatemi (1992). "Nonlinear total variation based noise removal algo- rithms". In: Physica D: Nonlinear Phenomena 60. pp. 259-268.
Sardy, S., A. G. Bruce, and P. Tseng (2000). "Block coordinate relaxation methods for nonparametric wavelet denoising". In: J. Comput. Graph. Stat. 9.2, pp. 361-379.
Schmidt, M., N. Le Roux, and F. Bach (2017). "Minimizing Finite Sums with the Stochastic Average Gradient". In: Math. Program.
pp. 83-112.
Schmidt, R. M., Schneider, and P. Hennig (2021). "Descending through a Crowded Valley Benchmarking Deep Learning Optimizers". In: ICML. Vol. 139. PMLR. pp. 9367-9376.
Sculley, D., J. Snock, A. Wiltschko, and A. Rahimi (2018). "Winner's curse? On pace, progress, and empirical rigor". In.
Silva, T. S. (2019). "How to Add Regularization to Keras Pre-trained Models the Right Way". In: htwtsthhllllsssg....ii.
16

---

Simonyan, K. and A. Zisserman (2015). "Very Deep Convolutional Networks for Large-Scall Scale Image Recognition". In: ICLR.
Sra, S., S. Nowozin, and J. Wright (2012). Optimization for machine learning. MIT Press.
Tan, M. and Q. Le (2019). "Efficientnet: Rethinking model scaling for convolutional neural networks". In: ICML, pp. 6105 61.
Tibshirani, R., Bien, J. Friedman, T. Hastie, N. Simon, J. Taylor, and R. Tibshirani (2012).
"Strong rules for discarding predictors in lasso-type problems". In: J. R. Stat. Soc. Ser. B Stat. Methodol. 74.2, pp. 245-266.
Tibshirani, R. (1996). "Regression shrinkage and selection via the lasso". In: J. R. Stat, Soc. Ser. B Stat. Methodol. 58.1, pp. 267-288.
Tibshirani, R. J. (Feb. 2014). "Adaptive Piecewise Polynomial Estimation via Trend Filtering". In: The Annals of Statistics 42.1.
Tibshirani, R. and J. Taylor (2011). 1). "The solution path of the generalized lasso". In: Ann. Statist. 39.3, pp. 1335-1371.
Tseng. P. (1993). "Dual coordinate ascent methods for non-strictly convex minimization". In: Math. Program. 59.1, pp. 231-247.
Vanschoren, J., J. van Rijn, B. Bischl, and L. Torgo (2013). "OpenML: networked science in machine
learning". In: SIGKDD Explorations 15.2, pp. 49-60.
Virtanen, P., R. Gommers, T. F. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, I. Polat, Y. Feng, E. W. Moore, VanderPlas, Laxalde, Perktold, Cimrman, I. Henriksen, E. A. Quintero, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors (2020). "SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python". In: Nature Methods 17, pp. 261-272.
R. Harris,
Wales, J. and P. Doye (1997). "Global optimization by basin-hopping and the lowest energy clusters containing up to 110 atoms". In: The Journal of Physical
structures of Lennard-Jow Chemistry 101.28. pp. 51 5111-5116.
Wightman, R., H. Touvron, and H. Jégou (2021), ResNet strikes back: An improved training procedure in timm. Tech, rep., pp. 1-22.
Wright, S. and J. Nocedal (1999). Numerical Optimization. Science Springer.
Xie, S., R. Girshick, P. Dollár, Z. Tu, and K. He (2017). "Aggregated residual transformations for deep neural networks". In: CVPR, pp. 1492-1500.
Zhang, (2010a). "Nearly unbiased variable selection under minimax concave penalty". In: Ann. Statist. 38.2, pp. 894-942.
Zhang, M., Lucas, J. Ba, and G. F. Hinton (2019). "Lookahead optimizer: k steps forward, step back". In: Advances in Neural Information Processing Systems. Vol. 32.
Zhang, T. (2010b). "Analysis of multi-stage convex relaxation for sparse regularization". In: J. Mach. Learn. Res. 11.Mar, pp. 1081-1107.
Zheng, Y., R. Zhang, and Y. Mao (2021). "Regularizing neural networks via adversarial model perturbation". In: CVPR, pp. 8156-8165.
17

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] These are specified in Appendix, a per-benchmark basis.
(c) Did you report error bars (e.g.. with respect the random seed after running experiments multiple times)? [No] Error bars are not reported for clarity, but Benchopt allows this in particular html versions of the plots that can be found in https: /bbechop/ gi thub. io/resul ts/prepring perrerrnttresullee ts html.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] In the introduction.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
18


Reviews:
[{"review_id": "kC-kEwiVMwk", "reviewer": "Reviewer_4Y29", "summary": "A python-based toolbox for benchmarking optimization methods, with special focus on machine learning applications, is presented in this paper. The modular design of the toolbox allows a flexible assessment, to account for the impact that rapid development of software and hardware has in the field. Moreover, although the benchmarking-suite is written in python, it allows the comparison of methods written in other programming languages such as julia. To exemplify the usage of this toolbox, the paper compares several methods for three widely-known problems in the machine learning and deep learning community (logistic regression with l2 regularization, Lasso and the paradigmatic case of ResNet). This case study illustrates the benefits of the proposed approach and manage to make some novel discoveries. \n", "questions": "- How does the toolbox determine that an algorithm has converged? Does this criteria vary with the type of problem (L2LogReg, lasso, ResNet, etc.)?\n\n- Regarding the alignment of TensorFlow and PyTorch for ResNet18 training, does the toolbox offer some built-in way of simplifying this process or just a series of recommendations on how to do it? Is it possible that two different researchers deal with the alignment in two different ways when comparing TensorFlow and PyTorch implementations, rendering the comparisons invalid?\nI believe that prospective users could benefit from a way of simplifying the process of “alignment” between those two frameworks that is as transparent as possible to the user, and not just guidelines.  For example, is it possible to implement “adaptive layers” to be added to the network architecture with the purpose of dealing with some of the aspects described by authors in section E.2? This would leave less important details of the benchmarking process free to the user.\n", "limitations": "I believe authors should mention other toolboxes with similar purpose and explicitly mention the differences between their approach and the currently available tools.\n\nTo my mind, authors could also discuss more about the limitations of the alignment process between frameworks.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Positive remarks:\n- The modular design of the benchmark guarantees the versatility that such a tool requires to meet the community needs.\n- Interesting findings for future research (that is what benchmarks are supposed to be used for!):\n\t- For the l2-regularized logistic regression, recommendations on the use of larger datasets to explore the stochastic solvers.\n\t- For Lasso, assessments include julia-based methods for the first time in a systematic comparison. Additionally, other objectives, like the sparsity of the solution, can be easily analyzed as well.\n\t- For ResNet 18 training, it is shown that reaching the state-of-the-art for this architecture is more dependent on parameters such as the data augmentation strategy and weight decay/learning rate adjustments.\n\nThe originality of the article is limited by the fact that there are similar approaches within this area of research. Nevertheless authors show an interesting proposal of a toolbox mainly oriented to the ML community. The presentation of the proposed toolbox is acceptable, but there are some points that, to my view, require clarification.  I believe that the analysis of the state-of-the-art of optimization benchmarks should include similar toolboxes like MLPerf, COCO [1], Nevergrad [2] or FitBenchmarking [3], and explicitly describe the differences between this toolbox and those approaches. Otherwise, authors risk not highlighting the significance of their work.\n\n[1]Hansen, Nikolaus, et al. \"COCO: A platform for comparing continuous optimizers in a black-box setting.\" Optimization Methods and Software 36.1 (2021): 114-144.\n\n[2] https://github.com/facebookresearch/nevergrad\n\n[3] Markvardsen et al., (2021). FitBenchmarking: an open source Python package comparing data fitting software. Journal of Open Source Software, 6(62), 3127,\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TuLSO02M2qj", "reviewer": "Reviewer_nYXq", "summary": "The authors provide and describe a library, Benchopt, which can be used to benchmark optimizers.  The authors demonstrate and discuss the results of three benchmark models on a number of tasks: l_2 regularized logistic regression, Lasso, and ResNet-18.  ", "questions": "- Could you explain in more detail the design decisions underlying the library?  How should it be used?  Does the library make strong design choices as to how a set of optimizers should be evaluated?\n\n- Can you discuss further in detail how a researcher using Benchopt should deal with the possibility that one could draw different conclusions when comparing methods based on how the benchmarking experiments are set up?  Similarly, how researchers can evaluate optimizers for multiple, possibly opposing, criteria such as accuracy and efficiency/computational cost?\n\n- You note that implementations across libraries may have notable differences in how the optimizer is designed (e.g. Pytorch vs Tensorflow).  Who's responsibility in the Benchopt ecosystem is it to deal with these issues? The Benchopt maintainers? Contributors? Researcher users?  ", "limitations": "See the discussion of the weaknesses in the limitations.  In particular, the technical and methodological difficulties regarding how one draws conclusions when comparing optimizers could be further emphasized, as conclusions are not always clear and can be contradictory.  The citations provided in the weaknesses section are some of the relevant prior work in this area.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The results and methods are clearly presented.  The paper overall is easy to read and understand.\n\n- The authors present a broad set of experiments across a variety of tasks, demonstrating the applicability of the library.\n\n- The authors identify an important problem in machine learning, the benchmarking and comparisons of optimizers.\n\n- The authors provide a good discussion of related work.\n\nWeaknesses:\n\n- The organization of the paper could be improved.  The discussion of the results and the specific experiments, to me, are not dramatically different compared to similar optimizer benchmarking papers e.g.:\n\nhttps://arxiv.org/abs/2206.05985\n\nhttps://proceedings.neurips.cc/paper/2021/hash/17fafe5f6ce2f1904eb09d2e80a4cbf6-Abstract.html\n\nhttps://arxiv.org/abs/1910.05446\n\nhttps://ieeexplore.ieee.org/document/9138989\n\nhttp://proceedings.mlr.press/v139/schmidt21a.html\n\nhttp://proceedings.mlr.press/v119/sivaprasad20a.html\n\nAs a result, the presentation and discussion of the results alone are not sufficient to demonstrate the utility of the library.  In my opinion, section B in the appendix provides further technical details about the library that suggest its broad applicability and ease of use.  Further emphasis on the specific technical aspects of the library would further strengthen the argument for the uniqueness of Benchopt.\n\n- The experiments specifically used do not address the concerns regarding optimizer benchmarking identified by the authors in the introduction:\n\n1. \" there are no clear community guidelines on how to perform, share, and publish benchmarks\" - these aspects of benchopt's design are discussed in passing in section 2, but the specific system details and the motivation for these choices is not discussed in detail.  Again, Appendix B provides some insight into these design choices.\n\n2. \"many others fall short in this regard due to lack of time and resources, and in turn feature results that are hard to reproduce by other researchers\" - The experiments presented, while broad, are relatively computationally cheap.  Moreover, the authors do not make clear why these results are specifically more reproducible than other optimizer benchmarks, as many of these other papers have provided code.  In fact, the limited discussion regarding the inability to reproduce the benchmarks provided by Papers with Code, leaves the reader wondering why that benchmark is not reproducible. \n\n- The authors note that \"for practical evaluation, the devil is in the details.\"  Prior work (including the citations mentioned above) has also emphasized this observation.  However, the authors presentation of comparisons across methods is relatively straightforward, in that the authors often provide one specific way to evaluate a task as opposed to multiple, possibly differing ways of performing evaluation, such as with broader ablations or hyperparameter searches.  Works such as Choi et al, Cooper et al, Daulton et al, Sivaprasad et al, and Schmidt et al have noted that *how* comparisons are made leads to different conclusions.  While the means by which these comparisons are made in Benchopt may be left to the user, this detail could be further emphasized in the methods and discussion.\n\n----\nOverall impression:\n\nI think this work is relevant to the NeurIPS community and would be of use.  The weaknesses in the paper are largely aesthetic, as they are not with respect to the library in itself.  I believe these limitations are surmountable, and that the paper would benefit by being addressed.  \n\n----\n\nUpdate post discussion period:\n\nI appreciate the discussion points of the authors and again believe that the adoption of tools such as BenchOpt would improve the level of rigor within the NeurIPS research community.  I am raising my score to an accept to reflect these events.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B1UCLqMs_8", "reviewer": "Reviewer_Fv41", "summary": "This paper introduced Benchopt, a collaborative benchmark suite for optimization algorithms in machine learning. The functionality is showcased on three experimental settings.", "questions": "My questions are embedded in the Strengths/Weaknesses Section above.", "limitations": "Since no new methodology has been proposed, I don't see a need for extensive discussion of limitations. Some of the weaknesses listed above constitute (in my opinion) limitations of the benchmark itself. I would encourage the authors to comment on them in the rebuttal and possibly add explanations/clarifications to the paper.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. I commend the effort to improve transparency and reproducibility in OptML research. This is much needed!\n\n2. Designing the Benchmark as a collaborative effort from the start is definitely great. I think the design of the benchmark will make it very easy for researchers to contribute.\n\n### Weaknesses\n\n1. I think it is a bad choice to report comparisons in terms of wall-clock-time. This can be extremely dependent on specific combinations of framework, implementation and hardware. Since the purpose of the BenchOpt is supposedly to compare optimization methods and not specific implementations, I would try to abstract away as much as possible these dependencies. One possibility would be to perform comparisons in terms of number of iterates and report (estimated) time per iteration alongside (or group algorithms roughly according to ther per-iteration complexity).\n\n2. A key quality of optimization methods, at least in deep learning, is ease of hyperparameter tuning. From the main text of the paper, it is not clear what hyperparameter tuning protocol has been employed. Hyperparameter sensitivities should definitely be part of the \"output\" of a benchmark suite like the one proposed here. Are there any plans for this?\n\n3. On thing that irritates me is that the choice of the weight decay parameter as well as data augmentation techniques are considered as part of the optimization method. In principle, these are part of the optimization objective. What led the authors to consider this part of the optimizer instead?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept