Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors provide and describe a library, Benchopt, which can be used to benchmark optimizers.  The authors demonstrate and discuss the results of three benchmark models on a number of tasks: l_2 regularized logistic regression, Lasso, and ResNet-18.  
strengths_and_weaknesses: Strengths:

- The results and methods are clearly presented.  The paper overall is easy to read and understand.

- The authors present a broad set of experiments across a variety of tasks, demonstrating the applicability of the library.

- The authors identify an important problem in machine learning, the benchmarking and comparisons of optimizers.

- The authors provide a good discussion of related work.

Weaknesses:

- The organization of the paper could be improved.  The discussion of the results and the specific experiments, to me, are not dramatically different compared to similar optimizer benchmarking papers e.g.:

https://arxiv.org/abs/2206.05985

https://proceedings.neurips.cc/paper/2021/hash/17fafe5f6ce2f1904eb09d2e80a4cbf6-Abstract.html

https://arxiv.org/abs/1910.05446

https://ieeexplore.ieee.org/document/9138989

http://proceedings.mlr.press/v139/schmidt21a.html

http://proceedings.mlr.press/v119/sivaprasad20a.html

As a result, the presentation and discussion of the results alone are not sufficient to demonstrate the utility of the library.  In my opinion, section B in the appendix provides further technical details about the library that suggest its broad applicability and ease of use.  Further emphasis on the specific technical aspects of the library would further strengthen the argument for the uniqueness of Benchopt.

- The experiments specifically used do not address the concerns regarding optimizer benchmarking identified by the authors in the introduction:

1. " there are no clear community guidelines on how to perform, share, and publish benchmarks" - these aspects of benchopt's design are discussed in passing in section 2, but the specific system details and the motivation for these choices is not discussed in detail.  Again, Appendix B provides some insight into these design choices.

2. "many others fall short in this regard due to lack of time and resources, and in turn feature results that are hard to reproduce by other researchers" - The experiments presented, while broad, are relatively computationally cheap.  Moreover, the authors do not make clear why these results are specifically more reproducible than other optimizer benchmarks, as many of these other papers have provided code.  In fact, the limited discussion regarding the inability to reproduce the benchmarks provided by Papers with Code, leaves the reader wondering why that benchmark is not reproducible. 

- The authors note that "for practical evaluation, the devil is in the details."  Prior work (including the citations mentioned above) has also emphasized this observation.  However, the authors presentation of comparisons across methods is relatively straightforward, in that the authors often provide one specific way to evaluate a task as opposed to multiple, possibly differing ways of performing evaluation, such as with broader ablations or hyperparameter searches.  Works such as Choi et al, Cooper et al, Daulton et al, Sivaprasad et al, and Schmidt et al have noted that *how* comparisons are made leads to different conclusions.  While the means by which these comparisons are made in Benchopt may be left to the user, this detail could be further emphasized in the methods and discussion.

----
Overall impression:

I think this work is relevant to the NeurIPS community and would be of use.  The weaknesses in the paper are largely aesthetic, as they are not with respect to the library in itself.  I believe these limitations are surmountable, and that the paper would benefit by being addressed.  

----

Update post discussion period:

I appreciate the discussion points of the authors and again believe that the adoption of tools such as BenchOpt would improve the level of rigor within the NeurIPS research community.  I am raising my score to an accept to reflect these events.  

questions: - Could you explain in more detail the design decisions underlying the library?  How should it be used?  Does the library make strong design choices as to how a set of optimizers should be evaluated?

- Can you discuss further in detail how a researcher using Benchopt should deal with the possibility that one could draw different conclusions when comparing methods based on how the benchmarking experiments are set up?  Similarly, how researchers can evaluate optimizers for multiple, possibly opposing, criteria such as accuracy and efficiency/computational cost?

- You note that implementations across libraries may have notable differences in how the optimizer is designed (e.g. Pytorch vs Tensorflow).  Who's responsibility in the Benchopt ecosystem is it to deal with these issues? The Benchopt maintainers? Contributors? Researcher users?  
limitations: See the discussion of the weaknesses in the limitations.  In particular, the technical and methodological difficulties regarding how one draws conclusions when comparing optimizers could be further emphasized, as conclusions are not always clear and can be contradictory.  The citations provided in the weaknesses section are some of the relevant prior work in this area.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: A python-based toolbox for benchmarking optimization methods, with special focus on machine learning applications, is presented in this paper. The modular design of the toolbox allows a flexible assessment, to account for the impact that rapid development of software and hardware has in the field. Moreover, although the benchmarking-suite is written in python, it allows the comparison of methods written in other programming languages such as julia. To exemplify the usage of this toolbox, the paper compares several methods for three widely-known problems in the machine learning and deep learning community (logistic regression with l2 regularization, Lasso and the paradigmatic case of ResNet). This case study illustrates the benefits of the proposed approach and manage to make some novel discoveries. 

strengths_and_weaknesses: Positive remarks:
- The modular design of the benchmark guarantees the versatility that such a tool requires to meet the community needs.
- Interesting findings for future research (that is what benchmarks are supposed to be used for!):
	- For the l2-regularized logistic regression, recommendations on the use of larger datasets to explore the stochastic solvers.
	- For Lasso, assessments include julia-based methods for the first time in a systematic comparison. Additionally, other objectives, like the sparsity of the solution, can be easily analyzed as well.
	- For ResNet 18 training, it is shown that reaching the state-of-the-art for this architecture is more dependent on parameters such as the data augmentation strategy and weight decay/learning rate adjustments.

The originality of the article is limited by the fact that there are similar approaches within this area of research. Nevertheless authors show an interesting proposal of a toolbox mainly oriented to the ML community. The presentation of the proposed toolbox is acceptable, but there are some points that, to my view, require clarification.  I believe that the analysis of the state-of-the-art of optimization benchmarks should include similar toolboxes like MLPerf, COCO [1], Nevergrad [2] or FitBenchmarking [3], and explicitly describe the differences between this toolbox and those approaches. Otherwise, authors risk not highlighting the significance of their work.

[1]Hansen, Nikolaus, et al. "COCO: A platform for comparing continuous optimizers in a black-box setting." Optimization Methods and Software 36.1 (2021): 114-144.

[2] https://github.com/facebookresearch/nevergrad

[3] Markvardsen et al., (2021). FitBenchmarking: an open source Python package comparing data fitting software. Journal of Open Source Software, 6(62), 3127,

questions: - How does the toolbox determine that an algorithm has converged? Does this criteria vary with the type of problem (L2LogReg, lasso, ResNet, etc.)?

- Regarding the alignment of TensorFlow and PyTorch for ResNet18 training, does the toolbox offer some built-in way of simplifying this process or just a series of recommendations on how to do it? Is it possible that two different researchers deal with the alignment in two different ways when comparing TensorFlow and PyTorch implementations, rendering the comparisons invalid?
I believe that prospective users could benefit from a way of simplifying the process of “alignment” between those two frameworks that is as transparent as possible to the user, and not just guidelines.  For example, is it possible to implement “adaptive layers” to be added to the network architecture with the purpose of dealing with some of the aspects described by authors in section E.2? This would leave less important details of the benchmarking process free to the user.

limitations: I believe authors should mention other toolboxes with similar purpose and explicitly mention the differences between their approach and the currently available tools.

To my mind, authors could also discuss more about the limitations of the alignment process between frameworks.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper introduced Benchopt, a collaborative benchmark suite for optimization algorithms in machine learning. The functionality is showcased on three experimental settings.
strengths_and_weaknesses: ### Strengths

1. I commend the effort to improve transparency and reproducibility in OptML research. This is much needed!

2. Designing the Benchmark as a collaborative effort from the start is definitely great. I think the design of the benchmark will make it very easy for researchers to contribute.

### Weaknesses

1. I think it is a bad choice to report comparisons in terms of wall-clock-time. This can be extremely dependent on specific combinations of framework, implementation and hardware. Since the purpose of the BenchOpt is supposedly to compare optimization methods and not specific implementations, I would try to abstract away as much as possible these dependencies. One possibility would be to perform comparisons in terms of number of iterates and report (estimated) time per iteration alongside (or group algorithms roughly according to ther per-iteration complexity).

2. A key quality of optimization methods, at least in deep learning, is ease of hyperparameter tuning. From the main text of the paper, it is not clear what hyperparameter tuning protocol has been employed. Hyperparameter sensitivities should definitely be part of the "output" of a benchmark suite like the one proposed here. Are there any plans for this?

3. On thing that irritates me is that the choice of the weight decay parameter as well as data augmentation techniques are considered as part of the optimization method. In principle, these are part of the optimization objective. What led the authors to consider this part of the optimizer instead?
questions: My questions are embedded in the Strengths/Weaknesses Section above.
limitations: Since no new methodology has been proposed, I don't see a need for extensive discussion of limitations. Some of the weaknesses listed above constitute (in my opinion) limitations of the benchmark itself. I would encourage the authors to comment on them in the rebuttal and possibly add explanations/clarifications to the paper.
review_title: null

Meta-review: metareview: I thank the reviewers and authors for their work throughout the reviewing process, including the detailed discussion. The paper was borderline after the first round of reviews, but the discussions and my own look at the paper makes the balance tip over to acceptance. 

I encourage the authors to use all information in the discussion threads to improve their manuscript. In particular, I agree with Reviewer Fv41 that the main weakness of the paper is the treatment of hyperparameters. That being said, I understand that comparing optimization algorithms for fixed (or a small set of) hyperparameters is already a useful task for the community, and already a difficult one to automatize. I add a few comments below to the reviews already provided, hoping to see them addressed in the camera-ready version.

### Major comments
* how do you report traces for randomized algorithms, such as SGD? I don't see error bars on the traceplots, are there no repetitions? Did you not observe variance in the traces for different seeds? Even for a deterministic algorithm, one typically randomizes, say, the starting point, especially on nonconvex problems. 
* In the same vein, if a solver means an algorithm and its hyperparameters, there might be a large number of solvers if many people contribute to a benchmark. If the output of each solver is random, ranking the means of all outputs is a parallel statistical testing problem. How do you propose to treat it? I agree that this is a problem for most benchmarks of randomized algorithms, but one that I rarely saw addressed in a systematic manner.

### Minor comments
L53: automatic 
L273: solver's?
L274: all possible combinations
L275: material
L1052: what is "on standard"?
confidence: Less certain
Decision: Accept