Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper introduces a new library for executing and benchmarking hyperparameter optimization algorithms. The library allows for different back-ends: local, cloud, and simulation, and facilitates distributed, asynchronous optimization. The library implements several popular algorithms for suggesting configurations: e.g., random search, Bayesian optimization, TPE, as well as various schedulers for multi-fidelity, transfer, multi-objective, and constrained scenarios. The library also includes a number of benchmarks: FCNet, NAS201, and LCBench.
overall_review: This is a nicely written library that implements many common and contemporary baselines, schedulers, and benchmarks for hyperparameter optimization. Based on the experiments, it appears to be significantly faster than alternative implementations, while also being far more broad and adaptable. I think that this could easily see usage either to facilitate research in the area, or as a tool for launching HPO experiments with existing approaches.

In terms of the baselines (the part that suggests a new configuration) I am a bit torn. This library implements a push model, meaning it is responsible for launching experiments and scheduling them. An alternative is a pull model, where a server is queried for configurations, and the launch details are left to the user. I prefer the latter for getting new configurations, but I appreciate that the push model makes more sense for the implemented schedulers. As long as the local scheduler and cloud schedulers are robust and continue to be maintained, then I don't see this as being an issue.
review_summary: See the Overall Review section.
review_title: null

final_score: rating: 6: Strong accept, should be highlighted
review_content: summary_of_contributions: The paper introduces a library that provides various components necessary for performing hyperparameter optimization, as well as doing benchmarks on HPO methods. The library comes with a variety of HPO-methods that offer multi-fidelity optimization, constrained optimization, multi-objective optimization, and transfer learning capabilities. The library furthermore provides access to established tabular HPO benchmarks. A demonstration run of the included methods on some included benchmarks is presented that show the benefits of multi-fidelity and transfer learning methods.
overall_review: The work is of high quality and explains the the presented library well. The library appears relevant and useful.

A few further comments / questions:
* Has the technique presented in lines 143-149, to the knowledge of the authors, already been used by other work? In that case it should be cited. Otherwise it may be worth stressing that this is an original contribution of the work.
* Typo in caption of Table 1: "a random configurations"

review_summary: The paper is of high quality and presents a library that appears relevant and useful. There are very few clarifications that I would suggest the authors to make.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: The paper describes a hyper-parameter optimization library called SyneTune. 
Different from existing libraries, SyneTune provides multiple methods for HPO like random search, Bayesian opt, and evolutionary search. It additionally supports HPO for transfer learning, constrained optimization and multi-objective optimization. There is also support for benchmarking, similar to HPOBench.
The authors provide minimal code examples for calling their API for HPO (in the main paper and supplementary). The experiments demonstrate HPO results showcasing the above capabilities: comparing multiple algorithms and transfer learning, constrained HPO, and multi-objective HPO.
Working code samples or a python notebook would have made the paper more impactful.
overall_review: Pros:
- Interesting library providing many interesting features for HPO in one place.

Cons:
- There are many existing libraries for HPO already and incentives to switch to a new one can be low.
review_summary: Overall I found the presentation clear, the analysis sufficient and the proposed library interesting to warrant consideration to use it in future HPO work.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper present a library for hyperparameter optimization (HPO) called Syne Tune. This library offers a wide set of HPO algorithms, from more traditional algorithms, to multi-fidelity, optimization with constraints and even transfer learning. The library also provides extensive benchmarking tasks with pre-built tabular datasets of the tasks to speedup benchmarking. The benchmarking system also takes into account the relative timing of trainings to simulate long training tasks and benchmark parallelization of HPO algorithms. The author provide 3 use cases to show-case the utility of the library, one benchmarking asynchronous HPO algorithms, one assessing the effectiveness of a algorithm for optimization with constraints (cBO), and one providing an example of multi-optimization with MOASHA.
overall_review: + The library has nice variety of algorithms
+ The library supports algorithms for transfer learning in HPO, which I haven't seen yet in any other library
+ The benchmarking system seems very helpful, especially how it can help simulate parallelized trainings on long training. Tabular datasets is helpful, but this could easily be reused outside the library. 

- Since many of the algorithms seems to be in-house reimplementations, it would have been great to have performance comparison to previous implementations (Hyperband/ASHA/PBT from Ray-tune, TPE from Hyperopt and Optuna, BO compared to DragonFly and BoTorch/Ax). This would be a great contribution for a paper.
review_summary: The library presented in this paper could be very useful for researchers developing new HPO algorithms. The apparent modularity of the algorithms implementation and the benchmarking system seams to be especially useful. The use case examples are clear and concise. I believe this paper is a very appealing presentation for this library.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This work presents Syne Tune, a backend agnostic HPO system, that contains implemented baselines and supports hyperparameter transfer learning, constrained HPO and multi-objective HPO. The paper presents the design of system and provides a benchmark for different HPO algorithms (asyn, syn), which is valuable to the community.
overall_review: See the summary below.
review_summary: This work introduces Syne Tune, another package in HPO. The presentation is clear and sound. But several issues should be better clarified.

Advantage:
- Syne Tune contains implemented baselines and supports hyperparameter transfer learning, constrained HPO and multi-objective HPO.
- The paper introduces a benchmark of different algorithms.
- Good illustration cases are introduced.

Disadvantage:
- The design part is too vague. It is not clear how in architecture design, API, Syne Tune differs from other works.
- The related work section already mentions a lot of HPO packages. The authors should elaborate more why we need another packge on HPO. Does it support something which is not at all supported by other packages? For example, from what I understand, Optuna covers the functionality of Syne Tune, the only difference is that Optuna doesn't come up with built-in most recent methods (but should totally implementable) Thus,  I'd suggest a revised part on related works to discuss more thoroughly (since we already have a lot packages...)
- Some features are a bit over-claimed, e.g. multi-objective (which in Syne-Tune, is mostly choose machine types on AWS)
review_title: null

Meta-review: metareview: Reviewers were satisfied with the empirical evaluation, description of architecture, and reproducibility. The unanimously agreed that this work should be accepted. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept