Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: N/A
overall_review: N/A
review_summary: N/A
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper proposes Open Source (OSS) Vizier, a Python implementation of Google Vizier's APIs.
overall_review: Proposing open source versions of existing commercial tools are a very important contribution to the literature, especially when key industry players are involved. That being said, the focus of this work is on (i) the architecture proposed and (ii) how the proposed system represents a contribution with regard to other existing systems. I believe the issues with this paper intersect (i) and (ii). In detail, authors have selected a very limited subset of systems to compare to, and their discussion on pros and cons from these systems is even more limited. As a result, when discussing (i), authors present many existing architectural choices as a contribution of the proposed system, when in fact they are indeed good choices, but far from novel. Overall, I recommend that the paper be reviewed to properly value the existing literature on the field, which is mature and highly relevant.
review_summary: This paper proposes a very important contribution to the literature, but I recommend that the paper be reviewed to properly value the existing literature on the field, which is mature and highly relevant.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: This paper describes and provides an open-source python-based interface for black-box optimization (i.e., OSS Vizier) based on the Google Vizier [1]. Compare with the previous work, it proposes an evolved backend design and several new features (e.g., multi-objective). The technical details are presented in the paper, providing a user-friendly black-box optimization toolbox for everyone who tends to solve a black-box optimization problem.

[1] Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J., & Sculley, D. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
overall_review: This paper describes and provides an open-source python-based interface for black-box optimization (i.e., OSS Vizier) based on the Google Vizier [1]. Compare with the previous work, it proposes an evolved backend design and several new features (e.g., multi-objective).  The contribution of this paper is clear, but it lacks performance evaluation.
review_summary: Overall an interesting paper with clear contributions. Its performance evaluation can be further improved.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The paper describes Open Source (OSS) Vizier. Vizier is a blackbox optimization service that has been used extensively internally at Google. The key aspect that sets OSS Vizier apart from existing frameworks/libraries is that it is designed to be deployed in a massive parallel distributed setting, and provides various solutions (communication protocols, faut tolerance, etc.) that enable it to scale, while at the same time staying conceptually simple, providing a high-level pythonic API, both for users (e.g., using BBO to do HPO) and developers (implementing an testing BBO algorithms).
overall_review: As listed on https://automl.cc/call-for-papers-on-special-track-for-systems-benchmarks-and-challenges/ a good system’s paper should show that
1.	It is a novel system that has features or application domains that were not available beforehand.
2.	It already has an established user base (shown by stars on github, active commit history by several developers, an active issue tracker, etc.)
3.	It is an open-source software package with an open-source software licence that allows users to easily use and contribute to it.
4.	It achieves excellent performance on the addressed application domains.

(ordered from positive to negative) In this work, the authors…
3) open source a proprietary system.
1) describe a system providing various general features. While many of these are not novel (many other libraries/frameworks support them), the fact that the system supports them all in a way that is not method specific, has the potential to allow AutoML researchers to leverage these features to focus on algorithmics rather than software engineering.
2) describe a system used exclusively(?) for Google internal projects (the Github repo has 15 stars). I am not sure this constitutes an active user base (will Google use the open-sourced version internally?). Also, broader adoption in the autoML  is non-obvious (see impact discussion)
4) do not provide any empirical evidence (see technical concerns).

review_summary: This paper open sources a proprietary system that claims great usefulness to the AutoML community.  I acknowledge that the system itself is interesting and that such efforts should be strongly encouraged. Sadly, all arguments the paper provides in support of its claim are of a conceptual nature. In my opinion, even a systems paper should show evidence for its claims, beyond “Google uses it”. It should provide algorithms the community cares about and demonstrate its features empirically. The lack thereof raises concerns about the usefulness of / adoption by the AutoML community and the impact of this work at large. The deciding factor is my decision (weak accept vs. reject) is my believe that the system, despite the lack of evidence, is useful. However, even if accepted, I would strongly encourage the authors to go through the extra effort of providing such evidence, as I feel that doing so would greatly increase the impact of their work.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: OSS Vizier provides a standalone - as a service -  API for blackbox optimisation with a focus on hyperparameter optimisation. Further, an API for users to interface their own algorithms with OSS Vizier enables the use of those too. The main difference between (OSS) Vizier and other frameworks is the distributed design, allowing function evaluations to be distributed over multiple clients that each report back the objective performance to the server running the optimisation algorithm.
overall_review: - A system to run BBO/HPO algorithms in a distributed fashion is useful and a welcome addition.
- Having this as a standalone OSS tool is great.
- Code seems to be of good quality and has tests included, this helps working with it and trusting it.
- Limitations and generality are somewhat unclear, which leaves questions about broadness of applicability and impact. Details under: "Potential Impact On The Field Of AutoML" and "Technical Quality And Correctness".
- The focus of OSS Vizier is discussed as being on HPO, but general BBO is also discussed. This makes the focus somewhat unclear. Details under: "Technical Quality And Correctness Rating".
- The many tools/frameworks/libraries related to in the context of OSS Vizier makes it confusing how OSS Vizier relates to them. Details under: "Technical Quality And Correctness Rating".
- The privacy issues related to data collection by the service provider are discussed, but in somewhat limited detail. Details under: "Ethics Details".
review_summary: This work presents a system to run BBO/HPO algorithms in a distributed fashion, which is both useful and a welcome addition. The main contribution is the release of the open source tool OSS Vizier, which provides APIs to optimise using existing public algorithms, and an API to add one's own algorithm implementations (and, presumably, optionally makes those publicly available). The public code seems to be of good quality and has tests included, this helps working with it and trusting it.

Unfortunately, there are also some issues with the current version of the paper. Limitations and generality of the tool are discussed, but in somewhat limited detail, which leaves questions about broadness of applicability (what can it be used for and what not) and impact. Further, while a focus on HPO is stated, discussion of both BBO and HPO leaves some confusion as to the goal.

Confusion is also caused by the many tools/frameworks/libraries related to in the context of OSS Vizier. At the same time, however, these may be of interest to users of those other tools. The privacy issues related to collection of data by the service provider are discussed, but also some cause for concern. However, since users can also set up their on service, it is possible for them to avoid this.

Overall, the paper and the tool have potential to be useful. The presentation clarity could be improved regarding (a) the broadness of applicability, (b) what is the exact relation to other systems. I believe the current version of the paper is not exceptional, but sufficient to be accepted. I want to encourage the authors to make the suggested improvements, and I hope to see the paper published.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper introduces Open Source (OSS) Vizier, an open-source package for black box optimization, based on Google Vizier. OSS Vizier provides a distributed system and service API for reliable and flexible optimization. The RPC-based infrastructure allows the users to implement client code using any language. The developer API allows the users to implement new algorithms conveniently. 
overall_review: Positive:
1.	OSS Vizier provides a reliable optimization by the distributed system and a flexible optimization by service API.
2.	OSS Vizier uses the RPCs to communicate which is compatible with other framework, like XManager.

Negative:
1.	OSS Vizier lacks more examples about how to utilize the functionalities and some experiment results on the simple synthetic functions. 
review_summary: This paper introduces OSS Vizier, an open-source package for reliable and flexible black box optimization. Researchers can use this package to solve their problems more quickly and easily. Therefore, I think it can be accepted.
review_title: null

Meta-review: metareview: 

This paper introduces an open source vizier (OSS) based on the google-internal vizier infrastructure. OSS vizier gives an API capable of defining and solving various optimization problems and it is designed to be a distributed system. 

The paper was after the revision of 6 reviewers with 4.5 in the average evaluated. At least one of the reviewers he went up from the 2 to a 4 because the clarity of the work improved. This already highlights that for this paper the rating has improved substantially because of the way the paper could be improved. As said, it is now in the range of the conference papers as the paper now has a 4.5 on average. The other marks for the potential impact, the technical quality and the clarity rating are all (except of one which is 2) between the 3 and the 4 rating.

Nevertheless, I have to admit that the open source paper is dependent on whether it will be used or not. Also, I don't understand why the 9-pages main PDF cannot be revised to avoid complications. I think it is a must! Hence, I would make it an accept, if in the 9-pages paper there can be improved according to these issues. 


confidence: 5: The area chair is absolutely certain
Decision: Accept