Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: he paper  provides a solution for multi-cloud configuration problem. Specifically, the paper tries to provide cloud customers with an optimal configuration to minimise runtime and cost. The paper also presents a dataset, for offline benchmarking, comprising of 60 different multi-cloud configuration tasks across 3 cloud service providers. 
main_review: Strengths
=========
1. The paper discusses a direction of research which will become more and more relevant in the years to come. As such this paper may inspire future researchers to work on the problem of multi-cloud configuration which will enable customers to minimize runtime and cost while running workloads on clouds provided by multiple vendors, without explicitly having to choose one vendor over another. 
2. The paper draws an analogy to AutoML and uses some of the concepts to come up with a novel approach based on best-arm identification problem. This is one of the key contributions of the paper.
3.  Another major contribution of the paper is the release of the multi-cloud configuration dataset which will be made publicly available. In line with my comment above this will further enable researchers to work on this area and make further inroads.
4. The experimental section appears very detailed. The authors have compared the use of different BBOs (black box optimizers) inside CB. The comparison has been done for both cost and runtime targets. However the 2nd part of the experimental section has some weakness as described under “weaknesses”.

Weaknesses
===========
1. The main algorithm for optimisation viz. CloudBandit appears to be a simple extension of existing approaches and does not seem to have enough novelty (considering the exclusiveness of ICLR)
2. Although there are multiple similar approaches the authors have run experiments against only 2 baselines viz. SMAC and RB. The authors argue that other approaches cannot be compared against as there is no public implementations available.Thus it not clear how this method compares against the SOTA approaches. 


summary_of_the_review: The authors have done a commendable job in a very relevant research area which will become more prominent in the coming years. The key contributions are: 1) CloudBandit a best-arm identification problem for multi-cloud configuration optimization. and 2) the public release of a multi-cloud configuration dataset. 
However the novelty does not seem to be adequate keeping in mind the standards of the conference. Also there are some weaknesses in the evaluation results and the comparision with SOTA is not clear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This pager presents CloudBandit to solve the multi-cloud selection-configuration problem. While the problem is heavily explored in several domains, this paper leverages the hierarchical aspect of the selection-configuration problem. The main novelty of the paper lies in creating the dataset for the above problem. Such dataset can be immensely useful for academic research as "executing code" in cloud can be prohibitively expensive in academic settings; thereby, hindering the academic research.
main_review: Overall, the motivation and idea presented in the paper is clear. However, it is significantly missing in details wrt to datasets, experiments and cloud provider configuration space. 

I would implore authors to think about following limitations of the work that should be addressed (and possibly their future work):

1. Taskset: Tasks in the current work only consist of ML-workloads which are algorithmically fixed, i.e., there is little to no control flow/branching. 

2. Dynamic environment: How will the results change across the cloud providers if the cloud itself is heavily loaded. Tail latency is a big issue in the cloud environment. See "Selecting the Best VM across Multiple Public Clouds: A Data-Driven Performance Modeling Approach" published in SOCC 17.

3. Moving away from black-box assumption: There is no requirement in this task to be workload agnostic, i.e., the users of the CloudBandit should have access to the workload and should be able to leverage insights from workload details (such as memory required, code complexity, processor, etc). One can obtain such details by profiling and tracing of the code and execution environment. Roofline models (see https://en.wikipedia.org/wiki/Roofline_model) should also help model aspects of the code and execution environment to significant reduce the accumulated cost while providing great performance. 
summary_of_the_review: Overall, the paper is well written. However, several aspects of the problem were ignored (see main review). The dataset presented could be meaningful but currently it lacks the rigor in generating the dataset or providing meaningful motivation for selecting "dask" tasks. This paper has the seed for significant contribution but the current quality of work is not sufficient for publication in a conference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to find the best cloud provider for multi cloud compute selection (time, cost) to avoid vendor lock in. They achieve this using a best-arm identification algorithm where each arm runs an BBO on each provider and eventually choosing the best arm. They test this approach on 60 different configuration across 3 providers using a proposed dataset and show that the algorithm provides cheaper, faster config. given a set of parameters.
main_review: The main contributions of the paper are: 1) A method to choose the best cloud provider in a multi cloud scenario to avoid vendor lock in given a task. The method provides an objective metric and helps narrow down the choice based on runtime or cost. 2) A dataset for benchmarking and comparisons (although this has to be released publicly yet). and 3) A comprehensive study against other Black box optimizers is provided. The optimization is studied firstly to choose which BBO is better for the search internally in CB and then it is evaluated against SMAC, RB to compare the overall algo.  Main concerns are the following 1) CB is evaluated over runtime and cost currently. However, in practice there are many variables in play such as region availability, SLA requirements, number of instance availability, networking requirements etc., Without actually looking at the current 60 multi cloud configurations that it has been evaluated on, it is hard to comment on the how the tasks were setup and how generally applicable this is or add more configurations. 2) Given that cloud providers provide custom rates for long term usage under corporate deals, in practice the cost estimation can vary quite a lot and passing these custom configurations to CloudBandit seems to be unknown.
summary_of_the_review: The authors study the problem of multi-cloud configuration in this paper, where the problem can be defined as how to choose the best cloud provider based on runtime or cost for a given workload. They propose an algorithm (CB) to help solve this and compare the algorithm for its effectiveness against SMAC, RB and show that it performs well at solving the optimization problem. More work needs to be done to show how complex workloads can be translated into configs that CB can solve and also account for more parameters when optimizing for runtime or cost as discussed above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors analyze the similarity between the multi-cloud configuration optimization task and the model selection-configuration problem, and try to solve the multi-cloud configuration optimization task inspired by an algorithm for best-arm identification from the AutoML domain.  They built a benchmark for the multi-cloud configuration task including  60 multi-cloud configuration tasks across 3 public cloud providers. The experimental results show the proposed method can find best cloud provider with best instances and other parameters more cheaper or faster, compared to BBOs.
main_review: The pros: 1) The authors firstly build a benchmark for the multi-cloud configuration task. 2) they evaluate the proposed methods comprehensively, compared to BBOs and other AutoML methods. 
The cons and questions：1) Can you justify the dataset is correct? Because it is an imperfect estimate, and the more precise estimates are difficult to obtain for the cost target. Does the dataset consider the different prices/costs from different providers? It seems to matter and impact the correctness of the proposed dataset for the cost target. 
2）Does the CB algorithm output only one best arm/provider? In practice, the users might choose multi-cloud to avoid vendor lock-in. Does it seem the goal of the algorithm should output top-n best arms/providers? 
3）The situation for AutoML and Multi-cloud configuration seems to be different in Fig. 1. In the Multi-cloud configuration case, each provider provides the same type of parameters, e.g., CPU number/type，memory size，VM/container type. But for AutoML，the parameter types are totally different for different models.  Is the special character in the multi-cloud configuration case helpful to solve the multi-cloud configuration problem? 
summary_of_the_review: In this paper, the authors try to solve the multi-cloud configuration problem with an algorithm from the AutoML domain. It looks good, but it requires more explanation and clarification why it is a good/best way by the proposed method, why it is better than other methods, why it works. And it also requires more justification of the correctness of the evaluation data.
review_title: null

Meta-review: This paper studies the problem of choosing the best cloud provider for a task. The problem is formulated as a bandit and solved using algorithm CloudBandit. The algorithm is compared to several baselines, such as SMAC, and performs well. The evaluation is done on 60 different multi-cloud configuration tasks across 3 public cloud providers, which the authors want to share with the public.

This paper has four borderline reject reviews. All reviewers agree that it studies an important problem and that the promised multi-cloud optimization dataset could spark more research in the area of cloud optimization. The weaknesses of the paper are that it is not technically strong and that the quality of the new dataset is not clear from its description. At the end, the scores of this paper are not good enough for acceptance. Therefore, it is rejected.
Decision: Reject