What can multi-cloud configuration learn from AutoML?

1 INTRODUCTION . We are currently living in the era of cloud computing , in which cloud providers compete to offer computing resources , including servers , storage and managed software , to businesses and consumers via the internet . Cloud is a high growth segment : according to Gartner ( 2021 ) , worldwide end-user spending on public cloud services is forecast to grow 23.1 % in 2021 to total $ 332.3 billion . From an enterprise perspective , there are many benefits to utilizing cloud services including , but not limited to : increased flexibility , reduced costs , improved security and rapid scaling . Coupled to the rise of cloud is the growing popularity of containers and container orchestration platforms . A container image ( e.g. , Docker ) is an executable package of software that includes everything needed to run an application : code , runtime , system tools , system libraries and settings ( Soltesz et al. , 2007 ; Merkel , 2014 ) . At runtime , container images become containers and can be deployed quickly and reliably from one computing environment to another , making them well-suited to the world of cloud computing . Many of today ’ s workloads , such as machine learning ( ML ) training , are distributed in nature and require the deployment and management of multiple containers , all working together in parallel . Container orchestration platforms such as Kubernetes ( Brewer , 2014 ) enable running such systems resiliently , providing features like auto-scaling and failure management . Most cloud providers now offer a Kubernetes service , meaning that they require only a few commands to create a Kubernetes cluster , and the user can specify the desired number of nodes and how each node should be configured ( how many CPUs , memory , etc. ) . Given a Kubernetes cluster , one can easily run a complex distributed workload , whilst remaining agnostic to which cloud provider is providing the resources in the backend . This level of abstraction presents the savvy business or consumer with a new optimization opportunity , which we will refer to as the multi-cloud configuration problem . Namely , given a workload , which cloud provider should be selected , and how should the nodes in the cluster be configured , in order to minimize runtime or cost ? In practice , this optimization problem is of particular interest to businesses with complex , distributed workloads that must be run repeatedly . For instance , consider a business with a ML-based recommendation engine that must be trained , in a distributed fashion , once every hour . By solving the multi-cloud configuration problem , the business will be able to exploit price/performance differences that may exist between cloud providers to improve their bottom-line . However , any optimization algorithm will involve making a certain number of evaluations ( e.g. , running the workload on a particular cloud provider , with a particular node configuration ) , and these evaluations will themselves incur a certain dollar cost . For this reason , effective algorithms with fast convergence are of particular interest in this domain , and are the primary focus of this paper . The main contributions of our work are as follows : • We formally introduce the multi-cloud configuration problem and draw an interesting connection with the selection-configuration problems that are commonly studied in the automated machine learning ( AutoML ) field . • Inspired by AutoML approaches , we propose a best-arm identification algorithm , CloudBandit , for solving the multi-cloud configuration problem . • We present a new dataset , MOCCA , for offline benchmarking of optimization algorithms applied to 60 multi-cloud configuration tasks , across 3 public cloud providers . • Extensive experiments on MOCCA show that CloudBandit outperforms many generic black-box optimizers and state-of-the-art selection-configuration algorithms . 2 PROBLEM STATEMENT AND BACKGROUND . 2.1 MULTI-CLOUD CONFIGURATION AS OPTIMIZATION . The runtime/cost of a workload in the cloud depends on many factors , such as the number of nodes used , the vCPU count , the amount of memory allocated to each node , the manufacturer/generation of the backend CPU , the network on which the nodes communicate and the region in which the nodes are deployed . While some of these parameters ( e.g. , vCPU count and amount of memory ) are , on the surface , common across different cloud providers , most providers do not give the option to set these parameters freely . Instead , one must select from a set of available VM categories ( often parameterized into sub-categories like family , type or size ) that define which CPU will be used , how many vCPUs and how much memory will be assigned to the node , as well as what network interfaces are used . These ( sub- ) categories are significantly different across cloud providers , making it difficult to construct a multi-cloud optimization problem over parameters that are common across all cloud providers ( e.g. , vCPU count ) without introducing complex constraints . Instead , it is much easier to view the multi-cloud configuration problem as an optimization over a hierarchical domain , where the domain for each cloud provider comprises a unique set of categorical parameters . Let { C1 , C2 , ... , Cn } denote the set of available cloud providers and Pi the set of all possible node configurations for the i-th cloud provider for i = 1 , 2 . . . , n. We can then define the functions fi ( p ) : Pi → R , which , for the i-th cloud provider with node configuration p , may measure either workload runtime , monetary cost or some other optimization target . We can then formally define the multi-cloud configuration problem as follows : i∗ = argmin i∈ [ 1 , n ] [ min p∈Pi fi ( p ) ] ( 1 ) We note that this is a joint optimization problem comprising the outer selection problem , i.e. , which cloud provider to use , as well as the inner configuration problem , i.e. , how to configure the nodes . 2.2 RELATED WORK IN CLOUD LITERATURE . Configuration subproblem . There is a significant body of work studying the configuration subproblem : i.e. , given a cloud environment Ci , how to find the number and types of computing instances that would maximize the performance or minimize the cost of a workload deployment . Chen et al . ( 2021 ) present a cloud configuration framework that builds prediction models from small-scale experiments to estimate the performance of experiments on large-scale clusters . Mahgoub et al . ( 2020 ) introduce an online system that adjusts the database and VM configurations as the behavior of the workload changes . Mariani et al . ( 2017 ) define a method to estimate the cloud performance of a workload by employing hardware-agnostic application profiles and random-forest-based prediction models . Other prior art from the cloud literature considered employing black-box optimizers ( BBOs ) , such as Bayesian Optimization ( BO ) , to optimize the cloud instance type , cluster size , or both ( Yadwadkar et al. , 2017a ; Alipourfard et al. , 2017 ; Bilal et al. , 2020 ; Hsu et al. , 2018b ) . Additionally , a number of works have looked at improving the performance of BO by augmenting it with low-level performance metrics captured from the workload Hsu et al . ( 2018a ; c ) . Selection subproblem . None of the approaches mentioned so far have addressed the cloud configuration problem in the multi-cloud setting . However , there has been work in the direction of cloud provider selection ( without configuration ) . Chen et al . ( 2020 ) introduce a provider selection optimizer ( FrugalML ) for a specific workload , namely ML inference . Given input data , FrugalML suggests which cloud providers ( cloud APIs ) should optimally be used for running an inference task so that the user budget is met and the prediction accuracy is increased . Moreover , the selection problem has been widely addressed in the context of multi-cloud service composition ( MCSC ) . Pang et al . ( 2020 ) propose a method based on formal concept analysis and skyline hierarchical computation to obtain an optimal MCSC in a mobile edge computing environment . To solve MCSC under uncertain QoS attributes , Haytamy & Omara ( 2020 ) describe a method based on Particle Swarm Optimization . Souri et al . ( 2020 ) use formal verification to prove the correctness of a MCSC approach that finds the service composition with the minimum number of cloud providers under given QoS user requirements . Joint selection-configuration problem . To the best of our knowledge , the joint cloud selectionconfiguration problem has been less studied . Yadwadkar et al . ( 2017b ) consider the problem of finding the best VM for a single-node workload across multiple cloud providers . The problem is framed as a prediction task and a random forest model is trained on a dataset comprising lowlevel performance metrics from pre-existing workloads . The trained model can then be used to effectively predict the performance of each VM for unseen workloads . An orthogonal direction , with which our paper is aligned , treats the multi-cloud configuration problem as an optimization task and aims to develop algorithms that do not require information collected from pre-existing workloads . Ramamurthy et al . ( 2020 ) propose a search-based method that ranks cloud providers for a given workload . The VM configuration for each cloud provider is optimized independently , using an arbitrary optimization algorithm that may take into account memory and/or disk space constraints . Shi et al . ( 2020 ) describe a Genetic Algorithm that finds a multi-cloud deployment configuration for multi-service applications . To decrease the complexity of the search , the authors design a clustering algorithm that groups dependent services , so that services in the same cluster are deployed to the same location ( provider or region ) . Challenges arise with this approach when mutations occur that cross cloud providers , since the set of VMs offered by each provider differ significantly . 2.3 RELATION TO AUTOML . In Bilal et al . ( 2020 ) the authors make an analogy between cloud configuration and hyperparameter tuning in machine learning . They point out important differences between these two problems . Firstly , cloud configurations consist mostly of integer or categorical variables , whereas hyperparameters in machine learning are often continuous . Secondly , cloud configurations are typically complex and expensive to evaluate ( involving spinning up a cluster of nodes ) , whereas hyperparameter configurations may be easily evaluated on a single node . Despite these differences , the authors found that methods that work well for hyperparameter tuning ( such as BO with random forests or gradient boosted trees ) are also extremely effective for solving the cloud configuration problem . The key insight of this paper is to go one step further and notice that , due to its hierarchical nature , the selection-configuration problem defined in ( 1 ) bears a striking resemblance to the joint optimization problem at the heart of AutoML . Namely , in AutoML one must select which ML model to use ( e.g. , neural networks vs. gradient-boosted decision trees ) , and decide how to configure each model ( e.g. , how each hyperparameter should be chosen ) . Figure 1 illustrates this analogy . 2.4 RELATED WORK IN AUTOML . Having established this analogy , it makes sense to ask the question : can existing methods from AutoML be applied to the multi-cloud configuration problem ( 1 ) ? We now proceed to review the state-of-the-art with this question in mind . Two early , ground-breaking AutoML solutions were auto-sklearn ( Feurer et al. , 2019 ) and AutoWEKA ( Thornton et al. , 2013 ) . Both of these frameworks remain highly active GitHub projects , and both use sequential model-based algorithm configuration ( SMAC ) for solving the model selectionconfiguration problem . SMAC , introduced in Hutter et al . ( 2011 ) and recently extended in Lindauer et al . ( 2021 ) , implements a form of BO that takes into account the natural hierarchy present in selection-configuration problems . SMAC is itself an actively maintained GitHub project , and provides APIs that can be used to solve arbitrary selection-configuration problems ( rather than being specific to AutoML ) . Accordingly , SMAC can be easily applied to multi-cloud configuration . More recent works have proposed splitting the model selection-configuration problem into simpler subproblems and solving them in an alternating fashion . In principle , all of these approaches can be readily applied to the analogous problem of ( 1 ) . Firstly , an approach based on the alternating direction method of multipliers ( ADMM ) was proposed in Liu et al . ( 2020 ) . Secondly , in Rakotoarison et al . ( 2019 ) , it was proposed to use Monte-Carlo tree search to solve the selection problem , and BO to solve the configuration problem . Finally , Li et al . ( 2020 ) proposed Rising Bandits ( RB ) , an algorithm that treats the model selection problem as a best-arm identification problem , in which each arm corresponds to a ML model and each arm pull corresponds to running BO with a fixed budget to optimize the corresponding model configuration . These methods , in particular RB , served as a main inspiration for the algorithmic approach described in the next section . On the other hand , there exist AutoML solutions which can not be applied to multi-cloud configuration in a straightforward manner . For example , frameworks like H2O ’ s AutoML ( LeDell & Poirier , 2020 ) and AutoGluon ( Erickson et al. , 2020 ) do not perform selection , but rather form ensembles of different , configured ML models . While ensembling has proven very successful in the AutoML context , achieving better accuracy than a single model , it does not have a natural analogy in the context of optimization problem ( 1 ) . Other AutoML frameworks , such as TPOT ( Olson & Moore , 2016 ) , focus more on the problem of feature engineering to achieve high accuracy , rather than selection-configuration , and thus do not relate directly to problem of minimizing runtime and/or cost of cloud workloads . Another popular direction in AutoML are multi-fidelity approaches like Hyperband ( Li et al. , 2017 ) or BOHB ( Falkner et al. , 2018 ) that try to eliminate bad configurations very quickly by evaluating them using a small resource ( for instance , a small subset of the training examples ) . Such methods can not be applied to multi-cloud configuration in the general setting since we may not know the specifics about the workload that is to be optimized , and even if we did , it may not admit any sensible notion of resource . Finally , there exist numerous commercial offerings in the AutoML space such as Google Cloud Platform AutoML Tables , IBM AutoAI , H2O Driverless AI , and many more . These proprietary offerings typically provide ML-specific interfaces and the details of how they solve the problem internally remains unknown .

Summary: This pager presents CloudBandit to solve the multi-cloud selection-configuration problem. While the problem is heavily explored in several domains, this paper leverages the hierarchical aspect of the selection-configuration problem. The main novelty of the paper lies in creating the dataset for the above problem. Such dataset can be immensely useful for academic research as "executing code" in cloud can be prohibitively expensive in academic settings; thereby, hindering the academic research.