What can multi-cloud configuration learn from AutoML?

1 INTRODUCTION . We are currently living in the era of cloud computing , in which cloud providers compete to offer computing resources , including servers , storage and managed software , to businesses and consumers via the internet . Cloud is a high growth segment : according to Gartner ( 2021 ) , worldwide end-user spending on public cloud services is forecast to grow 23.1 % in 2021 to total $ 332.3 billion . From an enterprise perspective , there are many benefits to utilizing cloud services including , but not limited to : increased flexibility , reduced costs , improved security and rapid scaling . Coupled to the rise of cloud is the growing popularity of containers and container orchestration platforms . A container image ( e.g. , Docker ) is an executable package of software that includes everything needed to run an application : code , runtime , system tools , system libraries and settings ( Soltesz et al. , 2007 ; Merkel , 2014 ) . At runtime , container images become containers and can be deployed quickly and reliably from one computing environment to another , making them well-suited to the world of cloud computing . Many of today ’ s workloads , such as machine learning ( ML ) training , are distributed in nature and require the deployment and management of multiple containers , all working together in parallel . Container orchestration platforms such as Kubernetes ( Brewer , 2014 ) enable running such systems resiliently , providing features like auto-scaling and failure management . Most cloud providers now offer a Kubernetes service , meaning that they require only a few commands to create a Kubernetes cluster , and the user can specify the desired number of nodes and how each node should be configured ( how many CPUs , memory , etc. ) . Given a Kubernetes cluster , one can easily run a complex distributed workload , whilst remaining agnostic to which cloud provider is providing the resources in the backend . This level of abstraction presents the savvy business or consumer with a new optimization opportunity , which we will refer to as the multi-cloud configuration problem . Namely , given a workload , which cloud provider should be selected , and how should the nodes in the cluster be configured , in order to minimize runtime or cost ? In practice , this optimization problem is of particular interest to businesses with complex , distributed workloads that must be run repeatedly . For instance , consider a business with a ML-based recommendation engine that must be trained , in a distributed fashion , once every hour . By solving the multi-cloud configuration problem , the business will be able to exploit price/performance differences that may exist between cloud providers to improve their bottom-line . However , any optimization algorithm will involve making a certain number of evaluations ( e.g. , running the workload on a particular cloud provider , with a particular node configuration ) , and these evaluations will themselves incur a certain dollar cost . For this reason , effective algorithms with fast convergence are of particular interest in this domain , and are the primary focus of this paper . The main contributions of our work are as follows : • We formally introduce the multi-cloud configuration problem and draw an interesting connection with the selection-configuration problems that are commonly studied in the automated machine learning ( AutoML ) field . • Inspired by AutoML approaches , we propose a best-arm identification algorithm , CloudBandit , for solving the multi-cloud configuration problem . • We present a new dataset , MOCCA , for offline benchmarking of optimization algorithms applied to 60 multi-cloud configuration tasks , across 3 public cloud providers . • Extensive experiments on MOCCA show that CloudBandit outperforms many generic black-box optimizers and state-of-the-art selection-configuration algorithms . 2 PROBLEM STATEMENT AND BACKGROUND . 2.1 MULTI-CLOUD CONFIGURATION AS OPTIMIZATION . The runtime/cost of a workload in the cloud depends on many factors , such as the number of nodes used , the vCPU count , the amount of memory allocated to each node , the manufacturer/generation of the backend CPU , the network on which the nodes communicate and the region in which the nodes are deployed . While some of these parameters ( e.g. , vCPU count and amount of memory ) are , on the surface , common across different cloud providers , most providers do not give the option to set these parameters freely . Instead , one must select from a set of available VM categories ( often parameterized into sub-categories like family , type or size ) that define which CPU will be used , how many vCPUs and how much memory will be assigned to the node , as well as what network interfaces are used . These ( sub- ) categories are significantly different across cloud providers , making it difficult to construct a multi-cloud optimization problem over parameters that are common across all cloud providers ( e.g. , vCPU count ) without introducing complex constraints . Instead , it is much easier to view the multi-cloud configuration problem as an optimization over a hierarchical domain , where the domain for each cloud provider comprises a unique set of categorical parameters . Let { C1 , C2 , ... , Cn } denote the set of available cloud providers and Pi the set of all possible node configurations for the i-th cloud provider for i = 1 , 2 . . . , n. We can then define the functions fi ( p ) : Pi → R , which , for the i-th cloud provider with node configuration p , may measure either workload runtime , monetary cost or some other optimization target . We can then formally define the multi-cloud configuration problem as follows : i∗ = argmin i∈ [ 1 , n ] [ min p∈Pi fi ( p ) ] ( 1 ) We note that this is a joint optimization problem comprising the outer selection problem , i.e. , which cloud provider to use , as well as the inner configuration problem , i.e. , how to configure the nodes . 2.2 RELATED WORK IN CLOUD LITERATURE . Configuration subproblem . There is a significant body of work studying the configuration subproblem : i.e. , given a cloud environment Ci , how to find the number and types of computing instances that would maximize the performance or minimize the cost of a workload deployment . Chen et al . ( 2021 ) present a cloud configuration framework that builds prediction models from small-scale experiments to estimate the performance of experiments on large-scale clusters . Mahgoub et al . ( 2020 ) introduce an online system that adjusts the database and VM configurations as the behavior of the workload changes . Mariani et al . ( 2017 ) define a method to estimate the cloud performance of a workload by employing hardware-agnostic application profiles and random-forest-based prediction models . Other prior art from the cloud literature considered employing black-box optimizers ( BBOs ) , such as Bayesian Optimization ( BO ) , to optimize the cloud instance type , cluster size , or both ( Yadwadkar et al. , 2017a ; Alipourfard et al. , 2017 ; Bilal et al. , 2020 ; Hsu et al. , 2018b ) . Additionally , a number of works have looked at improving the performance of BO by augmenting it with low-level performance metrics captured from the workload Hsu et al . ( 2018a ; c ) . Selection subproblem . None of the approaches mentioned so far have addressed the cloud configuration problem in the multi-cloud setting . However , there has been work in the direction of cloud provider selection ( without configuration ) . Chen et al . ( 2020 ) introduce a provider selection optimizer ( FrugalML ) for a specific workload , namely ML inference . Given input data , FrugalML suggests which cloud providers ( cloud APIs ) should optimally be used for running an inference task so that the user budget is met and the prediction accuracy is increased . Moreover , the selection problem has been widely addressed in the context of multi-cloud service composition ( MCSC ) . Pang et al . ( 2020 ) propose a method based on formal concept analysis and skyline hierarchical computation to obtain an optimal MCSC in a mobile edge computing environment . To solve MCSC under uncertain QoS attributes , Haytamy & Omara ( 2020 ) describe a method based on Particle Swarm Optimization . Souri et al . ( 2020 ) use formal verification to prove the correctness of a MCSC approach that finds the service composition with the minimum number of cloud providers under given QoS user requirements . Joint selection-configuration problem . To the best of our knowledge , the joint cloud selectionconfiguration problem has been less studied . Yadwadkar et al . ( 2017b ) consider the problem of finding the best VM for a single-node workload across multiple cloud providers . The problem is framed as a prediction task and a random forest model is trained on a dataset comprising lowlevel performance metrics from pre-existing workloads . The trained model can then be used to effectively predict the performance of each VM for unseen workloads . An orthogonal direction , with which our paper is aligned , treats the multi-cloud configuration problem as an optimization task and aims to develop algorithms that do not require information collected from pre-existing workloads . Ramamurthy et al . ( 2020 ) propose a search-based method that ranks cloud providers for a given workload . The VM configuration for each cloud provider is optimized independently , using an arbitrary optimization algorithm that may take into account memory and/or disk space constraints . Shi et al . ( 2020 ) describe a Genetic Algorithm that finds a multi-cloud deployment configuration for multi-service applications . To decrease the complexity of the search , the authors design a clustering algorithm that groups dependent services , so that services in the same cluster are deployed to the same location ( provider or region ) . Challenges arise with this approach when mutations occur that cross cloud providers , since the set of VMs offered by each provider differ significantly . 2.3 RELATION TO AUTOML . In Bilal et al . ( 2020 ) the authors make an analogy between cloud configuration and hyperparameter tuning in machine learning . They point out important differences between these two problems . Firstly , cloud configurations consist mostly of integer or categorical variables , whereas hyperparameters in machine learning are often continuous . Secondly , cloud configurations are typically complex and expensive to evaluate ( involving spinning up a cluster of nodes ) , whereas hyperparameter configurations may be easily evaluated on a single node . Despite these differences , the authors found that methods that work well for hyperparameter tuning ( such as BO with random forests or gradient boosted trees ) are also extremely effective for solving the cloud configuration problem . The key insight of this paper is to go one step further and notice that , due to its hierarchical nature , the selection-configuration problem defined in ( 1 ) bears a striking resemblance to the joint optimization problem at the heart of AutoML . Namely , in AutoML one must select which ML model to use ( e.g. , neural networks vs. gradient-boosted decision trees ) , and decide how to configure each model ( e.g. , how each hyperparameter should be chosen ) . Figure 1 illustrates this analogy . 2.4 RELATED WORK IN AUTOML . Having established this analogy , it makes sense to ask the question : can existing methods from AutoML be applied to the multi-cloud configuration problem ( 1 ) ? We now proceed to review the state-of-the-art with this question in mind . Two early , ground-breaking AutoML solutions were auto-sklearn ( Feurer et al. , 2019 ) and AutoWEKA ( Thornton et al. , 2013 ) . Both of these frameworks remain highly active GitHub projects , and both use sequential model-based algorithm configuration ( SMAC ) for solving the model selectionconfiguration problem . SMAC , introduced in Hutter et al . ( 2011 ) and recently extended in Lindauer et al . ( 2021 ) , implements a form of BO that takes into account the natural hierarchy present in selection-configuration problems . SMAC is itself an actively maintained GitHub project , and provides APIs that can be used to solve arbitrary selection-configuration problems ( rather than being specific to AutoML ) . Accordingly , SMAC can be easily applied to multi-cloud configuration . More recent works have proposed splitting the model selection-configuration problem into simpler subproblems and solving them in an alternating fashion . In principle , all of these approaches can be readily applied to the analogous problem of ( 1 ) . Firstly , an approach based on the alternating direction method of multipliers ( ADMM ) was proposed in Liu et al . ( 2020 ) . Secondly , in Rakotoarison et al . ( 2019 ) , it was proposed to use Monte-Carlo tree search to solve the selection problem , and BO to solve the configuration problem . Finally , Li et al . ( 2020 ) proposed Rising Bandits ( RB ) , an algorithm that treats the model selection problem as a best-arm identification problem , in which each arm corresponds to a ML model and each arm pull corresponds to running BO with a fixed budget to optimize the corresponding model configuration . These methods , in particular RB , served as a main inspiration for the algorithmic approach described in the next section . On the other hand , there exist AutoML solutions which can not be applied to multi-cloud configuration in a straightforward manner . For example , frameworks like H2O ’ s AutoML ( LeDell & Poirier , 2020 ) and AutoGluon ( Erickson et al. , 2020 ) do not perform selection , but rather form ensembles of different , configured ML models . While ensembling has proven very successful in the AutoML context , achieving better accuracy than a single model , it does not have a natural analogy in the context of optimization problem ( 1 ) . Other AutoML frameworks , such as TPOT ( Olson & Moore , 2016 ) , focus more on the problem of feature engineering to achieve high accuracy , rather than selection-configuration , and thus do not relate directly to problem of minimizing runtime and/or cost of cloud workloads . Another popular direction in AutoML are multi-fidelity approaches like Hyperband ( Li et al. , 2017 ) or BOHB ( Falkner et al. , 2018 ) that try to eliminate bad configurations very quickly by evaluating them using a small resource ( for instance , a small subset of the training examples ) . Such methods can not be applied to multi-cloud configuration in the general setting since we may not know the specifics about the workload that is to be optimized , and even if we did , it may not admit any sensible notion of resource . Finally , there exist numerous commercial offerings in the AutoML space such as Google Cloud Platform AutoML Tables , IBM AutoAI , H2O Driverless AI , and many more . These proprietary offerings typically provide ML-specific interfaces and the details of how they solve the problem internally remains unknown .

Summary: The paper proposes a method to find the best cloud provider for multi cloud compute selection (time, cost) to avoid vendor lock in. They achieve this using a best-arm identification algorithm where each arm runs an BBO on each provider and eventually choosing the best arm. They test this approach on 60 different configuration across 3 providers using a proposed dataset and show that the algorithm provides cheaper, faster config. given a set of parameters.