Cerebro: A Layered Data Platform for Scalable Deep Learning

Deep learning (DL) is gaining popularity across many domains thanks to tools such as TensorFlow and easier access to GPUs. But building large-scale DL applications is still too resource-intensive and painful for all but the big tech firms. A key reason for this pain is the expensivemodel selection process needed to get DL to work well. Existing DL systems treat this process as an afterthought, leading to massive resource wastage and a usability mess. To tackle these issues, we present our vision of a first-of-its-kind data platform for scalable DL, Cerebro, inspired by lessons from the database world. We elevate the DL model selection process with higherlevel APIs already inherent in practice and devise a series of novel multi-query optimization techniques to substantially raise resource efficiency. This vision paper presents our system design philosophy and architecture, our recent research and open research questions, initial results, and a discussion of tangible paths to practical impact. 1 DEEP LEARNING: THE PROMISE, THE PAIN Deep learning (DL) has revolutionized machine learning (ML), poweringmodern speech recognition, machine translation, e-commerce, radiology, and more. The successes of DL, primarily at Web giants, has led to high interest among domain scientists, enterprises, smaller Web companies, and even healthcare firms in trying DL for their analytics tasks. The excitement is mainly due to DL’s power to unlock valuable unstructured data, e.g., sensor time series in public health, satellite images in agriculture, and text corpora in finance. While tools such as Spark and Flink tame the “Volume” and “Velocity” aspects of “Big Data” analytics, the “Variety” issue is still largely open. DL is key to taming this third V for analytics. Naturally, new tools such as TensorFlow aim to make DL easier to use. Cloud computing is also making GPUs more accessible. In spite of all this progress, DL is still too hard to use at scale, resource-intensive, and costly. While off-the-shelf pre-trained models mitigate this issue for a few well-defined prediction tasks, most ML users still need to build custom models for bespoke datasets and applications, either manually or with AutoML procedures. This process is painful in large part due to DL’s flexibility, which is unprecedented in ML: the input/output can be almost any data structure and internal layers can be built in many ways [27]. Such end-to-end learning makes DL more accurate but it also makes DL training unusually resistant to software commodification. Even the cloud “whales” do not offer much beyond basic IaaS support for DL training (although SaaS exists for some inference tasks). This article is published under a Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/), which permits distribution and reproduction in any medium as well allowing derivative works, provided that you attribute the original work to the author(s) and CIDR 2021. 11th Annual Conference on Innovative Data Systems Research (CIDR ‘21), January 10–13, 2021, Chaminade, CA, USA DL’s flexibility also underlies a key reason for its cost: model selection, an empirical process comparing alternate training configurations with varying input/output representations, neural architectures, and/or hyperparameters [37]. This process is unavoidable due to fundamental theoretical reasons on the bias-variance-noise tradeoff in ML accuracy [57] and the bespoke nature of many tasks. Without proper model selection, users will just squander the power of their labeled data, hurting the application. Alas, there exist no reliable ways to tell the accuracy of a configuration on a given dataset without actually running it, leading to mass empiricism. If one tuned, say, 3 hyperparameters with, say, 4 possible values each, the number of configurations is 64 already. Multiply that with more trials for varying input/output representations and neural architectures. No wonder practitioners routinely build 10s to even 1000s of models to pick just one [6]. Overall, model selection is critical for effective use of DL but it massively amplifies DL’s cost and energy footprints and impedes usability. 1.1 What do Existing Tools Lack? Why? DL systems such as TensorFlow meet a rather low-level need: specify a neural architecture, train it with stochastic gradient descent (SGD) or its variants, and run inference [12]. They do this rather well, at least on a single node. But the user must also specify the model selection process somehow, which is mainly why APIs such as Keras [21], AutoKeras [31], and Hyperband [42] arose. Model selection is an afterthought in DL systems, a burden dumped on the user to be handled with ad hoc outer loops and libraries, not as the mission-critical process it is. This design has led to two issues: (1)Usabilitymess:Without first-class support for model selection, users handle extra code and metadata in ad hoc ways. This often results in unreproducible executions and stymies cross-organizational collaboration.While some cloud vendors andML platforms are starting to support this process better, it is still largely piecemeal and often limited to only basic tasks such as hyperparameter tuning. (2)High resourcewastage:Different configurations tried inmodel selection often overlap substantially on data and/or computations. Ignoring this structure leads to high wastage of compute, storage/memory, and/or network resources, especially at scale. This issue is orthogonal to both making single-model training faster and to devising better AutoML procedures to navigate the search space. While the Googles, Facebooks, and Amazons of the world may gluttonously throw 100s of GPUs/TPUs and engineers at their DL tasks with low regard for overall resource efficiency, most other DL users cannot afford that, especially on pay-as-you-go clouds. And while some Web giants may ignore reproducibility for some ML tasks, it is a showstopper issue for most enterprises and domain scientists. There is a pressing need to meet these concerns of the fast-growing user base of DL outside Web giants and cloud whales. Transfer Learning High-level