Partnering with Cloud Providers
By: Ryan Abernathey
Tags: Cloud Computing, Science, Data Science, Pangeo

Ryan Abernathey and Joe Hamman cowrote this blog post following discussion with the Pangeo Steering Council.

The basic ingredients for a Pangeo deployment are a fast parallel storage system, scalable high-performance compute nodes, access to the internet, and software which makes all these elements work together for an amazing interactive data-analysis experience.

Pangeo deployment architecture. Via http://pangeo.io/architecture.html

We have deployed Pangeo on a wide range of different systems, from small university clusters to Top 500 supercomputers. We have also found that commercial cloud environments (e.g. Amazon Web Services, Google Cloud Platform, Microsoft Azure) are a particularly good fit for Pangeo. The ability to quickly scale compute up and down on demand fits well with the workflow of interactive big-data analysis, and cloud object stores work very well with Xarray, Zarr, and Dask. Additionally, the cloud offers the ability to compose tailor-made computational environments that fit specific scientific workflows, allowing scientists to be more efficient with their resource consumption.

We have already benefitted from numerous partnerships with cloud providers, mostly in the form of grants of cloud credits. These credits have been invaluable in prototyping, debugging, and fine tuning the various components of our cloud architecture. Some cloud providers are understandably enthusiastic about Pangeo, as they see it as a way to attract new customers to their platforms. Moving forward, we think it is useful to articulate some general principles to guide potential future partnerships with cloud providers. In the spirit of Pangeo, we are sharing these principles openly and seeking feedback from both the the Pangeo community and the cloud providers.

We are keenly aware of some past friction between open-source projects and cloud providers. Examples include MongoDB, Redis, who chose to modify their licenses to prevent cloud providers from monetizing their work without contributing back to the project. Closer to home, some tension has evidently emerged between the open-source Project Jupyter and the very similar yet distinct proprietary product Google Colab.

We want to encourage cloud providers to collaborate with the Pangeo team. We wish to democratize the use of Pangeo within the scientific community by making the adoption and deployment of the platform easy, while maintaining the principles that keep Pangeo open. To that end, we hope that cloud providers who wish to improve the Pangeo experience for their customers pursue this goal through the open source process.

Principles for Pangeo Cloud Infrastructure

Here we articulate a few general principles which guide the more specific recommendations below. These principles mainly apply to academic research infrastructure. In contrast to, say, infrastructure for a startup, academic research infrastructure must consider relatively long timescales, unique challenges involving maintainability / sustainability, and a unique funding landscape. We believe that cloud-based academic research infrastructure, including Pangeo, should be

Community driven —The needs of different institutions are fundamentally very similar. By collaborating, we can be more effective. This means leveraging existing projects like Jupyter, Dask, Zarr, etc. which already have community buy-in, and starting new ones very reluctantly.

—The needs of different institutions are fundamentally very similar. By collaborating, we can be more effective. This means leveraging existing projects like Jupyter, Dask, Zarr, etc. which already have community buy-in, and starting new ones very reluctantly. Open source — In the cloud, infrastructure is code. In Pangeo, all our cloud infrastructure is essentially contained in an open source GitHub repo. There is no behind-the-scenes magic. Such code should be licensed in a way that enables the entire research community to reuse and build upon it.

— In the cloud, infrastructure is code. In Pangeo, all our cloud infrastructure is essentially contained in an open source GitHub repo. There is no behind-the-scenes magic. Such code should be licensed in a way that enables the entire research community to reuse and build upon it. Modular — “all in one” solutions are cumbersome difficult to maintain long term. Separation of concerns is a key principle of good software and systems engineering. The cloud infrastructure needs of climate scientists are mostly the same as those of neuroscientists; each group should not be building everything from scratch. For example, both groups may want to use JupyterLab, Dask for computing, and Zarr for cloud storage, but may require different high-level analysis libraries.

— “all in one” solutions are cumbersome difficult to maintain long term. Separation of concerns is a key principle of good software and systems engineering. The cloud infrastructure needs of climate scientists are mostly the same as those of neuroscientists; each group should not be building everything from scratch. For example, both groups may want to use JupyterLab, Dask for computing, and Zarr for cloud storage, but may require different high-level analysis libraries. Vendor neutral — The US currently has three main commercial cloud providers (AWS, Google Cloud, and Azure), plus many other smaller providers and academic clouds (e.g. NSF Jetstream). These providers compete with each other by rolling out unique new services. Academic research infrastructure should strive to use only vendor-neutral services APIs. (Kubernetes has been a game changer in this respect, allowing the orchestration of complex cloud services in a completely vendor neutral way.) If this principle is followed, it means we redeploy our infrastructure anywhere, preventing any one vendor from capturing the market, which could have negative consequences for the research enterprise.

The last point was expressed elegantly in a blog post by Yuvi Panda.

Pangeo Architecture is Still in Flux

As anyone who uses Pangeo knows, we are quite far from a “stable release” (in the software-development sense). This doesn’t mean the platform is buggy and unreliable; it means that the core elements are still under active development. A great example is Dask Gateway, a multi-tenant server for securely deploying and managing Dask clusters. Dask Gateway natively supports Dask on Kubernetes and offers something the current iteration of Pangeo doesn’t, a secure and centrally managed Dask service. While the advent of Dask Gateway is quite exciting, it is also quite new and we’re still working out how best it will fit with Pangeo. This is just one example — there are numerous other nascent development efforts that we expect to disrupt the current architecture of Pangeo.

How Can Cloud Providers Contribute?

If we can realize the potential of Pangeo, many scientific fields will begin to seriously consider moving their day-to-day interactive workflows to the commercial cloud. This is an outcome that would be beneficial to both scientists and cloud-providers. So how can cloud providers help make it happen?

Based on the discussion above, we have some concrete suggestions.

Improve managed services — Pangeo works best when we can leverage the managed services (e.g. Kubernetes) offered by cloud providers. Cloud providers can look to Pangeo to highlight areas these services can be improved. Hopefully, this leads to full-featured implementations of these services on all cloud platforms and enhanced performance for things like deployment, autoscaling, heterogeneous clusters, and the use of spot/preemptible instances.

Help develop deployment procedures — As we covered above, Pangeo is built on open infrastructure (e.g. Kubernetes). Cloud providers could contribute by documenting and streamlining the deployment procedure of Pangeo on their hosted infrastructure. We expect this would also lead to the cloud providers making some quality of life improvements to both their managed Kubernetes services and the Pangeo infrastructure.

Contribute to the tools that make up Pangeo — The Pangeo Cloud Platform is really just a collection of open source software projects (e.g. Jupyter, Dask, Zarr, Xarray). These project are developed and maintained by a community of stakeholders that includes scientists, software developers, and infrastructure providers. Cloud providers hoping to see increased use of these tools on their platform should view contributing to these projects as a way to enhance their customers’ experience. To provide a tangible example of what this might look like, cloud providers could contribute to the Dask-cloudprovider project by expanding the list of supported cloud APIs. On the storage side, cloud providers could engage with the zarr spec development process to help the Pangeo community optimize our interaction with cloud object storage.

Provide grants of credits — This is the obvious one, but it remains relevant. Many institutions are reluctant to invest in cloud until the value can be clearly demonstrated. But without credits to experiment, it’s hard to develop realistic demonstrations. So far, cloud providers have been very generous here, so that is an easy one and we hope to see this continue.

Host Public Datasets — Storage is a major cost for Pangeo. Many of our datasets have high public value and would be well suited for public dataset programs. We are very happy that several cloud providers have already expressed interest in this possibility.

Future Outlook

As the different components of Pangeo — i.e. JupyterHub, Dask, Zarr — mature and become more stable on different cloud platforms, it will become easier and easier to deploy Pangeo. It may require just a single simple configuration file, or a few clicks on a dashboard. Reaching this point will benefit everyone, from the academic research community, to the growing number of startups in the climate data science space, to the cloud providers themselves.

We hope we’ve provided a reasonably comprehensive overview of how the Pangeo Project can partner with cloud providers. We’re developing these thoughts in the open and encourage feedback from across the Pangeo community, including the cloud providers themselves. We believe that, by following the principles we outlined above, we can get buy-in from the more conservative sides of academia that are skeptical of commercial cloud as a research platform. We also feel it’s the best path for realizing the full potential of this still-young technology.

We welcome comments below or on our Discourse forum.