Pangeo 2.0
By: Ryan Abernathey
Tags: Science, Data Science, Python, Pangeo

This period has led to some major technological innovations that will have long-lasting impacts on climate data science (and beyond), including:

Despite the diversity of these activities, an increasing amount of energy from Pangeo core contributors has recently been devoted to operating cloud-based Jupyter infrastructure, the JupyterHubs on Google Cloud and AWS that now support hundreds of active users. The project has become victim of the unexpected success of these services, which we originally developed as experimental prototypes. Pangeo as currently constituted is simply not well suited to continue to operate production-grade Jupyter-in-the-cloud services. The reason is that the project is very informally organized, lacking any formal legal identity, HR department, etc. We cannot, for example, staff a helpdesk or provide a service-level agreement.

So going forward, we are going to try to wind down our current mode of operating JupyterHubs, seeking a transition to a more sustainable path! If you currently use our cloud Jupyter services, don’t worry! — they are not going away. Below we enumerate some options for those who want to keep using Pangeo in the cloud.

New Infrastructure Providers

A Pangeo-style cloud environment is more than just a vanilla JupyterHub — it also means access to Dask clusters on demand, plus a specialized software environment. When we started operating JupyterHubs in the cloud three years ago, there were few commercial options available for purchasing these services, and we had to roll our own. Now the situation has changed. Some exciting new companies have recently launched to provide Jupyter together with scalable Dask clusters in the cloud. These include

Coiled: Founded by the Dask creators, Coiled provides Dask as a service to both individuals and enterprise.

Saturn Cloud: one of the first to offer Jupyter + Dask as a service.

Both of these are venture-backed companies, and both have current / former Pangeo collaborators on staff. The cloud providers themselves are also showing signs of providing similar services. The UK Met Office informatics lab has been working closely with the Microsoft Azure team to streamline the Pangeo experience there. Microsoft’s newly announced Planetary Computer project team includes long-time Pangeo contributor Tom Augspurger, and we’re excited to see what they build!

Another new player in the space, aimed more at the academic research and education space, is the International Interactive Computing Collaboration (2i2c). A non-profit, 2i2c was founded by core members of Jupyter and Pangeo. 2i2c’s mission is “to accelerate research and discovery, and to empower education to be more accessible, intuitive, and enjoyable.” Its main activities are as follows:

2i2c provides managed hubs for data science in research and engineering communities. They are tailored for the communities they serve and 💯 open source. 2i2c develops, supports, leads, and advocates for open source tools in interactive computing that are created, used, and controlled by the community.

Going forward, 2i2c will eventually assume operational responsibility for Pangeo current cloud JupyterHub and Binders services. These services will remain free to users, thanks to ongoing support from the NSF EarthCube program. 2i2c is hiring an Open Source Infrastructure Engineer to work on this! If you’re interested in shaping the future of interactive cloud computing, please apply!!

Finally, for those who wish to deploy and operate their own Pangeo-style cloud infrastructure, Quansight recently launched the excellent Qhub project, which massively eases the pains of deployment and management. With Qhub, Groups can set up their own JupyterHubs with Dask Gateway in minutes using cloud-agnostic open-source code and then modify environments, users, groups and more by committing changes to a single YAML file on GitHub. In a similar vein, we have migrated the Pangeo helm chart to the Dask org which means its maintenance benefits from the larger Dask community.

Shifting away from infrastructure operation will allow Pangeo to refocus on our core mission: cultivating the development of innovative open-source tools for solving challenging scientific problems.

New Pangeo Funding and Initiatives

Pangeo will continue to evolve, supported by several major new grants to collaborating institutions.

Project Pythia: Education and Training

Pangeo has driven forward the capabilities of Python tools for geosciences massively in the past years. But these tools will only realize their potential impact if scientists have access to high-quality training for learning to use them!

A new NSF EarthCube grant, awarded to NCAR and the University at Albany, will fund the development of Project Pythia: a community educational resource. The Project Pythia portal aims to provide geoscientists at any point in their career with the educational content and real-world examples needed to learn how to navigate and integrate the myriad packages within the burgeoning Scientific Python Ecosystem. Pythia will cover a range of topics from beginning Python programming to advanced subjects such as developing scalable workflows. A particular emphasis will be placed on migrating workflows to the cloud. Educational content in the Pythia portal will be developed and vetted in part through integration with graduate and undergraduate-level coursework at the University at Albany. More on Project Pythia can be found here.

Pangeo Forge: A Cloud Native Data Repository

Pangeo has blazed a trail towards a “cloud native” way of working with geoscience data. This vision is laid out in the following blog post:

The cloud native approach means avoiding data downloads and instead working directly with massive cloud-based datasets using on-demand scalable computing. We believe that cloud-native has the potential to transform scientific research, making scientists more productive, creative, and flexible.

However, cloud native science requires cloud-based data. Currently, the process of producing analysis-ready, cloud-optimized (ARCO) data is rather painstaking and manual. Our existing Pangeo cloud data catalog is difficult to update.

One option is to wait for data providers such as NASA and NOAA to begin providing their data in ARCO format. However, with Pangeo, we are taking a more proactive approach, by developing a platform for “crowdsourcing” of ARGO data. This project is called Pangeo Forge.

The idea of Pangeo Forge is to copy the very successful pattern of Conda Forge for crowdsourcing the curation of an analysis-ready data library. In Conda Forge, a maintainer contributes a recipe which is used to generate a conda package from a source code tarball. Behind the scenes, CI downloads the source code, builds the package, and uploads it to a repository. In Pangeo Forge, a maintainer contributes a recipe which is used to generate an analysis-ready cloud-based copy of a dataset in a cloud-optimized format like Zarr. Behind the scenes, CI downloads the original files from their source (e.g. FTP, HTTP, or OpenDAP), combines them using xarray, writes out the Zarr file, and uploads to cloud storage.

The Pangeo Forge project is just getting off the ground, but we are very excited by the possibilities. Lamont Doherty Earth Observatory recently received a major award from the NSF EarthCube project to build out Pangeo Forge, in partnership with several high-profile data providers. Stay tuned for an announcement with more detail!

Pangeo for Earth System Machine Learning

For over a year now, Pangeo’s Machine Learning Working Group has held an open, monthly meeting to discuss challenges and solutions around big data geoscientific machine learning. The conversations in this meeting have often centered around how to utilize Pangeo’s ecosystem of software and infrastructure to accelerate machine learning with the high-dimensional datasets often found in the geosciences.

Out of those conversations grew the Pangeo-ML project and a now-funded proposal to the NASA ACCESS program. As a collaborative effort led by Joe Hamman between CarbonPlan, Columbia University, the University of Wisconsin-Madison, and Anaconda, the Pangeo-ML project aims to make it easier to use the same software tools for interactive data exploration and ML model development. To achieve this, the project team will improve the integration of data manipulation libraries (e.g. Xarray and Pyresample) — targeting known pain points found in ML applications, and will develop new high-level interfaces between Xarray and deep-learning libraries such as TensorFlow and PyTorch. The Xbatcher project is an early example of an interface tool that helps bridge the gap between Xarray and ML libraries.

Hypothetical research workflow for a scientist using Pangeo-ML. Data hosted on the cloud is integrated into a familiar ecosystem that can provide extract-transform-load functionality as well as exploratory data analysis and visualization. Using the same tool sets, scientists will be able to quickly iterate on model design, and training and validation.

A Fresh Approach to our Weekly Telecons

As a consequence of our drift towards operating cloud infrastructure, our weekly Pangeo meetings have sometimes been dominated by highly technical discussions of Kubernetes Taints and Tolerations (those are real terms, I promise), rather than geoscience software and its applications. As we shift our operational capacity to 2i2c, we also look forward to refreshing the structure of our weekly meeting.

Going forward, this meeting will operate as a virtual seminar, showcasing a cool new tool, dataset, or scientific problem in need of a software / infrastructure solution. Each meeting will feature a 15 minute presentation, which will be recorded and shared on our website. We are currently planning the schedule for Spring 2021, so please don’t hesitate to sign up if you’re interested in presenting!

Full form available at https://forms.gle/hJyhsFvueMXPgqGr6

Through this transition, Pangeo will remain an open, inclusive community. Anyone interested in engaging with the project is encouraged to check out our Discourse forum, which has become the primary forum for discussion and project communication.

Stay tuned for more detailed blog posts about these various new initiatives, and Happy New Year from the Pangeo community!