Thoughts on the state of Xarray within the broader scientific Python ecosystem
By: Benoît Bovy
Tags: Machine Learning, Python, Pangeo, Data Science, Data Visualization

This is a follow-up on Ryan Abernathey’s blog post about supporting new Xarray contributors.

How Xarray is positioned within the Python’s Scientific Stack? (modified from Travis Oliphant, 2017 and originally based on Jake VanderPlas PyCon 2017).

Xarray has slowly but steadily gained in popularity since it was created by Stephan Hoyer in late 2013. Although its growth has accelerated over the last two years (notably alongside the emergence of the Pangeo project), Xarray still remains widely unknown, especially outside the community of Atmospheric / Ocean / Climate (AOC) Sciences. This could easily be explained by the fact that early development of Xarray happened — and still happen mostly — within that community.

Yet, Xarray nicely solves a fundamental problem — i.e., handling labels and metadata on top of raw arrays — that is encountered in so many domains. Of course, the goal of Xarray is not to solve everything, but clearly:

Xarray does not seem to be used to its full potential within the broader scientific Python ecosystem.

I’d like to illustrate this observation through two examples:

My personal experience : I discovered Xarray some time ago while I was working in the AOC field. Since then I moved away from that field but haven’t stopped using Xarray, which still meets my current needs perfectly (I haven’t totally moved from Earth Science, though). I’m now trying to convince my colleagues that Xarray might help them too for some of their work and must admit that it is easier than I thought. Like me before, colleagues who have been using Python for quite a while were either using Pandas the hard (or wrong?) way or were spending effort on hacking their own labels/metadata wrappers around NumPy arrays (or even worse, they didn’t use any label/metadata explicitly in their code). Either they didn’t know the existence of Xarray, or they barely knew it and thought it was not relevant. Of course, we cannot blame them for that! But we can work towards a better discoverability of Xarray.

: I discovered Xarray some time ago while I was working in the AOC field. Since then I moved away from that field but haven’t stopped using Xarray, which still meets my current needs perfectly (I haven’t totally moved from Earth Science, though). I’m now trying to convince my colleagues that Xarray might help them too for some of their work and must admit that it is easier than I thought. Like me before, colleagues who have been using Python for quite a while were either using Pandas the hard (or wrong?) way or were spending effort on hacking their own labels/metadata wrappers around NumPy arrays (or even worse, they didn’t use any label/metadata explicitly in their code). Either they didn’t know the existence of Xarray, or they barely knew it and thought it was not relevant. Of course, we cannot blame them for that! But we can work towards a better discoverability of Xarray. Recent developments within the scientific Python ecosystem: although I have limited background in the area of machine (deep) learning, a project like NamedTensor and this related post both seem to show that labeled arrays/tensors is a real concern also in that area, and yet Xarray isn’t really visible there.

Two possible ways towards more visibility and adoption of Xarray throughout the scientific Python ecosystem.

Here are some general (and hopefully useful) thoughts on how we could tackle the issue described above.



Build a more robust stack on top of Xarray.

Xarray itself does not solve any real scientific/business problem. It rather provides a solid infrastructure with core data structures and a consistent API for solving those problems. Anything at a higher-level is generally out of the Xarray scope.

A lot of various features have been built on top of Xarray in third-party packages. Many of them are listed here and here. This is great! Without Xarray, those packages would probably have been built directly on top of NumPy, thereby missing important opportunities offered by a proper array labeling system such as great reduction of the amount of boilerplate code and potential bugs (and thus much easier maintenance).

Unfortunately, this ecosystem of packages is not organized as a clear and consistent stack, which doesn’t makes it very sustainable. Some development efforts have been duplicated and a number of those packages don’t seem to be actively maintained anymore (many of those are initiatives of individual developers as side projects).

We are probably missing an intermediate level between the Xarray core library and those domain-specific packages, i.e., a few projects with a scope that is well-defined but large enough to gather individual contributors and ensure good maintenance in the mid or long term. There have been some initiatives in this way (e.g., xr-scipy, geo-xarray, geoxarray) but none of them has really gained momentum yet. It turns out to be a big challenge.

If you’d like to share your point of view on this topic, please join us in this ongoing discussion on GitHub!

Suggest or add support for xarray.DataArray and/or xarray.Dataset as core data structures in popular, third-party projects.

This suggestion is complementary to the previous one: in addition to building new packages on top of Xarray, we could also try better advocating the utility of its core data structures in existing projects.

Xarray has already a very good connection with the rest of the Python scientific / PyData stack. It wraps NumPy arrays, has an API heavily inspired from Pandas (including tools for easy conversion from/to Pandas DataFrame), provides powerful plotting capabilities built on top of Matplotlib, and integrates tightly with Dask for parallel computing. By contrast, the connection in the opposite direction (i.e., Xarray as a dependency) could be much richer, given the core functionality that Xarray provides.

More specifically, I’m thinking about two kinds of tools that could benefit from a better integration with Xarray (and vice versa):

Machine (deep) learning libraries and frameworks

Tools like Scikit-Learn, TensorFlow, PyTorch, etc. are taking up a large part of the Python scientific ecosystem. Very recently there has been much work towards better integration between Scikit-Learn and Pandas. While it is not yet clear to me which use cases would directly leverage tight integration between Xarray and Scikit-Learn, it is worth mentioning these two projects: sklearn-xarray and sklearn-xarray.

It is more obvious to see how Xarray would interplay with deep learning frameworks that handle high-dimensional tensors. Tight integration between Xarray and those frameworks is currently tricky, which may also explain why similar projects like NamedTensor and tf.contrib.labeled_tensor exist. Hopefully, planned support for flexible arrays in Xarray and up-coming improvements upstream (like NEP-0018) will greatly help here.

Scientific visualization libraries

I’m tempted to believe that the wide adoption of Pandas over the lasts years had something to do with visualization libraries like Seaborn, which allows creating fancy figures easily through an elegant API and which added support for Pandas DataFrame objects early in its development.

Regarding Xarray, the current situation is actually not bad! Perhaps the most relevant examples of third-party visualization libraries that support (and promote) Xarray data structures are the PyViz project and ArviZ. Other examples include ipyleaflet (which uses Xarray to draw velocity maps), PyGMT and probably a few other examples that I have overlooked.

This is all very exciting, but much more could be done! For example, it would be awesome if libraries like Altair could natively support Xarray structures like they support Pandas (even though there are some challenges). There are also some nice examples of Plotly figures created from data imported using Xarray, although those examples require prior conversion into Pandas DataFrame objects.

Scientific visualization is important in all disciplines ; it is pretty much all about displaying raw data + labels. The data model implemented in Xarray includes both of these. Perhaps one of the most underestimated potentials of Xarray is precisely about its data model, which, while being perfectly suited for high-dimensional data, isn’t tied to this kind of data. For example, xarray.DataArray or xarray.Dataset objects are in practice widely used for storing collections of “basic” 2-d images or 1-d data with just one or two additional dimensions representing, e.g., time or batches of experiments. The extra dimension(s) and the global or variable attributes supported by the data model both enable storing those collections of data + metadata in a tidy format, which could be leveraged in many places, especially in (interactive) visualization libraries.

Conclusion

I have tried here to give a short but documented summary on the current position of Xarray within the scientific Python ecosystem. I encourage the reader to have a quick look at all the projects that I have mentioned in this post. I humbly hope that it will contribute to convincing more people from different horizons (both users and contributors) about the potential of Xarray and the benefits of working towards its wider integration within this very rich ecosystem.