The Modin view of Scaling Pandas. Comparing Modin with Dask, Ray, Vaex... | by Devin Petersohn | Towards Data Science

Recently, a blog post was written that compared a variety of tools in a set of head to heads. I wanted to take the opportunity to talk about our vision with Modin and where we’d like to take the field of data science.
Modin (https://github.com/modin-project/modin) takes a different view of how systems should be built. Modin is designed around enabling the data scientist to be more productive. When I created Modin as a PhD student in the RISELab at UC Berkeley, I noticed a dangerous trend: data science tools are being built with hardware performance in mind, but becoming more and more complex. This complexity and performance burden is being pushed onto the data scientist. These tools optimize for processor time at the cost of the data scientist’s time.
This trend is harmful to the overall field of data science: we don’t want to force every data scientist into becoming a distributed systems expert or require that they understand low-level implementation details to avoid getting punished by performance. Modin is disrupting the data science tooling space by prioritizing the data scientists time over hardware time. To this end, Modin has:
No upfront cost to learning a new APIIntegration with the Python ecosystemIntegration with Ray/Dask clusters (Run on/with what you have!)
No upfront cost to learning a new API
Integration with the Python ecosystem
Integration with Ray/Dask clusters (Run on/with what you have!)
Modin started as a drop-in replacement for Pandas, because that is where we saw the biggest need. Using Modin is as simple as pip install modin[ray] or pip install modin[dask] and then changing the import statement:
# import pandas as pdimport modin.pandas as pd
Modin is evolving into more than just a drop-in replacement for pandas (try latest version 0.7.4), and I will discuss this later in this post. First, I’d like to do a more detailed comparison of Modin vs. other libraries since many comparisons often leave out crucial information.
These systems aim to solve different problems, so fair comparisons are challenging. I will try to stick to the qualitative properties to avoid bias.
Dask Dataframe (https://github.com/dask/dask) covers roughly 40% of the Pandas API due to how the underlying system is architected. Dask Dataframe is a row store, like many SQL systems before it. However, dataframes are not SQL Tables. This architecture allows Dask Dataframe to scale extremely well, but prevents it from supporting all Pandas APIs. Dask Dataframe also places requirements on the user to add .compute() or .persist() calls into their data science workload to manage when compute is triggered. Users are also required to specify the number of partitions. Dask is more mature with >5 years since first commit.
Modin currently covers >80% of the Pandas API. Modin is architected as a flexible column store, so the partitioning within Modin can be changed based on the operation that needs to be computed. This allows Modin to scale algorithms that strict column or row stores could not. Modin also does not place any computation or partitioning burden onto the user. Optimizations on the user’s query are run under the hood, without input from the user. Modin is still in its infancy at 2 years since first commit.
Modin integrates with Dask’s scheduling API, so Modin can run on top of Dask clusters as well. More on this comparison here.
Vaex (https://github.com/vaexio/vaex) is a query engine on top of HDF5 and Arrow memory mapped files, so if your data is already in HDF5 format, it is a solid choice. Going from other file formats (CSV, JSON, Parquet) into a Vaex compatible format takes non-trivial compute time and disk space. Vaex can support roughly 35% of the Pandas API, and does not support key Pandas features like the Pandas row Index. Vaex has had nearly 6 years since the first commit.
As of writing, Modin relies on Pandas for its HDF5 file reader, and does not support the memory-mapped style of querying that Vaex supports. Modin scales to clusters, while Vaex tries to help users avoid the need for clusters with memory mapped files.
cuDF (https://github.com/rapidsai/cudf) is a dataframe library with a Pandas-like API that runs on NVIDIA GPUs. cuDF is bottlenecked by the amount of memory (up to 48GB), and spilling over to host memory comes with a high overhead. The performance you get on the GPU is great as long as the data reasonably fits into the GPU memory. You can use multiple GPUs with cuDF and Dask, and the resulting distributed GPU dataframe will have the same general properties outlined above in the Dask section. cuDF has had 3 years since the first commit.
As of writing, Modin does not have GPU support, but Modin’s architecture is layered, so Modin could integrate GPU kernels alongside CPU. Because of the flexibility of the underlying architecture Modin can even integrate hybrid CPU/GPU/*PU kernels. With this GPU support, we will need to ensure it is implemented in a way that allows users to avoid thinking about GPU memory constraints. More on this comparison here.
Modin is the data processing engine for Ray. Ray is a low-level distributed computing framework for scaling Python code, while Modin has higher level APIs for data manipulation and querying.
Modin has disrupted the interactive data science space, centering the discussion around data science productivity over benchmark performance. As Modin continues to mature, it is evolving into a platform for empowering data science at every scale. A few big features are coming at the end of July, with more later this year. I will discuss a couple of these features, if you’d like to find more, feel free to explore our future plans on the GitHub Issue tracker or ZenHub board.
One of the major difficulties that data scientists face is that they often must switch between Jupyter notebooks to run code on different clusters or environments. This becomes cumbersome when trying to debug code locally, so we have developed an API for switching between running code locally or on multiple different cloud or cluster environments with just one line of code. Note: this API may change, and is not yet available on the latest release (0.7.4). Feedback welcome!
import modinimport modin.pandas as pdtest_cluster = modin.cloud.create(provider="AWS",                                   instance="m4.xlarge",                                   numnodes=2)with modin.local():  # run this code locally    df_local = pd.read_csv("a_file.csv")    df_local.head()with modin.cloud.deploy(test_cluster):  # run on my test cluster    df_test = pd.read_csv("s3://bucket/a_file.csv")    df_test.head()with modin.local():    df_local.count()with modin.cloud.deploy(test_cluster):    df_test.count()with modin.cloud.deploy(production):  # run this on production    df_prod = ...
This works in standard Jupyter notebooks, or any python interpreter and enables data scientists to be more productive than ever in the context of a single notebook.
These APIs have been long requested and are being built out. This will integrate with the existing Pandas engine and avoid common bottlenecks in going to/from Modin’s dataframe and numpy or sklearn. As is the case with the Pandas API, they will be drop-in replacements as well.
import modin.pandas as pdimport modin.numpy as npfrom modin.sklearn.model_selection import train_test_split
As is the case with dataframes, Modin will be layered to integrate other technologies and compute kernels. We can integrate other computation engines or kernels, e.g. nums (another RISELab project).
Data scientists love their tools, and we love data scientists. Modin is designed around enabling the data scientist to be more productive with the tools that they love. We are working to shift the focus of data science tooling to value the data scientist’s time more than the time of the hardware they are using.