Faster Pandas with parallel processing: cuDF vs. Modin | by Déborah Mesquita | Towards Data Science

With Pandas, by default we can only use a single CPU core at a time. This is fine for small datasets, but when working with larger files this can create a bottleneck.
It's possible to speed-up things by using Parallel processing, but if you never wrote a multithreaded program don't worry: you don't need to learn how to do it. Some new libraries that can do it for us. Today we're going to compare two of them: cuDF and Modin. They both use pandas-like APIs so we can start using them just by changing the import statement.
cuDF is a GPU DataFrame library that provides a pandas-like API allowing us to accelerate our workflows without going into details of CUDA programming. The lib is part of RAPIDS, a suite of open source libraries that uses GPU-acceleration and integrates with popular data science libraries and workflows to speed up Machine Learning.
The API is really similar to pandas, so in most cases, we just need to change one line of code to start using it:
import cudf as pds = pd.Series([1,2,3,None,4])df = pd.DataFrame([('a', list(range(20))),                     ('b', list(reversed(range(20)))),                     ('c', list(range(20)))])df.head(2)df.sort_values(by='b')df['a']df.loc[2:5, ['a', 'b']]s = pd.Series([1,2,3,None,4])s.fillna(999)df = pd.read_csv('example_output/foo.csv')df.to_csv('example_output/foo.csv', index=False)
cuDF is a single-GPU library. For Multi-GPU they use Dask and the dask-cudf package, which is able to scale cuDF across multiple GPUs on a single machine, or multiple GPUs across many machines in a cluster [cuDF Docs].
Modin also provides a pandas-like API that uses Ray or Dask to implement a high-performance distributed execution framework. With Modin you can use all of the CPU cores on your machine. It provides speed-ups of up to 4x on a laptop with 4 physical cores [Modin Docs].
We'll be using the Maingear VYBE PRO Data Science PC and I'm running the scripts using Jupyter. Here are the technical specs:
125gb RAM
i9–7980XE, 36 cores
2x TITAN RTX 24GB
The dataset used for the benchmarks was the brazilian 2018 higher education census.
Let's go ahead and read a 3gb CSV file using Pandas, cuDF and Modin. We'll run it 30 times and get the mean values.
Modin is the winner with less than 4s on average. It automatically distributes the computation across all of the system’s available CPU cores and we have 36 cores so maybe this is the reason why 🤔?
In this benchmark we will fill the NaN values of the DataFrame.
Modin is the winner for this benchmark too. And cuDF is the lib that takes more time to run on average.
Let's group the rows to see how each library behaves.
Here cuDF is the winner and Modin has the worst performance.
To answer this question I think we have to consider the methods we most use in our workflows. In today's benchmark reading the file was much faster using Modin, but how many times do we need to use the read_csv() method in our ETL? By contrast, in theory, we would use the groupby() method more frequently, and in this case, the cuDF library had the best performance.
Modin is pretty easy to install (we just need to use pip) and cuDF is harder (you'll need to update the NVIDIA drivers, install CUDA and then install cuDF using conda). Or you can skip all these steps and get a Data Science PC because it comes with all RAPIDS libraries and software fully installed.
Also, both Modin and cuDF are still in the early stages and they don't have the complete coverage of the entire Pandas API yet.
If you want to dive deep into cuDF, the 10 Minutes to cuDF and Dask-cuDF is a good place to start.
For more info about Modin, this blog post explains more about parallelizing Pandas with Ray. If you want to dive even deeper there is the technical report on Scaling Interactive Data Science Transparently with Modin, which does a great job of explaining the technical architecture of Modin.