Here’s how you can speedup Pandas with cuDF and GPUs | by George Seif | Towards Data Science

Want to be inspired? Come join my Super Quotes newsletter. 😎
Pandas is the go-to data storage library for Data Scientists and Machine Learning practitioners. It allows for easy management, exploration, and manipulation of many different types of data, including both numbers and text.
Even on its own, Pandas is already a significant step up from Python in terms of speed. Whenever you find your Python code running slow, especially if you see a lot of for-loops, it’s always a good idea to change the code which performs data exploration and filtering into Pandas functions. Pandas functions are specifically developed with vectorized operations that run at top speed!
Still, even with that speedup, Pandas is only running on the CPU. With consumer CPUs typically having 8 cores or less, the amount of parallel processing, and therefore the amount of speedup that can be achieved, is limited. Modern datasets can have as much as millions, billions, or even trillions of data points that need processing — 8 cores just isn’t going to cut it.
Luckily, following the popular success of GPU acceleration in Machine Learning, there’s been a strong push towards getting Data Analysis libraries onto GPUs. The cuDF library is a step in that direction.
cuDF is a Python-based GPU DataFrame library for working with data including loading, joining, aggregating, and filtering data. The move to GPU allows for massive acceleration due to the many more cores GPUs have over CPUs.
cuDF’s API is a mirror of Pandas’s and in most cases can be used as a direct replacement. This makes it very easy for Data Scientists, Analysts, and Engineers to integrate it into their workflow.
All that needs to be done is to convert your Pandas DataFrame into a cuDF one and voila you have GPU speedup! cuDF will support most of the common DataFrame operations that Pandas does, so much of the regular Pandas code can be accelerated without much effort.
To get started with an example of using cuDF, we can install the library via conda:
conda install -c nvidia -c rapidsai -c numba -c conda-forge -c defaults cudf
Keep in mind that for the following experiments, the machine we’re testing on has the following specs:
i7–8700k CPU
1080 Ti GPU
32 GB of DDR4 3000MHz RAM
CUDA 9.2
We’re going to load a big dataset of randomized numbers and compare the speed of various Pandas operations vs doing the same thing on GPU with cuDF.
We can do this all in a Python Jupyter Notebook. Let’s start by initializing our DataFrames: one for Pandas and one for cuDF. The DataFrame has over 100 Million points!
For our first test, let’s measure how long it takes to calculate the mean value of the ‘a’ variable in our data in Pandas vs. cuDF. The %timeit command allows us to measure the speed of Python commands in a Jupyter Notebook.
Average run-time is shown in the code comments above. We got a 16X speedup just like that!
Now, what about something a bit more complicated like doing a huge merge?! Let’s merge our DataFrame on itself on the DataFrame’s ‘b’ column.
Merging here is a very large operation since Pandas will have to look for and match the common values — a time-consuming operation for a dataset with 100 Million rows! GPU acceleration will make this easier since we have many more parallel processes that can work together.
Here’s the code and results:
Even with a fairly powerful i7–8700k CPU, Pandas took 39.2 seconds on average to complete the merge. On the other hand, our friend cuDF on the GPU took only 2.76 seconds, a much more manageable time! That’s a total of over 14X speedup.
So there you have it! That’s how you can speedup Pandas on GPU using cuDF.
If you’re hungry for more, not to worry! There’s much more speed to be had with cuDF and the rest of the RAPIDS API. The best place to start is the official GitHub page!
Follow me on twitter, where I post all about the latest and greatest AI, Technology, and Science! Connect with me on LinkedIn too!