Process ~10M Row Datasets in Milliseconds In This Comprehensive Pandas Speed Guide | Towards Data Science

“Great... another article on how to make Pandas n times faster.”
I think I have said that countless times for the past two years I have been using Pandas. The most recent one I saw said, “make Pandas 71,803 times faster”.
But I won’t give you that kind of promise. I will just show you how to use Pandas in the fastest way possible. Because you can’t speed up something which is already fast. Doesn’t make sense?
Let me ask you this. Which hand do you use when I say, “Touch your right ear.” Of course, the right hand. You wouldn’t touch your right ear by putting your left hand over your head. That would be weird.
Well, using the functions, which the Pandas documentation clearly states slow instead of the fastest methods, would also be weird.
That’s why this article will show you the best practices of some of the most common data manipulation operations in Pandas. You can even consider these best practices as “common sense” because that’s how the creator of Pandas intended his library to be used.
Let’s start with the most basic operations. Specifically, we will see the fastest way of selecting rows and columns. As you know, there are two indexing operators — loc and iloc in Pandas. Even though their difference won't matter much for small datasets, they will be pretty noticeable as data size increases.
Firstly, for choosing a row or multiple rows, iloc is faster.
In contrast, loc does best for choosing columns with their labels:
>>> tps.loc[:, ["f1", "f2", "f3"]]
For sampling columns or rows, the built-in sample function is the fastest. You may think that sampling using random indices with NumPy would be faster, but that’s not the case:
# Sampling rows>>> tps.sample(7, axis=0)
# Sampling 5 columns and 7 rows>>> tps.sample(5, axis=1).sample(7, axis=0)
Most of the time, I see people using loc or iloc to replace specific values in a DataFrame:
True, this method seems the fastest because we specify the exact location of the value to be replaced instead of letting Pandas search it. However, this method is clumsy and not as fast as replace:
While speed is the first benefit of replace, the second is its flexibility. Above, we replaced all question marks with NaN - an operation that would take multiple calls with index-based replacement.
Also, replace allows using lists or dictionaries to change multiple values simultaneously:
When replacing a list of values with another, they will have a one-to-one, index-to-index mapping. You can be more specific with dictionaries like so:
It is possible to go even more granular with nested dictionaries:
Nested replacement helps when you only want to affect the values of specific columns. Here, we are replacing values only in education and income columns.
There are other benefits of replace, including regex-based replacement, which can be read in the documentation.
The golden rule for applying operations on entire columns or data frames is to never use loops. That’s the cliché advice that works. But, I get it — how do you manipulate entire arrays without looping?
The trick is to start thinking about arrays as vectors and the whole data frame as a matrix. After all, that’s how ML models accept the data — to put it bluntly, they are all multivariable vector-valued functions.
And from linear algebra, we know that operations on vectors get carried out on the entire vector, whether it is multiplication, addition, etc. Fortunately, this idea of operating on mathematical vectors is implemented in Pandas and NumPy as vectorization.
So, if you want to perform any mathematical operation on one or more columns, there is a good chance that the operation is vectorized in Pandas. For example, the built-in Python operators like +, -, *, /, ** work just like on vectors.
To get a taste of vectorization, let’s perform some operations on a massive dataset. We will choose ~1M row dataset of the Kaggle TPS September competition:
Let’s perform some mathematical operations on a few columns with Pandas apply function. That's the fastest built-in iterator of Pandas.
We will run this function on three columns with apply and check the runtime:
19.3 seconds for 1M row dataset. Not bad. Now, watch what happens when we pass columns as vectors rather than scalars. No need to modify the function:
37 milliseconds! That’s about 600 times faster than the fastest iterator. But we can do even better — vectorization is even faster when used on NumPy arrays:
Just add .values to get the underlying NumPy ndarray of Pandas Series. NumPy arrays are faster because they don't perform additional calls for indexing, data type checking like Pandas Series.
Should we call it a day? Well, Pandas has a few more tricks up its sleeve. Fair warning, though — these won’t benefit you much unless you have upwards of +1M rows.
OK, let’s get started. First, we will increase the dataset size ten times to give ourselves a challenge:
About 9 GBs — that should be enough. Oh, by the way, if datasets of this size are an issue for you, check out my article on taming those beasts:
towardsdatascience.com
Now, we will continue using our crazy_function, starting with NumPy vectorization as a baseline:
About 0.3 seconds for a 10M row dataset. When I tried this with apply, it didn't finish even in an hour.
Now, let’s improve on the runtime even more. The first candidate is Numba. We install it via pip (pip install numba) and import it. Then, we will decorate our crazy_function with its jit function. JIT stands for just in time, and it translates pure Python and NumPy code to native machine instructions, giving massive speed-ups.
Now, we run it just like any other function:
We achieved about 1.5 times speed-up. Note that Numba works best with functions that involve many native Python loops, a lot of math, and even better, NumPy functions and arrays.
Numba can do much more, including parallel computing and GPU-based compilation. Check out the documentation.
If you don’t care for additional dependencies, let me introduce you to eval function of Pandas. There are two versions - pd.eval (higher-level) and df.eval (in the context of DataFrames). Like Numba, you should have at least +10,000 samples in the DataFrame to see improvements. But once you do, you will see sizeable benefits in speed.
Let’s run our crazy_function in the context of df.eval:
Not as fast as vectorization or Numba, but it has several benefits. First, you write much less code by avoiding references to the DataFrame name. Next, it significantly speeds up non-math operations on DataFrames like boolean indexing, comparisons, and many more.
When you are not doing mathematical manipulation, evaluate your statements in pd.eval. I recommend checking out the caveats of using eval from the documentation.
Note that we were talking in the range of milliseconds even for a 10M row dataset.
Most of what we discussed has been taken from the informative enhancing speed section of the Pandas documentation. We have covered the most important and effective methods, ignoring small details. I highly suggest you go through it for more speed tips.