How to simply make an operation on pandas DataFrame faster
By: Mahdi Rasouli
Tags: Numba, Numpy, Python, Data Science, Pandas

If you ever had a big enough dataset before, you already know that sometimes a simple operation requires a lot of time. There are a bunch of methods available out there to make an operation on pandas DataFrame faster. You can use libraries like multiprocessing, modin[ray], cuDF, Dask, Spark to get the job done. Also, You can modify your algorithm to get the task executed way faster. But before going after a hard solution, let’s see if there is any simple way to gain higher speed and save some time!

As for my dataset, I am using a part of this. It has 2 numerical columns and 1 text column, consisting of some 0 in “HelpfulnessNumerator” column and some NaN in “ProfileName” column. (dataset and all notebooks have been uploaded in this Github repository.)

Numerical Operations

I want to simply check out different methods without any change in the main function. The function in the numerical part is a divide function.

def divide(t,h):

if h == 0:

return np.nan

return t/h

The first thing I learned on working on pandas DataFrame was df.iterrows()

%%timeit

iterrows_divide = []

for index,row in df.iterrows():

Time = row['Time']

HelpfulnessNumerator = row['HelpfulnessNumerator']

iterrows_divide.append(divide(Time,HelpfulnessNumerator))

5min 12s ± 31.1 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

The apply() method is another option:

%timeit df.apply(lambda row: divide(row['Time'],row['HelpfulnessNumerator']),axis = 1) 1min 25s ± 4.88 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

The best option among pandas row-wise operation is itertuples() method:

%%timeit

itertuples_divide = []

for _ in df.itertuples():

Time = getattr(_,'Time')

HelpfulnessNumerator = getattr(_,'HelpfulnessNumerator')

itertuples_divide.append(divide(Time,HelpfulnessNumerator)) 4 s ± 238 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Using row-wise operation is not compulsory for us most of the time. But if it is a must for you, df.itertuples() is the fastest way.

let’s try the map() method:

%timeit list(map(divide,df['Time'],df['HelpfulnessNumerator'])) 861 ms ± 22.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

let’s do some (fake) numpy vectorizer() method:

%timeit np.vectorize(divide)(df['Time'],df['HelpfulnessNumerator'])

636 ms ± 24.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

we can use numpy where() method for (real) numpy vectorizer:

%timeit np.where(df['HelpfulnessNumerator'] == 0 , 0 , df['Time']/df['HelpfulnessNumerator']) 21.6 ms ± 329 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

As you can see we can not use the np.where() method everywhere. But it is really fast. let's change the function a little bit:

def modified_divide(t,h):

return t/h

Now we can use this function in np.where():

%timeit np.where(df['HelpfulnessNumerator'] == 0 , 0 , modified_divide(df['Time'],df['HelpfulnessNumerator'])) 21.4 ms ± 200 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Also, we can use this function with pandas operation directly:

%timeit modified_divide(df['Time'],df['HelpfulnessNumerator']) 8.75 ms ± 114 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Is there any other option? Absolutely yes. If you can write code by Python, You can write code with Fortran. I made a notebook for Fortran magic in Jupyter Notebook. Just as an example, look at this one:

%%fortran



subroutine fortran_divide(x, y, z)

real, intent(in) :: x,y

real, intent(out) :: z



z = x/y

IF ( 0 .EQ. Y ) THEN

Z = 0

END IF





end subroutine fortran_divide %timeit np.vectorize(fortran_divide)(df['Time'],df['HelpfulnessNumerator']) 669 ms ± 22.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

We can use numba. Numba is a python library that we can use it very straight forward. It is 2X faster than np.where(), and you have not some of the limitations of np.where() here:

from numba import njit



@njit()

def divide(t, h):

res = np.empty(t.shape)

for i in range(len(t)):

if h[i] != 0:

res[i] = t[i] / h[i]

else:

res[i] = 0

return res



%timeit divide(df['Time'].values, df['HelpfulnessNumerator'].value 9.99 ms ± 133 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

Both numba and Fortran have the option to use the power of multiprocessing. It depends on the size of the DataFrame and the operation. Sometimes we can use multiprocessing and gain more speed and Sometimes multiprocessing in numba and Fortran can't help us much.

logarithmic time for each method.

As a conclusion, Do not use row-wise operations on pandas DataFrame. If it is a must, you can use df.itertuples(). Do not use df.iterrows() and df.apply(…,axis=1) never ever. You can use np.where() with some tricks most of the time. It is the best option. But if you can not use it, You can use np.vectorize() while you have numerical operations. There are some other options like numba and Fortran magics that can help us if we have a large dataset it worth spending some times to learn and use them. Also, sometimes pandas itself support what we want. it is better to check pandas at first. :)

Non-Numerical Operation

I will define a simple function to evaluate different methods. the function will add a phrase at the beginning of each record.

def edit_name(profileName):

if profileName != profileName:

return np.nan

return 'Name is {}'.format(profileName)

Using df.iterrows() is simple. But it is too slow:

%%timeit

iterrows_name=[]

for index,row in df.iterrows():

name = edit_name(row['ProfileName'])

iterrows_name.append(name) 4min 15s ± 2.45 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

Using the row-wise apply method():

%timeit df.apply(lambda row: edit_name(row['ProfileName']),axis = 1) 43.2 s ± 687 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Using itertuples() method:

%%timeit

itertuples_name=[]

for _ in df.itertuples():

name = edit_name(getattr(_,'ProfileName'))

itertuples_name.append(name) 3.78 s ± 128 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

A different kind of using apply() method:

%timeit df['ProfileName'].apply(lambda x: edit_name(x)) 1.58 s ± 86.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Let's use the map() method:

%timeit list(map(edit_name,df['ProfileName'])) 1.41 s ± 65.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Oops, np.vectorizer is not as fast as we expect. Yes, Because the operation is not numerical.

%timeit np.vectorize(edit_name)(df['ProfileName']) 2.16 s ± 155 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

There is a pandas.Series.map that can do limited tasks very fast. It can handle NaN values itself. For example, let's change the function and see the results:

def modified_edit_name(profileName):

return 'Name is {}'.format(profileName)

We can handle NaN values easily in np.where():

%timeit np.where(df['ProfileName'] == np.nan,np.nan, modified_edit_name(df['ProfileName'])) 2.69 s ± 98.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

The pandas series map method:

%timeit df['ProfileName'].map('Name is {}'.format, na_action='ignore') 1.16 s ± 50.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

logarithmic time for each method.

When it comes to text, I prefer to use map() because it is fast. And if I must have a row-wise operation, I will use df.itertuples().

As I said, After this simple optimizations, We can go after applying multiprocessing and using other libraries to have a faster operation on pandas DataFrame.

For more information about numba and Fortran magic take a look here: