Efficient Pandas: Apply vs Vectorized Operations | by Soner Yıldırım | Towards Data Science

Pandas is one of the most commonly used data analysis and manipulation libraries in data science ecosystem. It offers plenty of functions and methods to perform efficient operations.
What I like most about Pandas is that there are almost always multiple ways to accomplish a given task. However, we should consider time and computational complexity when selection a method from available options.
It is not enough just to complete a given task. We should make it as efficient as possible. Thus, having a comprehensive understanding of how functions and methods work is of crucial importance.
In this article, we will do examples to compare the apply and applymap functions of pandas to vectorized operations. The apply and applymap functions come in hand for many tasks. However, as the size of data increases, time becomes an issue.
Let’s create a sample dataframe with 100k rows. We first need to import the required libraries.
import numpy as npimport pandas as pdimport timeit df = pd.DataFrame({    'cola':np.random.randint(1,100, size=100000),    'colb':np.random.randint(100,1000, size=100000),    'colc':np.random.random(100000)})df.shape(100000, 3)
We will use the timeit library to measure the time it takes to execute an operation.
The apply function performs row-wise or column-wise operations by looping through the elements. The applymap function works in similar way but performs a given task on all the elements in the dataframe.
The following code will square each number in “cola” column.
%%timeitdf['cola'].apply(lambda x: x**2)best of 3: 54.4 ms per loop
It takes 54.4 miliseconds. Let’s complete the same task with a vectorized operation.
%%timeitdf['cola'] ** 2best of 3: 1.15 ms per loop
It is approximately 50 times faster on 100k rows.
The next example includes a task to find the minimum value in each row of the dataframe.
%%timeitdf.apply(lambda x: x.min(), axis=1)best of 3: 3.01 s per loop
It takes 3 seconds to execute with the apply function.
%%timeitdf.min(axis=1)best of 3: 1.58 ms per loop
It is still on the millisecond level with vectorized operation. There is a huge difference. The reason why the difference has substantially increased is that apply function loops through each column for every row.
Let’s slightly increase the complexity of the operation. We want to find the difference between the highest and lowest values in a row.
The operation is done with the apply function as below:
%%timeitdf.apply(lambda x: x.max() - x.min(), axis=1)best of 3: 5.29 s per loop
We use a lambda expression to calculate the difference between the highest and lowest values. The axis is set to 1 to indicate the operation is done on the rows. This operation takes 5.29 seconds to execute.
The same task can be accomplish by two vectorized operations.
%%timeitdf.max(axis=1) - df.min(axis=1)best of 3: 3.86 ms per loop
We use the max and min functions as vectorized operations. The axis parameter is 1 to indicate we need the min or max value in a row. If we do not specify the axis parameter as 1, the min or max value in a column is returned.
The vectorized version takes 3.86 milliseconds to execute which is more than a thousand times faster.
The next example compares the applymap function to a vectorized operation. The following code doubles each element in the dataframe.
%%timeitdf.applymap(lambda x: x * 2)best of 3: 93.6 ms per loop
It takes 93.6 milliseconds.
%%timeitdf * 2best of 3: 1.03 ms per loop
The same operation takes about 1 millisecond which is 90 times faster than the applymap function.
We have covered some examples to compare vectorized operations and the apply and applymap functions.
For small sized data, the time difference is usually negligible. However, as the size increases, the difference starts to become noticeable. We are likely to deal with vast amount of data so time should always be taken into consideration.
What makes the apply and applymap functions is the way they execute a given task. The given operation is done by looping through the elements which slows down the execution as the data size becomes large.
Thank you for reading. Please let me know if you have any feedback.