How To Make Your Pandas Loop 71.803 Times Faster
By: Benedikt Droste
Tags: Data Science, Tutorial, Pandas

If you use Python and Pandas for data analysis, it will not be long before you want to use a loop the first time. However, even for small DataFames it is time-consuming to use the standard loop and you will quickly realize that it can take a long time for larger DataFrames. When I first waited more than half an hour to execute the code, I looked for alternatives that I would like to share with you.

The standard loop

DataFrames are Pandas-objects with rows and columns. If you use a loop, you will iterate over the whole object. Python can´t take advantage of any built-in functions and it is very slow. In our example we got a Dataframe with 65 columns and 1140 rows. It contains soccer results for the seasons 2016 - 2019. We want to create a new column that indicates whether a particular team has played a draw. We might start like this:

Since we got every match from the Premier League in our DataFrame, we have to check if the team of interest (Arsenal) played and if that applies, whether they were the home or away team. As you can see, this loop was very slow and took 20,7 seconds to execute. Let´s look how we can be more efficient.

The Pandas Built-In Function: iterrows() — 321 times faster

In the first example we looped over the entire DataFrame. iterrows() returns a Series for each row, so it iterates over a DataFrame as a pair of an index and the interested columns as Series. This makes it faster than the standard loop:

The code took 68 milliseconds to run which is 321 times faster than the standard loop. However, many people advise against using it because there are still faster options and iterrows() does not preserve dtypes across the rows. That means if you use iterrows() on your DataFrame dtypes can be changed which can cause a lot of problems. Do preserve dtypes you can also use itertuples() . We will not go into detail here because we want to pay attention to efficiency. Here you can find the official documentation:

The apply() Method — 811 times faster

apply is not faster in itself but it has advantages when used in combination with DataFrames. This depends on the content of the apply expression. If it can be executed in Cython space, apply is much faster (which is the case here).

We can use apply with a Lambda function. All we have to do it to specify the axis. In this case we have to use axis=1 because we want to perform a column-wise operation:

This code is even faster than the previous methods and took 27 milliseconds to be finished.

Pandas Vectorization — 9280 times faster

Now we can come to a new topic. We use the advantages of vectorization to create really fast codes. The point is to avoid Python-level loops like in the examples before [1] and to use optimized C code which uses the memory much more efficient. We just need to modify the function slightly:

Now we can create with Pandas series as input thew new column:

In this case we don´t even need a loop. All we have to do is to adjust the content of the function. Now we can directly pass Pandas series to our function which causes the enormous speed gain.

Numpy Vectorization — 71.803 times faster

In the previous example we passed Pandas series to our function. By adding .values we receive a Numpy array:

Numpy arrays are so fast because we got the benefits of locality of reference [2]. Our code took 0,305 milliseconds to run and was 71803 times faster than the standard loop used in the beginning.

Conclusion

If you use Python, Pandas and Numpy for data analysis, there will always be some room for improving your code. We compared five different methods do add a new column to our DataFrame based on some calculation. We have noticed enormous differences in terms of speed: