You Don’t Always Have to Loop Through Rows in Pandas! | by Byron Dolon | Towards Data Science

I’ve been using Pandas for a while now, but I haven’t always used it correctly. My intuitive approach to perform calculations or edit my data tends to start with this question:
How can I loop through (iterate) over my DataFrame to do INSERT_ANY_TASK_HERE?
Iterating over rows in a DataFrame may work. In fact, I wrote a whole piece on how to edit your data in Pandas row by row.
The reason I did this is because I had a multi-layered calculation that for the life of me I couldn’t figure out how to solve without looping. I had multiple conditions, one of which involved taking a column value that had the name of another column in the DataFrame which was to be used in a calculation.
Iterating over the DataFrame was the only way I could think of to resolve this problem. But it shouldn’t be the method you always go to when working with Pandas.
In fact, Pandas even has a big red warning on how you shouldn’t need to iterate over a DataFrame.
Iterating through pandas objects is generally slow. In many cases, iterating manually over the rows is not needed and can be avoided (using) a vectorized solution: many operations can be performed using built-in methods or NumPy functions, (boolean) indexing.
Most of the time, you can use a vectorized solution to perform your Pandas operations. Instead of using a “for loop” type operation that involves iterating through a set of data one value at a time, vectorization means you implement a solution that operates on a whole set of values at once. In Pandas, this means that instead of calculating something row by row, you perform the operation on the entire DataFrame.
The focus here isn’t only on how fast the code can run with non-loop solutions, but on creating readable code that leverages Pandas to the full extent.
Now, let’s go through a couple examples to help reframe the initial thought process from “how do I loop through a DataFrame?” to the real question of “how do I perform this calculation with the tools from Pandas?”.
The data we’re going to use comes from an Animal Crossing user review data set from Kaggle. We’ll import the data and create two DataFrames, one called “old” and one called “new”. Then, to get started on the basics of alternative solutions to for loops, we’ll perform some operations with both a for loop and a vectorized solution and compare the code. (To understand the logic behind the for loop codes below, please check out my previous piece on it, as it already has an in-depth explanation on the subject.)
import pandas as pdold = pd.read_csv('user_reviews.csv')new = pd.read_csv('user_reviews.csv')
Let’s create a new column called “qualitative_rating”. This way, we can create some broad categories to label each user review as “bad”, “ok”, and “good”. A “bad” review will be any with a “grade” less than 5. A good review will be any with a “grade” greater than 5. Any review with a “grade” equal to 5 will be “ok”.
To implement this using a for loop, the code would look like this:
# if then elif else (old)# create new column old['qualitative_rating'] = ''# assign 'qualitative_rating' based on 'grade' with loopfor index in old.index:    if old.loc[index, 'grade'] < 5:        old.loc[index, 'qualitative_rating'] = 'bad'    elif old.loc[index, 'grade'] == 5:        old.loc[index, 'qualitative_rating'] = 'ok'    elif old.loc[index, 'grade'] > 5:        old.loc[index, 'qualitative_rating'] = 'good'
The code is easy to read, but it took 7 lines and 2.26 seconds to go through 3000 rows.
Instead, a better solution would look like this:
# if then elif else (new)# create new columnnew['qualitative_rating'] = ''# assign 'qualitative_rating' based on 'grade' with .locnew.loc[new.grade < 5, 'qualitative_rating'] = 'bad'new.loc[new.grade == 5, 'qualitative_rating'] = 'ok'new.loc[new.grade > 5, 'qualitative_rating'] = 'good'
This time, the code to add the qualitative ratings was only composed of 3 lines, and it only took 68 milliseconds. I also used the “.loc” DataFrame function again, but this time, I used it “properly”. By that, I mean instead of using a looped “if-else” solution, I assigned the “bad”, “ok”, and “good” qualitative rating directly from the “.loc” selection.
Our next new column “len_text” will show the number of characters in each review, so we can compare the length of different reviews in the data set.
To implement this using a for loop, the code would look like this:
# create column based on other column (old)# create new columnold['len_text'] = ''# calculate length of column value with loopfor index in old.index:    old.loc[index, 'len_text'] = len(old.loc[index, 'text'])
Again, 2 lines and 2.23 seconds for this calculation is not that long. But instead of going through each row to find the length, we could use a solution that only requires one line:
# create column based on other column (new)# create new columnnew['len_text'] = ''# calculate length of column value by converting to strnew['len_text'] = new['text'].str.len()
Here, we take an existing column’s values, turn them into strings, and then use “.len()” to get the number of characters in each string. This solution only took 40 milliseconds to run.
Now let’s create a new column called “super_category”. Here, we’ll identify if people qualify as a “super reviewer”, or in this case, if the length of their review is greater than 1000 characters. We’ll also mark a super reviewer as a “super fan” if the review “grade” is greater than or equal to 9, and a “super hater” if the review “grade” is less than or equal to 1. Everyone else will be categorized as “normal”.
Implementing this with a for loop would look like this:
# new column based on multiple conditions (old)# create new columnold['super_category'] = ''# set multiple conditions and assign reviewer category with loopfor index in old.index:    if old.loc[index, 'grade'] >= 9 and old.loc[index, 'len_text'] >= 1000:        old.loc[index, 'super_category'] = 'super fan'    elif old.loc[index, 'grade'] <= 1 and old.loc[index, 'len_text'] >= 1000:        old.loc[index, 'super_category'] = 'super hater'    else:        old.loc[index, 'super_category'] = 'normal'
This works, but let’s cut it in half:
# new column based on multiple conditions (new)# create new columnnew['super_category'] = 'normal'# set multiple conditions and assign reviewer category with .locnew.loc[(new['grade'] == 10) & (new['len_text'] >= 1000), 'super_category'] = 'super fan'new.loc[(new['grade'] <= 1) & (new['len_text'] >= 1000), 'super_category'] = 'super hater'
Here, we use the “&” operator inside our “.loc” function to implement the two conditions at once. The vectorized solution completed in 63 milliseconds, which again is significantly faster than the loop method, which took 2.23 seconds.
These were some basic operations used to expand the existing data with some of our own custom analysis. Yes, we could have done everything with loops, and you can even see that the same structure applied across many different operations. But Pandas comes with a lot of built-in methods specifically for the operations that we frequently need to perform.
Going through this helped me retrain my brain away from always going to for loops as a solution to looking for better ways to accomplish all sorts of operations.
I hope it helped you do the same!