Selecting Rows with .loc. A beginner’s guide to selecting subsets... | by Ednalyn C. De Dios | Towards Data Science

As data scientists, we spent most of our time wrangling knee-deep in manipulating data using Pandas. In this post, we’ll be looking at the .loc property of Pandas to select rows based on some predefined conditions.
Let’s open up a Jupyter notebook, and let’s get wrangling!
We will be using the 311 Service Calls dataset1 from the City of San Antonio Open Data website to illustrate how the different .loc techniques work.
Before we get started, let’s do a little housekeeping first.
Nothing fancy going on here. We’re just importing the mandatory Pandas library and setting the display options so that when we inspect our dataframe, the columns and rows won’t be truncated by Jupyter. We’re setting it up so that every output within a single cell is displayed and not just the last one.
In the code above, we’re defining a function that will show us the number of missing or null values and their percentage.
Let’s load the data into a dataframe.
Doing a quick df.head() we’ll see the first five rows of the data:
And df.info() will let us see the dtypes of the columns.
Then, show_missing(df) shows us if there are any missing values in the data.
Let’s select rows where the 'Dept' column has null values and also filtering a dataframe where null values are excluded.
First, we did a value count of the column ‘Dept’ column. The method .value_counts() returns a panda series listing all the values of the designated column and their frequency. By default, the method ignores NaN values and will not list it. However, if you include the parameter dropna=False it will include any NaN values in the result.
Next, the line df_null = df.loc[df['Dept'].isnull()] tells the computer to select rows in df where the column 'Dept' is null. The resulting dataframe is assigned to df_null , and all its rows will NaN as values in the ‘Dept’ column.
Similarly, the line df_notnull = df.loc[df['Dept'].notnull()] tells the computer to select rows in df where the column 'Dept' is not null. The resulting dataframe is assigned to df_notnull , and all its rows will not have any NaN as values in the ‘Dept’ column.
The general syntax for these two techniques are:
df_new = df_old.loc[df_old['Column Name'].isnull()]df_new = df_old.loc[df_old['Column Name'].notnull()]
The 'Late (Yes/No)' column looks interesting. Let’s take a look at it!
Again, we did a quick value count on the 'Late (Yes/No)' column. Then, we filtered for the cases that were late with df_late = df.loc[df['Late (Yes/No)'] == 'YES']. Similarly, we did the opposite by changing 'YES' to 'NO' and assign it to a different dataframe df_notlate.
The syntax is not much different from the previous example except the addition of == sign between the column and the value we want to compare. It basically asks, for every row, if the value on a particular column (left side) matches the value that we specified (right-side). If the match is True, it includes that row in the result. If the match is False, it ignores it.
Here’s the resulting dataframe for df_late:
And here’s the one for df_notlate:
The general syntax for this technique is:
df_new = df_old.loc[df_old['Column Name'] == 'some_value' ]
We’ve learned how to select rows based on ‘yes’ and ‘no.’ But what if the values are not binary? For example, let’s look at the ‘Category’ column:
One hundred ninety-two thousand one hundred ninety-seven rows or records do not have a category assigned, but instead of NaN, empty, or null value, we get 'No Category' as the category itself. What if we want to filter these out? Enter: the != operator.
As usual, we did customary value counts on the 'Category' column to see what we’re working with. Then, we created the df_categorized dataframe to include any records in the the df dataframe that don’t have 'No Category' as their value in the 'Category' column.
Here’s the result of doing a value count on the 'Category' column of the df_categorized dataframe:
As the screenshot above shows, the value counts retained everything but the ‘No Category.’
The general syntax for this technique is:
df_new = df_old.loc[df_old['Column Name'] != 'some_value' ]
Let’s consider the following columns, 'Late (Yes/No)' and 'CaseStatus':
What if we wanted to know which open cases right now are already passed their SLA (service level agreement)? We would need to use multiple conditions to filter the cases or rows in a new dataframe. Enter the & operator.
The syntax is similar to the previous ones except for the introduction of the & operator in between parenthesis. In the line df_late_open = df.loc[(df[‘Late (Yes/No)’] == ‘YES’) & (df[‘CaseStatus’] == ‘Open’)], there are two conditions:
(df[‘Late (Yes/No)’] == ‘YES’)(df[‘CaseStatus’] == ‘Open’)
(df[‘Late (Yes/No)’] == ‘YES’)
(df[‘CaseStatus’] == ‘Open’)
We want both of these to be true to match a row, so we included the operator & in between them. In plain speak, the & bitwise operator simply means AND. Other bitwise operators include pipe| sign for OR and the tilde ~ for NOT. I encourage you to experiment using these bitwise operators to get a good feel of what all they can do. Just remember to enclose each condition between parenthesis so that you don’t confuse Python.
The general syntax for this technique is:
df_new = df_old.loc[(df_old['Column Name 1'] == 'some_value_1') & (df['Column Name 2'] == 'some_value_2')]
Let’s look at the value count for the 'Council District' column:
What if we wanted to focus on districts #2, #3, #4, and #5 because they’re in south San Antonio, and they’re known for getting poor service from the city? (I’m so totally making this up by the way!) In this case, we could use the .isin() method like so:
Remember to pass your choices inside the .isin() method as a list like ['choice1', 'choice2', 'choice3'] because otherwise, it will cause an error. For integers like in our example, it is not necessary to include quotation marks because quotation marks are for string values only.
Here’s the result of our new dataframe df_south:
The general syntax for this technique is:
df_new = df_old.loc[df_old[Column Name'].isin(['choice1', 'choice2', 'choice3'])]
And that’s it! In this post, we loaded the 311 service calls data into a dataframe and created subsets of data using the .loc method.
Thanks for reading! I hope you enjoyed today’s post. Data wrangling, at least for me, is a fun exercise because this is the phase where I first get to know the data and it gives me a chance to hone my problem-solving skills when faced with really messy data. Happy wrangling folks!
Stay tuned!
You can reach me on Twitter or LinkedIn.
[1] City of San Antonio Open Data. (May 31, 2020). 311 Service Calls. https://data.sanantonio.gov/dataset/service-calls