How To Select Rows From Pandas DataFrame Based on Column Values | Towards Data Science

Selecting rows from a DataFrame is probably one of the most common tasks one can do with pandas. In today’s article we are going to discuss how to perform row selection over pandas DataFrames whose column(s) value is:
Equal to a scalar/string
Not equal to a scalar/string
Greater or less than a scalar
Containing specific (sub)string
A member of an iterable (e.g. a list)
Additionally, we will discuss how to combine multiple conditions together.
Before start discussing the different ways that can be used for selecting rows from pandas DataFrames, first let’s create an example DataFrame that will reference throughout this post to demonstrate a few concepts.
import numpy as npdf = pd.DataFrame( [  (73, 15, 55, 33, 'foo'),  (63, 64, 11, 11, 'bar'),  (56, 72, 57, 55, 'foo'), ], columns=['A', 'B', 'C', 'D', 'E'],)print(df)#     A   B   C   D    E# 0  73  15  55  33  foo# 1  63  64  11  11  bar# 2  56  72  57  55  foo
In the following sections we are going to discuss and showcase how to select specific rows from a DataFrame based on a variety of possible conditions.
Let’s assume that we want to select only rows with one specific value in a particular column. We can do so by simply using loc[] attribute:
>>> df.loc[df['B'] == 64]    A   B   C   D    E1  63  64  11  11  bar
We can even omit loc and provide the boolean condition when indexing the pandas DataFrame as shown below:
>>> df[df['B'] == 64]    A   B   C   D    E1  63  64  11  11  bar
Note that instead of df['B'] you can also reference column B using df.B. For example the statement below is equivalent to the one above.
>>> df[df.B == 64]
In the context of Python, it is a common practise to name such boolean conditions as mask that we then pass to DataFrame when indexing it.
>>> mask = df.B == 64>>> df[mask]    A   B   C   D    E1  63  64  11  11  bar
Having introduced a few possible ways you can use to select rows based on column value equality to a scalar, we should highlight that loc[] is the way to go. This is simply because df[mask] will always dispatch to df.loc[mask] which means using loc directly will be slightly faster.
Going forward, you may wish to select only a subset of rows whose values in a specific column is not equal to a scalar. To do so, we simply need to follow the same convention as before:
>>> df.loc[df['B'] != 64]    A   B   C   D    E0  73  15  55  33  foo2  56  72  57  55  foo
This is a very similar use case with the previous one. In this occasion we just need to use the required operator to perform a comparison. For instance,
>>> df.loc[df['B'] >= 64]    A   B   C   D    E1  63  64  11  11  bar2  56  72  57  55  foo
One of the most common use-cases is when you need to filter a DataFrame so that you only keep rows containing a specific (sub)string in a particular column.
As in the previous use-case, we can use loc to do so:
>>> df.loc[df['E'].str.contains('oo')]    A   B   C   D    E0  73  15  55  33  foo2  56  72  57  55  foo
Now let’s assume we want to select only those rows whose values in a specific column are included in a list. To do so, we simply need to use isin() as shown below.
>>> df.loc[df['B'].isin([64, 15])]    A   B   C   D    E0  73  15  55  33  foo1  63  64  11  11  bar
Now if you want to select rows whose value is not in an iterable (e.g. a list), then you can simply use the negation character ~ :
>>> df.loc[~df['B'].isin([64, 15])]    A   B   C   D    E2  56  72  57  55  foo
There’s a certain chance you may want or have to combine multiple conditions together when selecting rows from a pandas DataFrame.
To do so, we need to make use of the & operator
>>> df.loc[(df['A'] >= 59) & (df['E'].isin(['foo', 'boo']))]    A   B   C   D    E0  73  15  55  33  foo
In today’s article we discussed how to perform row selection over pandas DataFrames and showcased specific use-cases and examples.
In most of the sections we essentially used labelled indexing which is performed using loc. Note that pandas has an additional property called iloc that is usually used to perform positional indexing. It is important to understand the difference between the two and I’d suggest you to read the article below that explains what each does in detail.