Different Ways To Filter Pandas DataFrame | by Chi Nguyen | Towards Data Science

Subset selection is one of the most frequently performed steps in data manipulation. Pandas by far offers many different ways to filter your dataframes to get your selected subsets of data. In this article, I will show you some cases that I encounter the most when manipulating data.
Before coming to details, I will first create a sample dataframe.
#Create a simple dataframedf = pd.DataFrame({    'name' :     [ 'Chi', 'Alex', 'Sam', 'Hoang', 'Linh', 'Dung', 'Anh'],    'function' : [ 'Marketing', 'Tech', 'Tech', 'Finance', 'Finance', 'Marketing', 'HR'],    'address' :  [ 'Hanoi', 'Saigon', 'Hanoi', 'Saigon', 'Hanoi', 'Hanoi', 'Saigon'],    'gender' :   ['F', 'M', 'F', 'M', 'M', 'M', 'F'],    'favourite_fruit' : [ 'apple', 'banana', 'orange', 'watermelon', 'grape', np.NAN, 'kumquat'],    'age' : [20,25,21,26,30,31,23]})
Here is an overview of my df dataframe:
There are few ways to select range of rows based on some specific conditions.
#Using dataframe methoddf[(df.age >=25) & (df.address == 'Hanoi')] #Using query functiondf.query('age >= 25 & address == "Hanoi"') #Using loc functiondf.loc[(df.age >= 25) & (df.address == 'Hanoi')] 
All of these methods produce same result as follow:
In this section, let’s find out several ways of using loc and iloc to filter dataframe.
df.loc[0:3] 
Output:
df.iloc[0:3]
Output:
As you can see, there is a difference in result between using loc and iloc. The reasons for this difference are due to:
loc does not return output based on index position, but based on labels of the index.
iloc selects rows based on position in the index. It also means that iloc only works with integers.
You can refer to the example below for more reference. In this example, I will set index to column ‘name’, so that you can understand the usage and my explanation more clearly.
df4 = df.copy()df4.set_index('name', inplace = True) 
Output:
Now, let’s try again the example above to see how loc works with index labels.
index = ['Chi', 'Sam', 'Hoang']df4.loc[index] 
Output:
In case there are hundreds of columns and you just want to select a few of them, you can add a list of columns after the condition in the loc statement.
In the example below, I will try to get ‘name’, ‘gender’ and ‘address’ of people whose ages ≥ 25.
df.loc[(df.age >=25), ['name','gender','address']] 
Output:
#Using ilocdf4.iloc[:,[1,2,3]]#Using loc and column namesdf4.loc[:, ['address','gender','favourite_fruit']]
Output:
#Using ilocdf4.iloc[2:6,3:4]#Using locdf4.loc['Sam':'Dung',['favourite_fruit']]
Output:
#Using ilocdf4.iloc[[1,3],[2,4]]#Using locdf4.loc[['Alex','Hoang'],['gender','age']]
Output:
column_start = df4.columns.get_loc('address')column_end = df4.columns.get_loc('favourite_fruit')df4.iloc[1:4,column_start:column_end] 
You can also perform similar task using .loc method. These 2 operations produce the same result.
row_start = df4.index[1]row_end = df4.index[3]df4.loc[row_start:row_end, 'address':'gender']
Output:
There are a few ways to select a single element in the dataset, including loc, iloc, iat and at methods.
Look at the df4 data, and find out what Hoang’s favourite fruit is?
row_position = df4.index.get_loc('Hoang')column_position = df4.columns.get_loc('favourite_fruit')#Using ilocdf4.iloc[row_position,column_position]#Using iatdf4.iat[row_position,column_position]#Using locdf4.loc['Hoang','favourite_fruit'] #Using atdf4.at['Hoang','favourite_fruit'] 
Basically, these 4 ways above retrieve a same output, which is ‘watermelon’. However, there is a slight difference in running speed of these methods. As a result, at and iat methods are a bit faster than loc and iloc methods. You can take a look at figure 12 below for more reference.
Above are some of my tips when doing data filtering. Though it is quite simple, I hope it can be an useful source of reference for your work in the future.