How to Exclude the Outliers in Pandas DataFrame | by Khuyen Tran | Towards Data Science

Pandas is a common library for data scientists. There are different ways to process a Pandas DataFrame, but some ways are more efficient than others. This article will provide you 4 efficient ways to:
Assign new columns to a DataFrame
Exclude the outliers in a column
Select or drop all columns that start with ‘X’
Filter rows only if the column contains values from another list
Let’s get started!
Imagine we want to assign new columns whose values depend on the values of existing columns. There are different ways to do this, but the way I found most efficient is to use df.assign
We create column 3 based on column 1 and 2: data.col1 * 100 + data.col2. We then create column 4 based on column 2 and 3: data.col2 + data.col3. Everything is in one line of code!
We can check the first row to see if everything works as we expect.
col3 : 103 = 1 x 100 + 3
col3 : 106 = 103 + 3
It looks right!
Outliers are unusual values in your dataset, and they can distort statistical analyses. If you want to trim values that the outliers, one of the methods are to use df.clip.
df.clip allows us to assign values outside the boundary to boundary values. For example, if we have the data like below
And we want to assign any values below -2 to -2 and anything above 8 to 8, we can use
0    81   -22    03   -14    5Name: col0, dtype: int64
As you can see, -3 becomes -2, and 9 becomes 8. We will use this to exclude the outliers that are below .05 percentile or above .95 percentile
Cool! Now the outliers are closer to the median.
Sometimes we might want to select or drop all columns that start with a certain string. For example, if we have a Dataframe like this
We want to select the columns that start with ‘col’. Instead of dropping each of them manually, we might want to drop several columns at once. This is especially useful when you have many columns that start with ‘col’.
We use df.columns.str.startswith('col') to get a list of booleans indicating whether the column’s name starts with ‘col’
array([ True,  True, False])
Use df.loc to choose the columns that start with ‘col’
Cool! Now we just have col1 and col2 in our data!
To drop the columns that start with ‘col’, use
Now we just see the year column.
We might want to filter the rows if a column contains values from another list. For example, if we have data like this
And we want to keep the rows whose values are ‘a’, ‘c’, ‘d’, ‘e’ and exclude the rest.
To do this, use
Now we no longer see the row whose value is ‘b’ in col2 !
Congratulations! You have just learned 4 Pandas tricks to:
Assign new columns to a DataFrame
Exclude the outliers in a column
Select or drop all columns that start with ‘X’
Filter rows only if the column contains values from another list
Each trick is short but works efficiently. I hope you also find these tricks helpful.
The source code of this article could be found here.
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: