Best Python Tricks for Awesome Data Analysis | Towards Data Science

Quicker Exploratory Data Analysis! 💡
Using Python Pandas built-in methods, it is super-easy and straightforward to analyze the dataset and summarize its important characteristics.
The commonly used Pandas DataFrame methods are .head(), .tail(), .info(), .describe() , .plot() and .value_counts().
Here I am presenting four more methods, which I use to further speed up the EDA. ⚡
In the next five minutes, you will certainly master these 4 methods.
Consider an example of students' test scores. You can create a DataFrame as shown below and follow along.
Let’s get started!!
In many situations, we want to split the data set into groups and do something with those groups. The Pandas method groupby() is used to split the data in DataFrame into groups.
Instead of using groupby() and aggregation methods together, create a groupby() object. Ideally, this object can be used directly when required.
Let’s divide the given DataFrame into groups based on the column “City”
df_city_group = df.groupby("City")
This creates an object df_city_group which can be combined with different aggregations such as min(), max(), mean(), describe(), and count(). An example is shown below.
To get the subset of DataFrame df where “City” is Berlin, you simply need to use the method .get_group()
This doesn’t require creating a copy of each sub-DataFrame for every group, so will be memory-efficient. This is because it uses data structures already available in the object.
Slicing with .groupby() is 2X faster than with logical comparison!!
.groupby() and .get_group() are real game-changers when it comes to slicing the DataFrame. ⏳
Here is an interesting read about Python One-Liners for efficient coding!
towardsdatascience.com
Often, we are interested in knowing the Top 3 or Top 5 entries of the DataFrame based on values from a specific column.
For example, getting the Top 3 scorers from an exam or getting Top 5 most-watched movies from a dataset.
Pandas .nlargest() is the easiest answer with its syntax.df.nlargest(N, column_name, keep = ‘first’ )
Using the method .nlargest(), the DataFrame rows containing Top ‘N’ values of a specified column can be retrieved.
In the above example, let’s get the rows of the DataFrame “df” with Top 3 “Maths_Score”.
If there is a tie between two values, the additional and optional parameters can be modified. It takes the values “first”, “last” and “all” to retrieve the first, last, and all values from the tie.
1 advantage of this method is, you don’t need to specifically sort the DataFrame. The method .nlargest() does it on behalf of you.
Similar to the Top 3 or 5, sometimes we also need the Last 5 records in the DataFrame.
for example, getting 5 movies with the lowest rating or the 5 students scoring the lowest grades in an exam.
.nsmallest() can be the easiest answer with its syntax.df.nsmallestst(N, column_name, keep = ‘first’ )
Using the method .nsmallest(), the DataFrame rows containing bottom ‘N’ values of a specified column can be retrieved.
In the same example, let’s get the rows of the DataFrame “df” with the 3 lowest “Maths_Score”.
However, this was all about the easiest methods. But, from the execution time ⏳ point of view, the methods .nlargest() and .nsmallest() are poor choices.
sort_values() is 2.8X faster than nlargest()!!
Execution time tells a different story.
🚩 From my experiences, I recommend using the method .sort_values() for speedy EDA.
The comparison operators <, >, <=, >=, ==, != and their wrappers .lt(), .gt(), .le(), .ge(), .eq() and .ne() respectively, are quite handy in cases like,
Comparing the DataFrame to a base valueSuch comparison results in the Series of Boolean values which can be used as indicators for later purposes.
Slicing the DataFrame based on comparisonA subset can be extracted from the DataFrame based on comparison with a value.
Creating a new column in the existing DataFrame based on a comparison of two columns.
All these scenarios are explained in the example below,
Read more documentation on wrappers for the comparison operators here.
These Python tricks even further saved my time in many projects.
towardsdatascience.com
Summing up,
I found these methods quite handy while using Python for Data Analysis. And it really makes the analysis quicker. You can try these and also let me know which other methods you use to accelerate your EDA!
Now you can become a Medium member by signing up here to read all the stories published by me and other writers. when you do so, I will get a small portion of your fees without any additional cost to you. Feel free to join my email list to stay updated about my writing.
Thank you for reading and investing your time!!