The Upgraded “Top N” Analysis You Haven’t Seen Yet with Pandas | by Byron Dolon | Towards Data Science

Did you know that you can have a Top N analysis based on more than one column in Pandas?
A Top N analysis can be useful to select a subset of your data matching a specific condition. For example, what if you owned a restaurant and wanted to look at which customers contributed most to your overall sales? The easiest way to do this would be to look at the total sales for all your customers, then sort that list from highest to lowest.
Another interesting subset for you to look at might be the customers who had the lowest (or negative) profit contribution. You could then accomplish this in a similar fashion, getting a list of all customers by their profit contribution and then taking only the lowest members.
But what if you wanted to find out if there were customers that appeared on both lists?
That could help you identify areas in which you’re actually losing money, even if it looks like the total sales value is very high. For example, if you had a dish that had a very low-profit margin, repeated orders of this dish alone might not be beneficial for your bottom line.
Let’s take a look at how you can combine the built-in Pandas functions to do this kind of analysis!
The data used in this piece is sourced from Yahoo Finance. We’ll be using a subset of Tesla stock price data. Run the code below if you want to follow along. (And if you’re curious as to the function I used to get the data, scroll to the very bottom and click on the first link.)
import pandas as pddf = pd.read_html("https://finance.yahoo.com/quote/TSLA/history?period1=1546300800&period2=1550275200&interval=1d&filter=history&frequency=1d")[0]df = df.head(30)df = df.astype({"Open":'float',                "High":'float',                "Low":'float',                "Close*":'float',                "Adj Close**":'float',                "Volume":'float'})
To demonstrate how we can combine the Top N and Bottom N analysis, we’re going to answer the following question:
Which days had the highest increase in stock price while also having the lowest Open price in the data set?
First, we’ll need to calculate how much the stock price changed during the day. This can be achieved with a simple calculation:
df['Gain'] = df['Close*'] - df['Open']
We’ve stored the difference between the “Close*” and “Open” columns into a new column called “Gain”. As you can see in the table above, not all column values are positive, as there were some days where the stock price decreased.
Next, we’ll be creating two new DataFrames: one with the top 10 highest “Gain” values and one with the top 10 lowest “Open” values.
Here, we’ll be using the nlargest method in Pandas. This method accepts the number of elements you want to keep, the column you want to order the DataFrame by, and which duplicate values (if any) should appear in the outputted DataFrame. By default, nlargest will only keep the first of any duplicates, and the rest will be excluded from the returned DataFrame.
This method will return the same results as df.sort_values(columns, ascending=false).head(n). This code is very easy to understand and will also work, but according to the documentation, the nlargest method is more performant.
The code to get the 10 rows with the highest “Gain” values is as follows:
df_top = df.nlargest(10, 'Gain')
The returned DataFrame now gives us only the values in the original DataFrame with the highest 10 “Gain” values. This new DataFrame is also already sorted in descending order.
Next, we’ll use the nsmallest method on the DataFrame to get the rows with the lowest “Open” values. This method works exactly like the previous one, except it would sort and slice the values ascending order.
The code to achieve this is as follows:
df_bottom = df.nsmallest(10, 'Open')
We’re now ready to combine the two DataFrames to create a combined set. I’m borrowing this term from a built-in Tableau function, but all it refers to is a subset of the data that matches multiple conditions based on two or more columns. In this case, we’re looking for the data that exists only in the top 10 of “Gain” and the bottom 10 of “Open”.
To get our combined set, there are two main steps:
Concatenate the top N and bottom N DataFramesRemove all rows except the duplicates.
Concatenate the top N and bottom N DataFrames
Remove all rows except the duplicates.
The code to achieve this is as follows:
df_combined = pd.concat([df_top, df_bottom])df_combined['Duplicate'] = df_combined.duplicated(subset=['Date'])df_combined = df_combined.loc[df_combined['Duplicate']==True]
First, we simply call pd.concat and stick the two DataFrames together. Since the top N and bottom N DataFrames come from the exact same source, we don’t need to worry about renaming any columns or specifying an index.
Next, to create the “Duplicate” column, we make use of the duplicated method. This function returns a boolean Series with marking a row as “True” if it’s a duplicate and “False” otherwise. You can call this on a DataFrame and specify which column to search for duplicates in by writing the column name as an argument (in this case subset=['Date']). For demonstration, I created a new column “Duplicate” to store the new boolean values in.
I won’t go into how the loc[] function works, but if you haven’t used it before, quickly skim through this introduction so you understand how you can use it to filter your DataFrame in various ways. All we’re doing with it here is taking the values in the “Duplicate” column that are True because those are the ones that appear in both DataFrames.
We don’t even really need to create a new column to mark the duplicate values. A slightly condensed (and equivalent) version of the above code would look like this:
combined = pd.concat([df_top, df_bottom])combined = combined.loc[combined.duplicated()==True]
Voila! Now we can see the rows in which there was a high “Gain” during one of the days with the lowest “Open” in the whole dataset.
And that’s all!
I hope you found this quick look at the Top N (and Bottom N) analysis useful. Combining multiple conditions can allow you to filter and work with your data in new ways, which can help you extract valuable information from your dataset.
Good luck with your Pandas work!
More Pandas stuff by me:- 2 Easy Ways to Get Tables From a Website with Pandas- How to Quickly Create and Unpack Lists with Pandas- Top 4 Repositories on GitHub to Learn Pandas- A Quick Way to Reformat Columns in a Pandas DataFrame