8 Popular SQL Window Functions Replicated In Python | by AnBento | Towards Data Science

Update: Many of you contacted me asking for valuable resources to learn Python for Data Science or Data Analysis while transitioning from SQL. Below I share 4 courses/platforms that I strongly recommend to keep learning:
SQL & Python Advanced Coding Problems (StrataScratch)→ Best platform I found to prepare SQL coding interviews so far! Much cheaper than LeetCode.
Programming For Data Science With Python → Very high quality content.
Data manipulation With Python: A Pandas Crash-Course
SQL Summary Stats & Window Functions
Hope you’ll find them useful too! Now enjoy the article :D This post includes affiliate links for which I may earn a small commission at no extra cost to you, should you make a purchase.
Any data analyst who worked on building their company KPIs or created reports that analyzed business performance over time, knows that SQL Window Functions are a powerful tool. The real advantage using them is that a number of metrics can be calculated over different time frames without affecting the granularity of the original dataset. This in turn means that more can be achieved without the need of using multiple self joins or CTEs, then saving a lot of lines of code.
If so far you have mainly used SQL to build metrics and extract insights and are in the process of learning Python, you are probably wondering how to replicate your beloved Window Functions in Pandas. In the tutorial that follows, I will show how you can take advantage of window functions in Python to make your code even more compact and efficient.
For the first part of the tutorial, we will be working with a mock customer orders dataset that can be downloaded via GitHub. The dataset originally includes seven columns (Order date, Order ID, Customer ID, Item ID, Item Price, Quantity and Amount Paid (£)), but we will also add Order Month as shown below:
We found that the dataset includes 50 distinct Order IDs and 8 columns, but if we wanted to explore it further, we could analyze the distribution of orders and amount paid by customer by running the following commands:
As we can see, there are total of four customers, with customer_1 being the one that has completed the most orders (18) and spent the highest amount (£1291.75). Note how the DataFrame.groupby() method can be used in Python to achieve the same result we would achieve in SQL grouping by that column at the end of our query. But there is more, as this method is also used to replicate the over(partition by...) clause typical of window functions, as we will learn below.
towardsdatascience.com
Row Number() → Rank(method=’first’)
Row Number() → Rank(method=’first’)
The SQL Row Number() function, assigns a sequential integer to each row within the partition of a dataset. It is often used in practice to create an auxiliary column that ranks a field based on the specified partition and order. The column can then be used to more effectively filter or join the dataset.
Let’s suppose that we wanted to rank orders by customer based on the order date, starting from the less recent one. To achieve this result in SQL , we can write:
row number() over(partition by customer_id order by order_date)
With pandas the same result can be achieved by applying the .rank(method = ‘first’) function to a GroupBy object filtered by Order Date to create the Row Num column:
We can verify that customer_1 has indeed completed 18 orders, the first of which on 2019–02–13 (row num = 1) and the last on 2020–04–25 (row num = 18). Because we partitioned and ordered by Customer ID, once all the 18 orders have been ranked, the function will start ranking orders for customer_2:
QUESTION 1: What if we wanted to assign the row number in descending order (from most recent to oldest order)?
This would be straightforward because, despite it wasn’t shown in the code, pandas rank(method = ...) function has an ascending parameter that is set to true by default. By setting it to false we will invert the way the rank is assigned:
orders.groupby([‘Customer ID’])[‘Order Date’].rank(method=’first’, ascending = False)
QUESTION 2: What if we wanted to partition by multiple columns (like Customer ID and Order Month)?
In this case we just need to add the preferred fields to the GroupBy object :
#SQL Syntaxrow number() over(partition by customer_id, order_month order by order_date)#Python Syntaxorders.groupby([‘Customer ID’, 'Order Month'])[‘Order Date’].rank(method=’first')
#2. Rank() → Rank(method=’min’)
The SQL RANK() function, assigns a rank to each row within a partition of a result set. Unlike ROW NUMBER(), the rank is not sequential, meaning that rows within a partition that share the same values, will receive the same rank. Using the same example as above, the SQL syntax would be:
rank() over(partition by customer_id order by order_date)
Whereas its counterpart in Python would be:
Because customer_1 executed two orders on 2019–02–21, both of them have been assigned rank = 3 and the following order has been assigned rank = 5, then completely skipping rank = 4. This is exactly the type of ranking we would get If we had to use the SQL rank() window function.
#3. Dense_rank() → Rank(method=’dense’)
If we wanted to avoid gaps in ranking values we should use the SQL dense_rank() function instead. In effect, unlike the rank() function, using dense_rank() returns consecutive rank values. In our case, the SQL syntax would be:
dense_rank() over(partition by customer_id order by order_date)
Whereas in Python we just need to replace method=’min’ with method=’dense’ in the rank() function:
As expected, the order executed by customer_1 on 2019_02_23 has now been assigned rank = 4 , despite being the customer’s 5th order in the period of analysis.
To summarize what we have learnt so far: despite in SQL there are 3 distinct functions to compute numerical data ranks, in pandas we just need to use the rank() function with the method (‘first’, ‘min’ or ‘dense’) and ascending (True or False) parameters to obtain the desired result.
#4.Sum(...) over(partition by ... order by .. rows unbounded preceding) → cumsum()
We now wish to compute the cumulative sum of the amount paid by each customer, in each month, sorted by order date. This calculation is also known as a running total and it’s probably one of the most used metrics in business analytics. One way to achieve this in SQL is:
sum(amount_paid) over(partition by customer_id, order_month order by order_date rows unbounded preceding)
The syntax above is not exactly the most efficient and intuitive: there is quite a lot of code to write and if we wish to sort by order_date, we must also include the rows... clause to compute a cumulative sum instead of a total sum in the chosen window.
This is one of the cases where Python shines for its brevity, as to achieve the same result as above, we just need to write:
As expected, the Run Tot (£) column displays a progressive sum of the Amount Paid (£) by each customer in a month, with the row corresponding to the last in-month order, eventually showing the total amount spent by the customer in that month.
#5. Avg(...) over(partition by... ) → transform(np.mean)
In a similar fashion, we may also wish to compute the average amount spent by each customer per month. This time the SQL syntax is quite intuitive:
avg(amount_paid) over(partition by customer_id, order_month)
whereas it’s worthwhile elaborating on the Python solution:
The reason why we used pandas DataFrame.transform() function is that it allows us to keep the length of the Series equal to the length of the original dataset after computing the mean on the GroupBy object. This approach leads us to the desired result:
In the second part of this tutorial, we are going to pull AAPL and AMZN stocks Adj. Close Prices from Yahoo Finance, using the yfinance package. We should already have imported it at the very beginning, but if it is not the case we can do it now by running the following commands:
!pip install yfinance # <-- to install the packageimport yfinance as yfaapl = yf.download(“AAPL”, start=”2020–04–20", end=”2020–04–30").resample(‘D’).ffill()amzn = yf.download(“AMZN”, start=”2020–04–20", end=”2020–04–30").resample(‘D’).ffill()
With the code above, we have also created two new datasets (aapl and amzn) for the period 2020–04–20 to 2020–04–30 using the yf.download() method. In order to include weekends and holidays, data has been resampled and the missing stock prices replaced with the prior most recent Adj. Close Price observation. Additional manipulation has been applied to the datasets (full code available in GitHub) that have been eventually combined in the stocks DataFrame:
#6 Lead/lag(return_value, offset) → Shift(n)
Using the stocks dataset, we now wish to compute the DoD and WoW Adj. Close Price % Change and to do that by keeping the DataFrame length unchanged, we need a function to access rows at a specific physical offset which comes before the current row. In SQL this window function is named lag() and in our example its syntax would be:
#Retrieves the stock price located 1 row back from the current rowlag(adj_close, 1) over(partition by symbol order by date)#Retrieves the stock price located 7 rows back from the current rowlag(adj_close, 7) over(partition by symbol order by date)
IMPORTANT: Because we have resampled it, our dataset has now a daily granularity, meaning that to get yesterday stock price, we simply need to move back one row, whereas to get the stock price recorded same day last week, we just need to move back 7 rows. If we had excluded weekends and holidays, these calculations would not have been so straightforward.
In Python we can achieve the same result by applying the shift() function on a GroupBy object (grouped by Symbol and filtered by Adj. Close) to create the Lag1 and Lag7 columns. Note that because we wish to move back by 1 and 7 rows respectively (and our dataset has a descending order), the shift() functions takes a negative integer as an argument:
We can see that the shift(n) function has been correctly applied on each stock, so that on the first observed date (2020–04–20), where no prior price is available, a NaN value is displayed. In the same fashion, the first valid value displayed in the Lag7 column appears 7 days after (2020–04–27) the first observed date.
So far, we just managed to retrieve lagged stock prices, not to compute the % changes. In order to do that we could either apply the formula:
[(Adj Close / Lag1) - 1] x 100 <-- DoD % Change[(Adj Close / Lag7) -1] x 100 <-- WoW % Change
Or use the pct_change(n) function that computes the shifted stock prices under the hood and returns their % change:
For the sake of brevity, in this tutorial we won’t show an equivalent example using the lead() function, but its easy to guess that the only real change would be the sign of the integer assigned to the shift() and pct_change() function. Bear in mind that, the way the dataset is sorted, will affect the sign that should be passed to the functions to get the expected result.
#7. first/last value() → rank(method =’first’, ascending = True/False) == 1
The first value() and last value() window functions are used to retrieve the value of the first or last row, in an ordered partition of a dataset. Let’s pretend we wanted to find the first and the last available Adj. Close Prices in each month for the AAPL stock. The first 10 rows of the modified aapl dataset we will use are displayed below:
and the way we will tackle the query in SQL is:
#The aapl dataset includes prices for one stock only, therefore the #partition is just applied on order_monthfirst value(adj_close) over(partition by order_month order by date)last value(adj_close) over(partition by order_month order by date)
In Python, a function that works exactly like first/last value() does not exist but it is possible to achieve a similar result by using the Pandas rank(method =’first’) function to rank the values in a partition and then return just the one with rank = 1. Depending on how we set the ascending parameter in the rank() function, the value with rank = 1 will match with either the first or last value in the partition.
Despite not particularly challenging, this process is a little bit of a hack, as we will eventually need to back fill or forward fill the first or the last value on the entire partition to obtain the desired result:
In effect for the period 2020–03–16 to 2020–04–14, the first (or less recent) available Adj. Close Price for AAPL in March is $242.2 (Rank Asc = 1) and the last (or most recent) is $254.3 (Rank Desc = 1). Similarly in April, the first available Adj. Close Price for AAPL is $240.9 and the last is $287.
#8. Avg(...) over(partition by ... order by .. rows between n preceding and current row) → rolling(window = ...).mean()
The last window function we are going to replicate in this tutorial is a moving average. The moving average is a special case of the more general function:
avg(return_value) over()
as to compute it in SQL, we need to declare a rows ... clause (like we did for the running total). For example, if we wanted to compute a 7 days moving average for the AAPL Adj. Close Price in SQL (assuming the dataset was in ascending order) we would write:
avg(adj_close) over(order by date rows between 6 preceding and current row)
In the rows ... clause we are simply stating that we wish to include the most recent observation while computing the average, therefore we just need the 6 preceding records to get a 7 days window. This concepts may seem trivial for more experienced analysts, but could cause confusion among newbies.
Fortunately, computing moving averages with Pandas it’s a real delight, as we just need to create a rolling object (specifying the desired window) and then apply the mean() function on it. Alternatively, we may also wish to compute and expanding mean, that is itself a special case of moving average:
In practice, the difference between the two metrics is that when we apply a function on a rolling object, the window size remains constant, whereas with an expanding object, the window keeps growing. The logic behind using an expanding window is that with every day that passes, we observe a new price that can be added to our mean calculation. That is new information that we may wish to include in our calculated metrics.
In this article we have learnt how to replicate 8 popular SQL Window Functions in Python using Pandas. If you are working in analytics, you should now be well equipped to build KPIs to monitor performance using SQL or Python interchangeably. In both languages, there are of course multiple ways to achieve the same result, therefore feel free to share your experience in the response section. Hope you have enjoyed this tutorial and stay tuned for the next one!
This post includes affiliate links for which I may make a small commission at no extra cost to you should, you make a purchase.