Ultimate Pandas Guide: Time Series Window Functions | by Skyler Dale | Towards Data Science

In my last post, I walked through how to run window functions in Pandas based on column values.
This approach is useful anytime we want to know information about both the individual records and the groups they belong to.
For example, if we have customer-level transaction data, an approach like this can provide us with information about each individual transaction, as well as the total sales during the month in which it took place:
In this post, I’ll walk through another type of the window function — one where we perform our calculation based on the position of the rows, rather than the values of a categorical column.
For my examples below, I’ll work with some game-level basketball data from the NCAA ML competition on Kaggle.
If basketball is not your thing, fear not.
Here is a quick data dictionary that tells you all you need to know about the variables:
DayNum: Our measure of time. It counts how many days into the season the game occurred.Season: The year the game took place.Team1/Team2: The IDs of the teams that played in the game.Efficiency: A measure of how well Team1 performed in the game.Outcome: Flag for whether or not Team1 won the game (1 is win, 0 is loss).
DayNum: Our measure of time. It counts how many days into the season the game occurred.
Season: The year the game took place.
Team1/Team2: The IDs of the teams that played in the game.
Efficiency: A measure of how well Team1 performed in the game.
Outcome: Flag for whether or not Team1 won the game (1 is win, 0 is loss).
Finally, let’s assume our goal is to predict who won each game. We won’t do any machine learning here, but this will help motivate the use cases.
Let’s start by doing a simple “shift.”
This method does exactly what it sounds like. It shifts the values in a column either forward or backwards.
Below we shift the GameEfficiency by 1:
game_data[‘prior_game_outcome’] = game_data[‘Outcome’].shift(1)
Now — for each game — we know how well Team1 did in the prior game. Perhaps this could help us predict the outcome.
Note that we can pass either positive or negative integers to shift forward or backwards:
game_data['NextEfficiency'] = game_data['GameEfficiency'].shift(-1)
Also keep in mind that when we do a positive shift on the first row we get a null value because there is no available data in the preceding one. We can address this by setting the “fill_value” parameter to replace the null with a different value of our choice.
While the shift method is useful, it doesn’t allow us to perform any functions on the prior or future rows.
For example, we might want to find the average efficiency of Team1 over the prior three games.
This is where we can leverage the rolling method.
The basic syntax is pretty simple — we just need to pass the number of prior rows we want to look at and then perform an aggregation:
game_data[‘AvgEfficiency’] = game_data[‘GameEfficiency’].rolling(3).mean()
Note that there are technically two steps here: the “rolling” method creates a Rolling object, and then the “mean” method returns the new column.
We probably want to make one more adjustment here. By default, each record gets included in its own window average. If we want to predict each game, this is bad because it means we have information about the outcome encoded in the average.
To solve this, we can add a shift function so that the data from previous 3 rows is included (rather than the current row and the prior 2):
game_data[‘AvgEfficiency’] = game_data[‘GameEfficiency’].shift(1).rolling(3).mean()
The expanding method is very similar to the rolling method, except that it creates as large of window as it can given the data.
Here’s an example:
game_data[‘AvgEfficiency’] = game_data[‘GameEfficiency’].shift(1).expanding().mean()
Note that the AvgEfficiency value in the second to last row is the same as when we ran the rolling method above (12.02), but the last rows are different (2.41 vs. 13.46).
That’s because when we’re at the second to last row, we only have 3 prior records available, but when we reach the last row we have 4 prior records. Again, the expanding method uses as much of the data as possible.
Finally, we can use the “min_periods” parameter if we want to make sure that our expanding window has at least a certain number of records in order for the aggregation to be applied:
game_data[‘AvgEfficiency’] = game_data[‘GameEfficiency’].shift(1) .expanding(min_periods = 4).mean()
We started off this post by differentiating between column-based windowing and position-based windowing.
Our final use case leverages both.
Here’s the problem with all of the work above: we actually have multiple teams and seasons in our dataset, so we run into issues at the edges of our sorted data:
In order to resolve this, we need to split our data up into groups of Team/Season and perform an expanding window aggregation:
game_data[‘AvgEfficiency’] = game_data.groupby([‘Season’, ‘Team1’])[[‘GameEfficiency’]].transform(lambda x: x.shift(1).expanding().mean())
Now our expanding average calculation starts fresh for each team’s schedule!
There’s a lot going on in the code above, but here’s how I would think about it:
First we split our data up into groups of Season and Team1Then we apply a custom anonymous function (lambda) to each group which shifts the data and calculates an expanding meanFinally we return these values back to the original index
First we split our data up into groups of Season and Team1
Then we apply a custom anonymous function (lambda) to each group which shifts the data and calculates an expanding mean
Finally we return these values back to the original index
In this post we walked through time series window functions in Pandas. The syntax for some of these methods can be a bit tricky, but when you break it down into smaller steps everything becomes a little more clear. It also doesn’t hurt to spend some time playing around with each method until you get the hang of it. Happy coding!