Window Functions In Pandas. Running Totals, Period To Date Returns... | by Tony Yiu | Towards Data Science

SQL has a neat feature called window functions. By the way, you should definitely know how to work with these in SQL if you are looking for a data analyst job. Pandas (with a little bit of legwork) allows us to do the same things. Let’s see how.
Window functions allow us to perform an operation with a given row’s data and data from another row that is a specified number of rows away — this “number of rows away value” is called the window.
For example, let’s say you have 10 days of stock prices:
Window functions allow us to perform computations among the values of a specified column. For example, I might want to compare today’s stock price with yesterday’s — then I would want a window of “1” looking backwards. A window function allows us to do that. If on the other hand, I want to compare today’s price with the price 1 year ago, then I would want a window of “356” (assuming weekends are in your dataset).
Window functions are especially useful for time series data where at each point in time in your data, you are only supposed to know what has happened as of that point (no crystal balls allowed). The good news is that windows functions exist in pandas and they are very easy to use.
Let’s say we want to calculate the daily change in price of our stock. To do this we would need to take each day’s price and divide it by the previous day’s price and subtract 1. We get our data as a list:
stock_list = [100, 98, 95, 96, 99, 102, 103, 105, 105, 108]
Lists are not very math friendly so we can put our data in a numpy array (I reshape it to be a 9 by 1 array so that it’s easier to view and display):
In:stock_array = np.array(stock_list)(stock_array[1:]/stock_array[:-1] - 1).reshape(-1,1)Out:array([[-0.02      ],       [-0.03061224],       [ 0.01052632],       [ 0.03125   ],       [ 0.03030303],       [ 0.00980392],       [ 0.01941748],       [ 0.        ],       [ 0.02857143]])
Cool, we got the returns. No need for dataframes right? Well, not exactly. Dataframes are more versatile than numpy arrays (which are optimized for dealing with numerical data). And the developers of pandas have created all these nifty window methods to make our lives easier — they would be sad if we didn’t take advantage.
So let’s put our stock prices into a dataframe:
stock_df = pd.DataFrame(stock_list, columns=['price'])
We can use the shift method to get the previous day’s price. The shift method is very similar to SQL’s lag function. The “1” tells it to lag things by one day, giving us the previous day’s price:
stock_df['prev_price'] = stock_df.shift(1)
Now stock_df looks like this:
   price  prev_price0    100         NaN1     98       100.02     95        98.03     96        95.04     99        96.05    102        99.06    103       102.07    105       103.08    105       105.09    108       105.0
Cool, now we just need to divide price by prev_price and subtract 1 to get the daily return:
In:stock_df['daily_return'] = stock_df['price']/stock_df['prev_price']-1print(stock_df)Out:   price  prev_price  daily_return0    100         NaN           NaN1     98       100.0     -0.0200002     95        98.0     -0.0306123     96        95.0      0.0105264     99        96.0      0.0312505    102        99.0      0.0303036    103       102.0      0.0098047    105       103.0      0.0194178    105       105.0      0.0000009    108       105.0      0.028571
We can do a lot more than just calculating returns. For example, say we want to compare our daily return to an expanding window average return in order to see how each return compares to the historical average. You might think, why not just calculate the average of all the values in the daily return_column and use that? The answer is data leakage.
In time series analysis, when we are trying to forecast the future, we need to be really careful about what could have been observed and what could not have been observed on a specific date. For example, on day 5 of our dataset, we can only observe the first 5 prices: 100, 98, 95, 96, 99. So if we are testing features in order to make a forecast for day 6, we can’t compare day 5’s return of 3.03% with the mean daily change of the entire period because on day 5, we have not yet observed days 6 through 9.
That’s where an expanding window comes in. In case you are not familiar with expanding and rolling windows, the following picture visualizes what they are. With an expanding window, we calculate metrics in an expanding fashion — meaning that we include all rows up to the current one in the calculation. A rolling window allows us to calculate metrics on a rolling basis — for example, rolling(3) means that we use the current observation as well as the two preceding ones in order to calculate our desired metric.
The rationale behind using an expanding window is that with every day that passes, we get another price and another daily change that we can add to our mean calculation. That’s new information that we should capture in our calculated metrics. We can do this with the following code (I also threw in a 3 day rolling window as well for fun).
stock_df['expand_mean']=stock_df['daily_return'].expanding().mean()stock_df['roll_mean_3']=stock_df['daily_return'].rolling(3).mean()
Calling .expanding() on a pandas dataframe or series creates a pandas expanding object. It’s a lot like the more well known groupby object (which groups things based on specified column labels). The expanding (or rolling) object is what allows us to calculate various metrics in an expanding fashion. Let’s see what our dataframe looks like now:
   price  prev_price  daily_return  expand_mean  roll_mean_30    100         NaN           NaN          NaN          NaN1     98       100.0     -0.020000    -0.020000          NaN2     95        98.0     -0.030612    -0.025306          NaN3     96        95.0      0.010526    -0.013362    -0.0133624     99        96.0      0.031250    -0.002209     0.0037215    102        99.0      0.030303     0.004293     0.0240266    103       102.0      0.009804     0.005212     0.0237867    105       103.0      0.019417     0.007241     0.0198418    105       105.0      0.000000     0.006336     0.0097409    108       105.0      0.028571     0.008807     0.015996
Notice that on day 1, expand_mean and daily_return are equal — that’s necessarily the case because we are calculating the expanding mean with only one daily return on day 1. Also, on day 3 when we finally have enough data to calculate our rolling 3 day mean, roll_mean_3’s first value is equal to expand_mean. That makes sense too — on day 3, our expanding mean is also calculated using the most recent 3 days’ returns.
Here is the plot (and code) of the stock’s daily returns and the 2 means that we calculated:
plt.subplots(figsize=(8,6))plt.plot(stock_df['daily_return'], label='Daily Return')plt.plot(stock_df['expand_mean'], label='Expanding Mean')plt.plot(stock_df['roll_mean_3'], label = 'Rolling Mean')plt.xlabel('Day')plt.ylabel('Return')plt.legend()plt.show()
We can apply other functions besides the mean as well. Let’s say our boss comes over and says, “I want you to keep track of how many days this stock has been up.”
We can do this using an expanding object and the sum method (for keeping a running total). First we need to add a column to our dataframe to denote whether the stock was up that day or not. We can take advantage of the apply method (which applies a function to each row in the dataframe or series). We can either define a function to give apply or use a lambda function — I opted for a lambda function (fewer lines of code) that takes each return and returns 1 if it’s positive and 0 if it’s negative.
stock_df['positive'] = stock_df['daily_return'].apply(lambda x: 1 if x>0 else 0)
Once we have the “positive” column, we can apply an expanding window to it and the sum method (since each positive day is denoted by a 1, we just need to keep a running total of the number of 1s):
stock_df['num_positive'] = stock_df['positive'].expanding().sum()
And the dataframe that we would send to our boss looks like this:
   price  daily_return  num_positive0    100           NaN           0.01     98     -0.020000           0.02     95     -0.030612           0.03     96      0.010526           1.04     99      0.031250           2.05    102      0.030303           3.06    103      0.009804           4.07    105      0.019417           5.08    105      0.000000           5.09    108      0.028571           6.0
So as of right now, there are 6 positive days for the stock. But going forward as we collect more prices, our running total will update with it. Nice, our boss should be happy with this.
Thanks for reading! Cheers and stay safe and healthy everyone.
If you liked this article and my writing in general, please consider supporting my writing by signing up for Medium via my referral link here. Thanks!