Using the Pandas “Resample” Function | by Jeremy Chow | Towards Data Science

This article is an introductory dive into the technical aspects of the pandas resample function for datetime manipulation. I hope it serves as a readable source of pseudo-documentation for those less inclined to digging through the pandas source code!
If you’d like to check out the code used to generate the examples and see more examples that weren’t included in this article, follow the link here.
Let’s get started!
Resampling is necessary when you’re given a data set recorded in some time interval and you want to change the time interval to something else. For example, you could aggregate monthly data into yearly data, or you could upsample hourly data into minute-by-minute data.
The syntax of resample is fairly straightforward:
<DataFrame or Series>.resample(arguments).<aggregate function>
I’ll dive into what the arguments are and how to use them, but first here’s a basic, out-of-the-box demonstration. You will need a datetime type index or column to do the following:
# Given a Series object called data with some number value per date>>>╔═══════════════════════╦══════╗║         date          ║ val  ║╠═══════════════════════╬══════╣║ 2000-01-01 00:00:00   ║    0 ║║ 2000-01-01 00:01:00   ║    2 ║║ 2000-01-01 00:02:00   ║    4 ║║ 2000-01-01 00:03:00   ║    6 ║║ 2000-01-01 00:04:00   ║    8 ║║ 2000-01-01 00:05:00   ║   10 ║║ 2000-01-01 00:06:00   ║   12 ║║ 2000-01-01 00:07:00   ║   14 ║║ 2000-01-01 00:08:00   ║   16 ║╚═══════════════════════╩══════╝Freq: T, dtype: int64# We can resample this to every other minute instead and aggregate by summing the intermediate rows:data.resample('2min').sum()>>>╔═════════════════════╦══════╗║        date         ║ val  ║╠═════════════════════╬══════╣║ 2000-01-01 00:00:00 ║    2 ║║ 2000-01-01 00:02:00 ║   10 ║║ 2000-01-01 00:04:00 ║   18 ║║ 2000-01-01 00:06:00 ║   26 ║║ 2000-01-01 00:08:00 ║   16 ║╚═════════════════════╩══════╝Freq: 2T, dtype: int64
Now that we have a basic understanding of what resampling is, let’s go into the code!
In the order of the source code :
pd.DataFrame.resample(rule, how=None, axis=0, fill_method=None, closed=None, label=None, convention=”start”, kind=None, loffset=None, limit=None, base=0, on=None, level=None)
I’ve bolded the arguments that I will cover. The rest are either deprecated or used for period instead of datetime analysis, which I will not be going over in this article.
string that contains rule aliases and/or numerics
This is the core of resampling. The string you input here determines by what interval the data will be resampled by, as denoted by the bold part in the following line:
data.resample('2min').sum()
As you can see, you can throw in floats or integers before the string to change the frequency. You can even throw multiple float/string pairs together for a very specific timeframe! For example:
'3min' or '3T' = 3 minutes'SMS' = Two times a month'1D3H.5min20S' = One Day, 3 hours, .5min(30sec) + 20sec
To save you the pain of trying to look up the resample strings, I’ve posted the table below:
Once you put in your rule, you need to decide how you will either reduce the old datapoints or fill in the new ones. This function goes right after the resample function call:
data.resample('2min').sum()
There are two kinds resampling:
Downsampling — Resample to a wider time frame (from months to years)
Downsampling — Resample to a wider time frame (from months to years)
This is fairly straightforward in that it can use all the groupby aggregate functions including mean(), min(), max(), sum() and so forth.
In downsampling, your total number of rows goes down.
2. Upsampling — Resample to a shorter time frame (from hours to minutes)
This will result in additional empty rows, so you have the following options to fill those with numeric values:
1. ffill() or pad()2. bfill() or backfill()
‘Forward filling’ or ‘padding’ — Use the last known value to fill the new one.
‘Backfilling’ — Use the next known value to fill the new one.
You can also fill with NaNs using the asfreq() function with no arguments. This will result in new data points having NaNs in them, which you can use a fillna() function on later.
Here are some demonstrations of the forward and back fills:
Starting with months table:╔════════════╦═════╗║    date    ║ val ║╠════════════╬═════╣║ 2000-01-31 ║   0 ║║ 2000-02-29 ║   2 ║║ 2000-03-31 ║   4 ║║ 2000-04-30 ║   6 ║║ 2000-05-31 ║   8 ║╚════════════╩═════╝print('Forward Fill')print(months.resample('SMS').ffill())╔════════════╦═════╗║    date    ║ val ║╠════════════╬═════╣║ 2000-01-15 ║ NaN ║║ 2000-02-01 ║ 0.0 ║║ 2000-02-15 ║ 0.0 ║║ 2000-03-01 ║ 2.0 ║║ 2000-03-15 ║ 2.0 ║║ 2000-04-01 ║ 4.0 ║║ 2000-04-15 ║ 4.0 ║║ 2000-05-01 ║ 6.0 ║║ 2000-05-15 ║ 6.0 ║╚════════════╩═════╝# Alternative to ffill is bfill (backward fill) that takes value of next existing months pointprint('Backward Fill')print(months.resample('SMS').bfill())╔════════════╦═════╗║    date    ║ val ║╠════════════╬═════╣║ 2000-01-15 ║   0 ║║ 2000-02-01 ║   2 ║║ 2000-02-15 ║   2 ║║ 2000-03-01 ║   4 ║║ 2000-03-15 ║   4 ║║ 2000-04-01 ║   6 ║║ 2000-04-15 ║   6 ║║ 2000-05-01 ║   8 ║║ 2000-05-15 ║   8 ║╚════════════╩═════╝
'left', 'right', or None
I’m going to include their documentation comment here, since it describes the basics fairly succinctly.
Which side of bin interval is closed. The default is ‘left’for all frequency offsets except for ‘M’, ‘A’, ‘Q’, ‘BM’,‘BA’, ‘BQ’, and ‘W’ which all have a default of ‘right’.
The closed argument tells which side is included, ‘closed’ being the included side (implying the other side is not included) in the calculation for each time interval. You can see how it behaves here:
# Original Table 'minutes'╔═════════════════════╦═════╗║        date         ║ val ║╠═════════════════════╬═════╣║ 2000-01-01 00:00:00 ║   0 ║║ 2000-01-01 00:01:00 ║   2 ║║ 2000-01-01 00:02:00 ║   4 ║║ 2000-01-01 00:03:00 ║   6 ║║ 2000-01-01 00:04:00 ║   8 ║║ 2000-01-01 00:05:00 ║  10 ║║ 2000-01-01 00:06:00 ║  12 ║║ 2000-01-01 00:07:00 ║  14 ║║ 2000-01-01 00:08:00 ║  16 ║╚═════════════════════╩═════╝# The default is closed='left'df=pd.DataFrame()df['left'] = minutes.resample('2min').sum()df['right'] = minutes.resample('2min',closed='right').sum()df>>>╔═════════════════════╦══════╦═══════╗║ index               ║ left ║ right ║╠═════════════════════╬══════╬═══════╣║ 2000-01-01 00:00:00 ║ 2    ║ 6.0   ║║ 2000-01-01 00:02:00 ║ 10   ║ 14.0  ║║ 2000-01-01 00:04:00 ║ 18   ║ 22.0  ║║ 2000-01-01 00:06:00 ║ 26   ║ 30.0  ║║ 2000-01-01 00:08:00 ║ 16   ║ NaN   ║╚═════════════════════╩══════╩═══════╝
'left', 'right', or None
Once again, the documentation is pretty useful.
Which bin edge label to label bucket with. The default is ‘left’ for all frequency offsets except for ‘M’, ‘A’, ‘Q’, ‘BM’, ‘BA’, ‘BQ’, and ‘W’ which all have a default of ‘right’.
This argument does not change the underlying calculation, it just relabels the output based on the desired edge once the aggregation is performed.
df=pd.DataFrame()# Label default is leftdf['left'] = minutes.resample('2min').sum()df['right'] = minutes.resample('2min',label='right').sum()df>>>╔═════════════════════╦══════╦═══════╗║                     ║ left ║ right ║╠═════════════════════╬══════╬═══════╣║ 2000-01-01 00:00:00 ║    2 ║ NaN   ║║ 2000-01-01 00:02:00 ║   10 ║ 2.0   ║║ 2000-01-01 00:04:00 ║   18 ║ 10.0  ║║ 2000-01-01 00:06:00 ║   26 ║ 18.0  ║║ 2000-01-01 00:08:00 ║   16 ║ 26.0  ║╚═════════════════════╩══════╩═══════╝
stringthat matches rule notation.
This argument is also pretty self explanatory. Instead of changing any of the calculations, it just bumps the labels over by the specified amount of time.
df=pd.DataFrame()df['no_offset'] = minutes.resample('2min').sum()df['2min_offset'] = minutes.resample('2min',loffset='2T').sum()df['4min_offset'] = minutes.resample('2min',loffset='4T').sum()df>>>╔═════════════════════╦═══════════╦═════════════╦═════════════╗║ index               ║ no_offset ║ 2min_offset ║ 4min_offset ║╠═════════════════════╬═══════════╬═════════════╬═════════════╣║ 2000-01-01 00:00:00 ║         2 ║ NaN         ║ NaN         ║║ 2000-01-01 00:02:00 ║        10 ║ 2.0         ║ NaN         ║║ 2000-01-01 00:04:00 ║        18 ║ 10.0        ║ 2.0         ║║ 2000-01-01 00:06:00 ║        26 ║ 18.0        ║ 10.0        ║║ 2000-01-01 00:08:00 ║        16 ║ 26.0        ║ 18.0        ║╚═════════════════════╩═══════════╩═════════════╩═════════════╝
numeric input that correlates with the unit used in the resampling rule
Shifts the base time to calculate from by some time amount. As the documentation describes it, this function moves the ‘origin’.
minutes.head().resample('30S').sum()>>>╔═════════════════════╦═════╗║        date         ║ val ║╠═════════════════════╬═════╣║ 2000-01-01 00:00:00 ║   0 ║║ 2000-01-01 00:00:30 ║   0 ║║ 2000-01-01 00:01:00 ║   2 ║║ 2000-01-01 00:01:30 ║   0 ║║ 2000-01-01 00:02:00 ║   4 ║║ 2000-01-01 00:02:30 ║   0 ║║ 2000-01-01 00:03:00 ║   6 ║║ 2000-01-01 00:03:30 ║   0 ║║ 2000-01-01 00:04:00 ║   8 ║╚═════════════════════╩═════╝minutes.head().resample('30S',base=15).sum()>>>╔═════════════════════╦═════╗║        date         ║ val ║╠═════════════════════╬═════╣║ 1999-12-31 23:59:45 ║   0 ║║ 2000-01-01 00:00:15 ║   0 ║║ 2000-01-01 00:00:45 ║   2 ║║ 2000-01-01 00:01:15 ║   0 ║║ 2000-01-01 00:01:45 ║   4 ║║ 2000-01-01 00:02:15 ║   0 ║║ 2000-01-01 00:02:45 ║   6 ║║ 2000-01-01 00:03:15 ║   0 ║║ 2000-01-01 00:03:45 ║   8 ║╚═════════════════════╩═════╝The table shifted by 15 seconds.
These arguments specify what column name or index to base your resampling on.
If your data has the date along the columns instead of down the rows, specify axis = 1
If your date column is not the index, specify that column name using:
on = 'date_column_name'
If you have a multi-level indexed dataframe, use level to specify what level the correct datetime index to resample is.
The rest of the arguments are deprecated or redundant due to functionality being captured using other methods. For example, how and fill_method remove the need for the aggregate function after the resample call, but how is for downsampling and fill_method is for upsampling. You can read more about these arguments in the source documentation if you’re interested.
That’s all for today! I hope I shed some light on how resample works and what each of its arguments do. Stay tuned for more tutorials and other data science related articles!