Time Series Data Analysis — Resample | by James Ho | Towards Data Science

When it comes to time series analysis, resampling is a critical technique that allows you to flexibly define the resolution of the data you want. You can either increase the frequency like converting 5-minute data into 1-minute data (upsample, increase in data points), or you can do the other way around (downsample, decrease in data points).
Quoting the words from documentation, resample is a “Convenient method for frequency conversion and resampling of time series.”
In practice, there are 2 main reasons why using resample.
To inspect how data behaves differently under different resolutions or frequency.To join tables with different resolutions.
To inspect how data behaves differently under different resolutions or frequency.
To join tables with different resolutions.
Without further ado, let’s get our hands dirty and learn it from hands-on practice!
Pandas and NumPy are all we need.
import pandas as pdimport numpy as np
Let’s generate some random data first.
# Create a 2-month range, 1-minute resolution datatime datetime = pd.date_range(start='01-01-2020',end='02-28-2020',freq='1min')# Generate a randome integer array between 0 and 100 as the value. # It could be any variable in your datasetrand = np.random.randint(100,size=len(datetime))# Put them as a pandas dataframedf = pd.DataFrame({'Datetime':datetime, 'value':rand})df.head(10)
There are 2 prerequisites to carry out resampling:
The Datetime column must be in “datetime” or “timestamp” data type.The Datetime column must be the index. Alternatively, you can use the ‘on’ parameter to define the column to carry out resampling (not supported in upsampling).
The Datetime column must be in “datetime” or “timestamp” data type.
The Datetime column must be the index. Alternatively, you can use the ‘on’ parameter to define the column to carry out resampling (not supported in upsampling).
In case your Datetime column is not in the right format (they are usually string), you have to use pd.to_datetime to convert it first.
df[‘Datetime’] = pd.to_datetime(df[‘Datetime’])
Then, set the Datetime column as the index, or use the ‘on’ parameter.
df.set_index(‘Datetime’, inplace = True)# Alternativedf.resample(on='Datetime')
Now we can start playing around with the magic function!
Since 1-minute resolution is quite high, we’ll start with downsampling, i.e decrease the frequency. Say like we’d like to convert the data into 10-minute.
Downsample
The most basic syntax is
df.resample(‘desired resolution(frequency)’).aggregate_function()
Firstly, choose the desired resolution, which is 10 minutes in this case:
df.resample(‘10min’)# Using 'on' parameterdf.resample('10min', on = 'Datetime')
Then choose the aggregate function you’d like to implement. Options such as sum(), min(), max(), std(), mean(), etc. In this case, we’ll just use sum() for the sake of example. Note that after resampling, your dataframe will use Datetime as index. If that is not your intention, remember to reset the index at the end.
df_10min = df.resample(‘10min’).sum().reset_index()df_10min.head(10)
Now you should have a dataset with 10 minutes resolution and the value is the sum of the value within that 10 minutes.
Upsample
In some cases, you got the data with a resolution higher than you need. Upsampling can help you sort out this issue. However, since upsampling is more or less creating data points that didn’t exist (in our case, you don’t know what happened in that 1 minute), it’s kinda dangerous. You have to be really careful about the data integrity and what does the filling method you use mean to the data.
Let’s say there is another table with 30 seconds resolution and you need to join these two tables. Similarly, we specify the resolution first. Then instead of defining the aggregation function, we define the fillna method after resampling. There are mainly 2 ways to fill na values: ffill(), or pad(), which are basically the same thing, will assign the value of the previous data point, i.e 00:00:30 will be the same as 00:00:00. On the other hand, bfill() will assign the value of the later data point. i.e 00:00:30 will be assigned the value of 00:01:00.
Note that the on parameter is not supported in upsampling, therefore, we have to use set_index() here instead.
df_30s = df.set_index('Datetime').resample('30s').bfill().reset_index()df_30s.head(10)
An exhaustive list of resolution definitions can be found in the following table.
From the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases
It is possible to apply your self-created function. Keep in mind that the resample is gathering the value of a variable within a defined period, therefore the variable of your function should be an array. Here I use a very simple function that subtracts one from the sum as an example.
def sub_one_from_sum(var_array):    return np.sum(var_array)-1# Use apply functiondf.set_index('Datetime').resample('10min').apply(sub_one_from_sum).reset_index().head(10)
Apply different aggregation functions on multiple variables
So far we’ve been using only one variable in our example dataset. What if we have multiple variables and we’d like to apply different aggregation functions on each of them? It’s as simple as creating a dictionary of the variables and corresponding aggregation function.
Say we now have three variables, var1, var2, var3. We’d like to apply sum on var1, mean on var2, and our sub_one_from_sum onvar3. Firstly is to create a dictionary of this matchup.
df['value2'] = np.random.randint(100,size=len(datetime))df['value3'] = np.random.randint(100,size=len(datetime))agg_dict = {‘value’:'sum', ‘value2’:'mean', ‘value3’:sub_one_from_sum}# Then pass it to the resample using agg functiondf.resample('10min', on = 'Datetime').agg(agg_dict).reset_index()
Some useful parameters
label: {‘right’, ‘left’}, default ‘left’ except for ‘M’, ‘A’, ‘Q’, ‘BM’, ‘BA’, ‘BQ’, and ‘W’ time string.
Which edge label of the bin to label the bucket with. When resampling from 1 minute to 5 minutes, 00:01 to 00:04 will be aggregated together and can be labeled as 00:00 (‘left’) or 00:05 (‘right’).
closed: {‘right’, ‘left’}, default ‘left’ except for ‘M’, ‘A’, ‘Q’, ‘BM’, ‘BA’, ‘BQ’, and ‘W’ time string.
Which side of bin interval is closed. Following the 5 minutes resample, 00:00–00:04 will be aggregated together and labeled as 00:00 if using ‘left’. 00:01–00:05 will be aggregated together and labeled as 00:05 if using ‘right’.
loffset: timedelta, default None
Adding the offset to your resample label.
df.resample(‘5min’, on = ‘Datetime’, loffset = ‘2min’)>> 00:02   00:07   00:12   00:17   .   .   .
This is a pretty tricky task I got. There are 2 variables, average speed and the number of cars, in the dataframe. When downsampling, you can’t just simply apply mean() to average speed. It doesn’t really reflect the right average speed since the weight wasn’t taken into account. Each average speed has to times the number of cars first and then take the mean of them to really reflect the real average speed. The dataset may look like this:
In order to weight, I created a new variable, which is basically average speed times no. of cars as a transition. Take the sum() of that variable and then divide it by the sum() of no. of cars .
df[‘trans’] = df[‘average speed’] * df[‘no. of cars’]agg_dict = {'average speed':mean, 'no. of cars':sum, 'trans':sum}df = df.resample('5min',on = 'Datetime').agg(agg_dict)df['weighted avg. speed'] = df['trans']/df['no. of cars']df
You can clearly see the difference before and after weighting. It could be significant if the no. of cars varies within that 5 minutes.
The above example is actually not that difficult. You just need one transition variable. Now, the following question bugged me for weeks. Weighting the standard deviation. Here’s a reminder of how standard deviation is calculated:
Again, don’t forget to weigh the no. of cars. So every (x-u)2 has to time no. of cars. The biggest blocker here is that you need to subtractweighted avg. speed (5 min resolution) fromaverage speed (1 min resolution). Resample doesn’t allow you to do anything across different resolutions.
The resolution I came up with was breaking the formula into the most atomic composition, then use the similar method as the previous one to create transition variables. Recall from high school math class, (a-b)2 = a2-2ab+b2. Therefore, if we use S for average speed and F for no. of cars as aliases, the fraction of the SD formula becomes Σ(S-u)2·F = Σ(S2·F — 2S·u·F + u2). After expanding the formula, it looks like this:ΣS2·F -2u·ΣS·F+n·u2
Now it becomes very simple, we just need to create 2 transition variables, S2·F and S·F.
df[‘S2F’] = df[‘average speed’]**2 * df[‘no. of cars’]df[‘SF’] = df[‘average speed’] * df[‘no. of cars’]agg_dict={‘average speed’:’mean’, ‘no. of cars’:’sum’, ‘S2F’:’sum’, ‘SF’:’sum’}df = df.resample(‘5min’, on=’Datetime’).agg(agg_dict)df['weighted avg. speed'] = df['SF']/df['no. of cars']df[‘Speed STD’] = df[‘S2F’] -2*df[‘weighted avg. speed’]*df[‘SF’] + 5*df[‘weighted avg. speed’]**2
It’s kinda ugly but it does the work. If you got a better solution, do comment and let me know!
Resampling really opens a lot more possibilities for time series analysis. By inspecting the data behaviours under different resolution, we can gain more insights than merely looking at the original data. What is your favorite tip of time series data analysis? Comment and share to let everyone know!