Time Series Analysis: Resampling, Shifting and Rolling | by Soner Yıldırım | Towards Data Science

There are many definitions of time series data, all of which indicate the same meaning in a different way. A straightforward definition is that time series data includes data points attached to sequential time stamps. The sources of time series data are periodic measurements or observations. We observe time series data in many industries. Just to give a few examples:
Stock prices over time
Daily, weekly, monthly sales
Periodic measurements in a process
Power or gas consumption rates over time
Advancements in machine learning have increased the value of time series data. Companies apply machine learning to time series data to make informed business decisions, do forecasting, compare seasonal or cyclic trends. Large Hadron Collider (LHC) at CERN produces a great amount of time series data with measurements on sub-atomic particles. So, it is everywhere. Handling time series data well is crucial for data analysis process in such fields. Pandas was created by Wes Mckinney to provide an efficient and flexible tool to work with financial data. Therefore, it is a very good choice to work on time series data.
In this post, I will cover three very useful operations that can be done on time series data.
Resampling
Shifting
Rolling
Let’s first import the data. I will use crypto currency dataset available on Kaggle.
As always, we start with importing the libraries we need:
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport seaborn as snssns.set_style("darkgrid")%matplotlib inline
It is a huge dataset but I will just use opening price of litecoin which is enough to demonstrate how resampling, shifting and rolling windows work.
df = pd.read_csv("crypto-markets.csv", parse_dates=["date"], index_col="date")df = df[df.slug == "litecoin"][["open"]]df.shape(2042,1)df.head()
When dealing with time series data, it is better to use dates as the index of dataframe.
Resampling basically means representing the data with a different frequency. Assume we have a temperature sensor which takes measurements every minute. If we do not need to have a minute-level precision, we can take the average of 60 minute measurements in an hour and show the changes in the temperature hourly. This is down-sampling which means converting to a lower frequency.
Resampling can be done using resample() or asfreq() functions.
Resample: Aggregates data based on specified frequency and aggregation function.
Asfreq: Selects data based on the specified frequency and returns the value at the end of the specified interval.
Let’s also see how it looks on a real dataset.
fig, ax = plt.subplots(3, sharex=True)df[-500:].plot(figsize=(12,8), ax=ax[0], title="Original Data",legend=False)df[-500:].resample('10D').mean().plot(figsize=(12,8), ax=ax[1], title="Down-sampled to 10-day periods",legend=False)df[-500:].resample('30D').mean().plot(figsize=(12,8), ax=ax[2],title="Down-sampled to 30-day periods",legend=False)
As you can see, down-sampling makes the data smoother as the frequency increases.
Asfreq selects the value at the end of the specified interval where as resample aggregates (usually mean) the values in the specified interval.
As the names suggest, up-sampling is the opposite of down-sampling. The frequency is increased. When we increase the frequency, there will be some missing data.
We can leave these missing values blank (missing) or select a method to fill these missing values such as forward fill or backward fill.
Time series data analysis may require to shift data points to make a comparison. The shift and tshift functions shift data in time.
shift: shifts the data
tshift: shifts the time index
The difference between shift and tshift is better explained with visualizations. Let’s take a sample from our dataset and apply shifting:
data = df.iloc[:20,:]
We can now plot the original data along with shifted versions:
fig, ax = plt.subplots(nrows=3, figsize=(10,6), sharey=True)fig.tight_layout(pad=2)data.plot(ax=ax[0], title="Original Data", legend=None)data.shift(5).plot(ax=ax[1], title="Shifting Data", legend=None)data.tshift(5).plot(ax=ax[2], title="Shifting Index", legend=None)
Depending on the task, we can shift the data points or the index.
tight_layout function of matplotlib allows to adjust the gap between subplots using the pad parameter.
Rolling is a very useful operation for time series data. Rolling means creating a rolling window with a specified size and perform calculations on the data in this window which, of course, rolls through the data. The figure below explains the concept of rolling.
It is worth noting that the calculation starts when the whole window is in the data. In other words, if the size of the window is three, the first aggregation is done at the third row. Let’s apply rolling with size 5 to the sample dataframe we created in the previous part:
The value on 2013–05–02 after rolling is the mean of the first 5 values in the original data. The value on 2013–05–03 is the mean of the values from the second to sixth in the original data and so on.
Time series data is at the core of many problems that predictive analytics aim to solve. Hence, if you plan to work in the field of predictive analytics, you should definitely learn how to handle time series data well.
Thanks for reading. Please let me know if you have any feedback.