Advanced Time Series Analysis in Python: Decomposition, Autocorrelation | Towards Data Science

Following my very well-received post and Kaggle notebook on every single Pandas function to manipulate time series, it is time to take the trajectory of this TS project to visualization.
This post is about the core processes that make up an in-depth time series analysis. Specifically, we will talk about:
Decomposition of time series — seasonality and trend analysis
Analyzing and comparing multiple time series simultaneously
Calculating autocorrelation and partial autocorrelation and what they represent
and if seasonality or trends among multiple series affect each other.
Most importantly, we will build some very cool visualizations, and this image should be a preview of what you will be learning.
I hope you are as excited about learning these things as I when writing this article. Let’s begin.
Read the notebook of the article here.
Any time series distribution has 3 core components:
Seasonality — does the data have a clear cyclical/periodic pattern?Trend — does the data represent a general upward or downward slope?Noise — what are the outliers or missing values that are not consistent with the rest of the data?
Seasonality — does the data have a clear cyclical/periodic pattern?
Trend — does the data represent a general upward or downward slope?
Noise — what are the outliers or missing values that are not consistent with the rest of the data?
Deconstructing a time series into these components is called decomposition, and we will explore each one in detail.
Consider this TPS July Kaggle playground dataset:
Obviously, summer months have higher temperatures, and we would expect this behavior to repeat every year. However, the human eye and its ability to detect patterns can only go so far.
For example, it might be harder to find seasonal patterns from plots such as these:
To find hidden seasonal patterns from time series like above, we will use the seasonal_decompose function from statsmodels:
Using sm.tsa.seasonal_decompose on 'beef' time-series returns a DecomposeResult object with attributes like seasonal, trend and resid (more on the last two later).
Above, we are plotting the seasonality, but the plot is not useful since it has too much noise. Let’s choose an interval to give the line some room to breathe:
decomposition.seasonal["1999":"2005"].plot();
This plot shows that beef production really goes down at the beginning of each year, but it reaches its peak towards the end.
Note on seasonal_decompose function: it produces small figures by default. You have to control its aspects on your own and the plot function does not accept most of the regular Matplotlib parameters.
Now, let’s plot the seasonality of all types of meat over a 5-year interval:
As you can see, each meat types have rather different seasonality patterns. Now, let’s explore trends.
Once again, the overall trend of a time series shows whether it increased, decreased, or stayed constant (flat) over a time period. The above DecomposeResult object contains values that show the overall slope of a time series under the trend attribute.
Let’s plot them for the meat production dataset:
This plot is massively insightful compared to the simple line plot we saw in the beginning. Indeed, we now see that meat from lambs and veal production has decreased dramatically since the 1940s.
This might be in part caused by the double or triple production increases in beef, broilers, and turkey. We are performing informed guesses now, but we will explore some powerful methods to validate them in later sections.
The third component of time series is noise. There is nothing fancy about it like the other two components. It only shows random and irregular data points that could not be attributed to either seasonality or noise.
You can plot them using the resid attribute from the DecomposeResult object:
Calling plot on the whole DecomposeResult object will produce a plot with all components displayed on the same axes.
Decomposing your times series helps you think of them in a structured manner. Instead of imagining a series as a value changing over time, you can think of it as a distribution with a particular seasonality signal or a feature with a particular slope. This level of data understanding can be a key factor during feature engineering and modeling.
Working with multiple time series presents certain challenges. One example, as we saw, is the different scales each distribution comes in:
When features with larger scales squish others to a flat line (lamb and veal), it is impossible to compare their growth. One solution is using normalization.
When normalizing time series, you divide every data point in the distribution by the first sample. This has the effect of representing every single data point as the percentage increase relative to the first sample:
meat.div(meat.iloc[0]).head()
The best part is now, each distribution has the same scale. Let’s plot the meat production data by performing normalization:
Beef and pork saw the highest percentage increases while veal and lamb meat production plummeted over the given time period.
I know what you are saying: “Correlation? Really? What’s new....”
But bear with me. A simple correlation heatmap can indeed tell a lot about the linear relationships between variables:
What we are more interested in is how underlying components of time series affect each other. For example, let’s see how the seasonality of each time series influences others:
This time, we are using a ClusterMap rather than a heatmap to see closely correlated groups with the help of dendrograms immediately.
The plot tells us that the seasonality of beef, broilers, and other chicken meats are heavily correlated. The same is true with pork and lamb/mutton meats. This positive correlation can be indicative of close seasonality matches.
For example, it is possible that increase/decrease patterns of beef, broilers, and other chicken meats often matched over the given period.
Let’s do the same for trends, which I think should be more interesting:
The above plot is awesome because it helps us validate our assumptions in the trend analysis section. Let’s look at the normalized plot of meat production once again:
Can you match the patterns in the cluster map to the line plot? For example, beef has strong negative correlations with lamb/mutton and veal. This is matched by the fact that beef production tripled in amount while the production of the other two decreased by ~75% (seen from the line plot).
The same observations can be made between pork and veal, lamb/mutton.
I want you to tread carefully when making assumptions about correlated features. Always remember that correlation does not mean causation. When two features are heavily correlated, it does not mean an increase in one causes an increase in another.
An example I like to use is that even though the number of storks in a town can correlate with the number of newborn babies, it does not mean that storks deliver the babies.
It might take a while to draw the line between correlation and causation clearly, so why don’t you take a look at my other article on the topic.
Autocorrelation is a powerful analysis tool for modeling time series data. As the name suggests, it involves computing the correlation coefficient. But here, rather than computing it between two features, correlation of a time series is found with a lagging version of itself.
Let’s first look at an example plot and explain further:
The XAxis of an autocorrelation function plot (ACF) is the lag number k. For example, when k=1, the correlation is found by shifting the series by 1. This is the same as using the shift function of Pandas:
The YAXis is the amount of correlation at each lag k. The shaded red region is a confidence interval — if the height of the bars is outside this region, it means the correlation is statistically significant.
Please pause and think of what you can learn from an ACF plot.
They offer an alternative way of detecting patterns and seasonality. For example, the ACF plot of temperature in Celcius shows that the correlation at every 15 lags decreases or every 25 lags increases.
When a clear trend exists in a time series, the autocorrelation tends to be high at small lags like 1 or 2. When seasonality exists, the autocorrelation goes up periodically at larger lags.
Let’s look at another example:
The ACF of carbon monoxide confirms that small lags tend to have high correlations. It also shows that every 25 lags, the correlation increases significantly but quickly drops down to the negative. But most of the downward bars are inside the shaded area, suggesting that they are not statistically significant.
This ability to compare the relationship between past and present data points present a unique advantage. If you can associate the present value to points k periods before, this also means you can find a link to values that come after k periods.
Besides, understanding autocorrelation is key to modeling time series with ARIMA models (a topic for another article).
Even though discussing partial autocorrelation means we are getting way ahead of things, I will give you the gist.
It is similar to autocorrelation — it is calculated using the series and its lagged version at k:
The only difference is that this method tries to account for the effect the intervening lags have. For example, at lag 3, partial autocorrelation removes the effect lags 1 and 2 have on computing the correlation.
While autocorrelation is useful for analyzing a time series's properties and choosing what type of ARIMA model to use, partial autocorrelation tells what order of autoregressive model to fit.
Again, this topic will be discussed in-depth when we talk about forecasting. So, stay tuned!
Congratulations!
By reading this post, you learned powerful techniques to dissect any time series and derive meaningful insights. Most importantly, you now have the ability to apply these techniques to multiple time series and critically evaluate the relationships between them.
Thank you for reading, and I will see you in the next one!
Every Pandas Function You Can (Should) Use to Manipulate Time Series
Comprehensive Guide on Multiclass Classification Metrics
Practical Sklearn Feature Selection in 3 stages