How to Detect Seasonality, Outliers, and Changepoints in Your Time Series | by Khuyen Tran | Towards Data Science

Have you ever looked at a time series and wonder if there are any repetitive patterns, outliers, or change points in it?
I asked the same questions when looking at the daily number of users viewing my page from January to August 2021.
When looking at the graph above, I wonder:
Is there a shift in the mean number of views on my page? If yes, when?
Are there any outliers in my time series? If yes, when?
I suspected that there are some hours that have more active users on the page than other hours. Is this true?
Wouldn’t it be nice if you can detect the sudden changes in your time series like below in a few lines of code?
That is when Kats comes in handy. In the last article, I introduced some useful methods Kats provides to analyze time series. In this article, I will go more in-depth into Kats’ detection modules.
Kats is a lightweight, easy-to-use, and generalizable framework to perform time series analysis in Python, developed by Facebook Research. You can consider Kats as a one-stop-shop for time series analysis in Python.
To install Kats, type:
pip install --upgrade pippip install kats==0.1 ax-platform==0.2.3 statsmodels==0.12.2
I uploaded the data of my page views on Google Drive so that you can play with the data yourself. To download the data, use gdown:
pip install gdown
The data above shows the number of visitors on my website hourly from June 1st to August 2nd. Let’s visualize what this data looks like.
I suspect that there is uptime and downtime every day. Let’s check that by zooming in on the data on June 1st, 2021:
There are some patterns here. Let’s see if we can see the same pattern on June 2nd, 2021:
It looks like there is some repetitive pattern here, but it is not very easy to see. There is a mixture of seasonality, trend, and noise in the graph above. Let’s use Kats to decompose the time series into trend, seasonality, and residuals.
What is seasonality? Seasonality is a characteristic of a time series in which similar changes occur at specific regular intervals less than a year such as hourly, daily, weekly, or monthly.
To decompose a time series using Kats, start with constructing a time series object.
Next, decompose the time series into 3 different components:
Aha! It looks like there is seasonality in the data. Let’s take a closer look at the seasonality component on June 1st.
Let’s take another look at the seasonality on June 2nd:
Time series on June 3rd:
The pattern is much clearer now! There is a peak in the number of viewers from 8 am to 12 pm. This pattern makes sense since I write a new post around 8 am and send a newsletter at 10 am every day.
Instead of looking at the graphs manually, is there a test that allows us to confirm that there is indeed seasonality in the time series? Yes, we can do that with Kats’ FFTDectector.
Besides decomposing time series, Kats also allows us to use Fast Fourier Transform to detect seasonality and find out the potential cycle’s length.
{'seasonality_presence': True, 'seasonalities': [56.0, 24.0]}
Cool! Now we know that there is indeed seasonality in the time series, and the seasonalities are either 24 hours or 56 hours. 24-hour seasonality confirms what we saw earlier.
Now, another question I’m curious about is whether there is a period after which there is a shift in the mean number of views per day.
To answer this question, start with getting the daily data from January 1st to August 2nd:
Visualize the data:
It looks like there is a shift of means somewhere at the beginning of April 2021. Let’s confirm that using CUSUM.
CUSUM is a method that tests if a changepoint is statistically significant. Kats allows you to use the CUMSUM method with CUSUMDetector.
TimeSeriesChangePoint(start_time: 2021-04-01 00:00:00,                       end_time: 2021-04-01 00:00:00,                       confidence: 0.9999999999999967)
Yes! Now we are confirmed that there is a shift of means on April 1st.
Bayesian Online Change Point Detection (BOCPD) is a method to detect sudden changes in a time series that persist over time. Some properties of BOCPD are:
Online Model: This detection only needs to look at a few steps ahead instead of looking at the entire series to make predictions. It revises its predictions as new data arrives.
Bayesian Model: You can specify the prior beliefs about the probability of a changepoint. For example, if you believe 1% of your data will be a changepoint, you can set changepoint_prior=0.01 .
Currently, Kats supports 3 probability models:
Normal Distribution
Trend Change Distribution
Poisson Process Model.
Find all parameters available in BOCPDetector here.
We will use the normal distribution as an underlying probability model. Since I believe about 1% of my data is a changepoint, I will set changepoint_prior=0.01 .
TimeSeriesChangePoint(start_time: 2021-06-14T00:00:00.000000000,                      end_time: 2021-06-14T00:00:00.000000000,                       confidence: 0.74723810504334)TimeSeriesChangePoint(start_time: 2021-06-23T00:00:00.000000000,                      end_time: 2021-06-23T00:00:00.000000000,                        confidence: 0.5250604974512938)
The change points are on 06/14/2021 and 06/23/2021. This result makes sense because we see a significant increase in the number of views on 06/14/2021 and a drop in the number of views on 06/23/2021.
Removing outliers is important in a time series since outliers can cause problems in downstream processing. Luckily, Kats makes it easy to detect and remove outliers.
Here is how Kats’ outlier detection algorithm works:
Decompose the time series using seasonal decompositionRemove trend and seasonality to generate a residual time seriesDetect points in the residual which are outside 3 times the interquartile range
Decompose the time series using seasonal decomposition
Remove trend and seasonality to generate a residual time series
Detect points in the residual which are outside 3 times the interquartile range
Let’s try out this detection algorithm using OulierDectector:
[Timestamp('2021-04-03 00:00:00'), Timestamp('2021-06-20 00:00:00'), Timestamp('2021-06-21 00:00:00')]
Nice! We are able to detect the outliers on 2021–04–03, 2021–06–20, and 2021–06–21.
Now that we have detected these outliers, let’s remove them using the remover method. We will also replace the removed values with new linear interpolation values:
Plot the original time series and the new time series whose outliers are removed.
Cool! As we can see from the plot above, the time series with outliers being removed (the orange line) is different from the original time series (the blue line) on 2021–04–03, 2021–06–20, and 2021–06–21.
There doesn’t seem to be an outstanding outlier in the new time series. Now, you can be confident using this new time series for other processes such as forecasting.
Congratulations! You have just learned how to detect seasonality, changepoints, and outliers using Kats. I hope this article will give you the motivation to detect patterns in your own time series.
You can get useful information while using only a few lines of code. What can be better than that?
Feel free to play and fork the source code of this article here:
github.com
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: