Time Series Forecasting with Splunk. Part I. Intro & Kalman Filter. | by Nikolay Ryabykh | Towards Data Science

Time series forecasting is a quite typical task in data analysis. Splunk ML Toolkit provides a couple of well-known methods for this task: Kalman filter and ARIMA. Unfortunately, these methods sometimes fail when encountering a real-world problems. So, let’s take a closer look on them and then consider a couple of other techniques of time series forecasting in Splunk.
OK, let’s define a goal. Let we use Splunk to monitor some metrics, such as number of logins to a system, number of transactions in DB, etc. We want to analyse growth of our business (for example, in purposes of capacity planning and resources management). As part of this analysis, we want to predict number of logins for a year based on data of the last several years.
Quite a common task, as you can see. But let’s make it more complex with the requirement of flexibility. Because of monitoring not the only metric, our method or algorithm must be quite universal to predict almost any metric without lots of tuning by user. After all, we need to construct a dashboard with such functionality: user selects a metric, specifies few tuning parameters, and gets an appropriate forecast of selected metric for a year.
Let’s refresh some basic concepts about time series. In general, time series can be decomposed into several components: trend component, seasonal component, and residual component. Trend is a general direction of changing time series value. Increasing trend, for example, means that the values of time series are, in average, larger than previous ones. Seasonal component stands for some periodic fluctuations in data. For example, number of logins in night hours is constantly smaller than in day hours, as well as number of logins in weekdays is larger than on weekends. Residual component is the difference between time series values and determinate components (trend, seasonality) and often stands for some kind of noise.
There are two main models to deal with these components. In additive model initial time series represents as the sum of components, in multiplicative model — as the product of them. Sometimes it might be a tough choice what model to prefer. Generally, multiplicative model is more preferable if amplitude of seasonal fluctuations increases over time.
For each time series one can calculate mean, variance, and autocorrelation. Time series is stationary if these parameters are constant over time. In the picture below you find some examples of stationary (blue) and non-stationary (yellow, red, violet) time series.
Mean and variance of ‘blue’ series are equal to 0 and 1 respectively, so this series is stationary. ‘Yellow’ series shows an increasing trend, that is mean is not constant. ‘Red’ series shows an obvious daily seasonality, that is autocorrelation function has a local maximum at the lags of 24, 48, ... hours — and it’s not constant, too. ‘Violet’ series shows an increasing trend and disturbance, that is mean and variance are not constant as well.
Real-word time series are seldom stationary. But there are a lot of techniques and transformations for turning them into stationary ones.
Let’s turn ML Toolkit on and try to predict our series.
Kalman filtering is an algorithm that produces estimates of unknown variables that tend to be more accurate than those based on a single measurement alone (sorry, I copypasted definition from wiki article). In other words, Kalman filter takes time series as input and performs some kind of smoothing and denoising. After this, smoothed series might be predicted. But one should take into account that Kalman filter is firstly for denoising, rather than forecasting. That’s why the results of forecasting might be a bit inappropriate.
Go to Assistants > Forecast Time Series and select Kalman filter as algorithm.
ML Toolkit provides several methods for prediction:
LL (Local Level) to predict only local level of time series (without any trend or seasonality);
LLT (Local Level Trend) to predict only trend;
LLP (Seasonal Local Level) to predict only seasonal component;
LLP5 (combines LLT and LLP) to account for both trend and seasonality.
First of all, let’s try to apply Kalman filter to one of non-stationary time series from example above just to check smoothing capabilities. Let’s select “yellow” series — it’s just a sum of constant trend and normally distributed noise. I got it by this SPL query (normal is a custom external command that returns a normally distributed value with mean and variance from fields specified in loc and scale parameters respectively):
| gentimes start=”01/01/2018" increment=1h| eval _time=starttime, loc=0, scale=20| normal loc=loc scale=scale | streamstats count as cnt| eval gen_normal = gen_normal + cnt| table _time, gen_normal| rename gen_normal as “Non-stationary time series (trend)”
Ok, let’s apply Kalman filter with algorithm “LLT”, future timespan = 200, confidence interval = 95.
Looks nice. Filter really removes normally distributed noise and predicts actual trend.
Now let’s try to predict our number of logins. Our time series includes apparently both trend and seasonal components, so let’s select prediction algorithm = LLP5, future timespan = 365 (predict for a year), period = 365 (as we expect this seasonal period to be the biggest one).
Well... It’s pretty obvious that the result is slightly not as expected. It seems that filter considers our fluctuations in data as a noise. Thereby, filter eliminates this noise and keeps only the trend component. But it’s a bit confused that “LLP” part of algorithm, which is responsible for seasonal component, doesn’t work at all. Anyway, this forecast is unacceptable.
There is, however, a silver lining. First of all, upper and lower bounds of confidence interval are very useful for outliers detection. Secondly, it cannot be denied that the forecast of trend component is rather well. So, let’s try to filter out all outliers and remove trend component from initial data. Notice that variance of initial time series increases over time. That’s why it seems multiplicative model to be more appropriate in our case. So, let’s divide initial values by the values of trend component.
| inputlookup daylogins.csv| timechart span=1d fixedrange=f max(daylogins) as daylogins| predict “daylogins” as prediction algorithm=”LLT” future_timespan=365 period=365 lower”95"=lower”95" upper”95"=upper”95"| eval daylogins = case(daylogins < ‘lower95(prediction)’, ‘lower95(prediction)’, daylogins > ‘upper95(prediction)’, ‘upper95(prediction)’, 1=1, daylogins)| eval season = daylogins / prediction| table _time, season| where _time<strptime(“2018–01–01”, “%F”)
Here I use a predict command in SPL. This is a command for Kalman filtering, you can check it by clicking the “Show SPL” button on the “Forecast Time Series” dashboard.
Change method to LLP and click “Forecast”.
Well, not bad. Main seasonal profiles (like a big recession after the New Year) are captured and predicted almost perfectly.
There is an extended New Year holiday in Russia. It starts on the January, 1 and lasts about 8–10 days (actual duration varies from one year to the next). It figures that user activity in many areas in those days is much less than in other days.
But, as always, the devil is in the detail. Look at the predicted spike right after New Year holiday. It seems it is the same as a year ago. Actually, they seems to be too identical. Notice that in 2015 and 2016 there wasn’t anything similar to this peak. Similarity becomes even more obvious if turning off outliers removal.
Outlier from Feb, 2017 is repeated in prediction, but outlier from May, 2016 is not. So, it seems that Kalman filter with LLP algorithm overrates values of the last period (last 365 days in our case) and not pays any mind to corresponding values of previous periods.
The same situation is with the high season in New Year’s Eve.
However, such feature of Kalman filtering does not make this forecast totally inadequate. It might be useful if we really want to rely only on the last period of data. For example, we’re going to predict number of logins for a week based on data of previous weeks just for real-time detection of critical problems. If number of logins decreases remarkably, it likely stands for some issues with availability of authentication server or something like that. For that monitoring we can create a simple and clean forecast and just compare new data to the bounds of confidence interval.
OK, let’s get back to initial forecast for a year. Despite its apparent shortcomings, forecast of seasonal component of our time series is rather reasonable. So, let’s multiply predicted trend component by predicted seasonal component for reverting to desired number of logins.
| inputlookup daylogins.csv | timechart span=1d fixedrange=f max(daylogins) as daylogins | predict "daylogins" as prediction algorithm="LLT" future_timespan=365 lower"95"=lower"95" upper"95"=upper"95" | eval daylogins = case(daylogins < 'lower95(prediction)', 'lower95(prediction)', daylogins > 'upper95(prediction)', 'upper95(prediction)', 1=1, daylogins) | eval season = daylogins / prediction | predict "season" as season_prediction algorithm="LLP" future_timespan=365 period=365 lower"0"=lower"0" upper"0"=upper"0"| eval prediction = prediction * season_prediction| table _time, daylogins, prediction| eval prediction = if(isnull(daylogins), prediction, null)| where _time<strptime("2019-01-01", "%F")
Looks pretty nice. Notice that I’ve finally implemented this forecast in pure SPL without ML Toolkit Assistant GUI. So, I can insert this search into any dashboard of any application (just take care of permissions to knowledge objects). The only parameter I need to specify is the period of seasonal component (i.e. maximum seasonal period in case of multiple seasonality). You can allow user to specify this parameter on a dashboard via a simple input text box with default value of 365. Also it’s possible to calculate the maximum seasonal period automatically using Fourier transform or analyzing autocorrelation function (we’ll talk about it a bit later, I think).
Let’s make a brief review of Kalman filter in Splunk.
It is a simple and useful tool for time series forecasting.
One can create a forecast easily with only one SPL command without tuning tons of parameters.
Filter (following its name) is good in smoothing of noisy time series.
But this simplicity means the lack of flexibility.
You cannot control seasonal periods at all. In case of multiple seasonality you can rely only on internal algorithms of filter.
Seasonal forecast takes too little account of the values outside of the last period.
There is no any parameter like a “smoothing coefficient” for the tuning of “denoising sensitivity”. Therefore, filter sometimes performs very aggressive smoothing, and it causes to the loss of valuable information.
Well, let’s call it a night. In the next part let’s look more closely at ARIMA in Splunk. Stay tuned!