Prediction using Bayesian State Space Model | by Sarit Maitra | Towards Data Science

Time series observations are assumed to depend linearly on a state vector that is unobserved and is generated by a stochastic and dynamic process in a state space analysis. The observations are further assumed to be subject to measurement error and independent of the state vector. Therefore, two main components which make up state space models are an observed data and the unobserved states.
A time-series is described probabilistic by a stochastic process (Yt; t = 1, 2, . . .), that is, by an ordered sequence of random vectors with the index t (time). For simplicity, we will think of equally spaced time points (daily data, monthly data, and so on). The target is to make forecasts about the value of the next observation [yn+1 having observed data up to time n, (y1 = y1, . . . , yn)].
barChart(NG)
This series looks clearly non-stationary and in fact quite irregular and volatile. The data is on daily level since 2007 till date. We can see that, price drooped significantly from nearly $20 to close to $1 during 2008/2009, before being going up till around $15 during 2010/2011; however, a highly volatile state since can be seen all across as on date and last price recorded as $8.61.
We can see from this time series that there is certainly some seasonal variation in the price; there is a peak every winter, and a low turning point almost every winter.
Again the it seems like this could be described using an additive model, as the seasonal fluctuations are roughly constant in size over time and do not seem to depend on the level of the time series, and the random fluctuations seem constant over time.
We will do a uni-variate analysis; so, let us extract the monthly “Close” prices from the data-set. Monthly data was taken for computational ease. Depending on available computational power, data can be taken on hourly, daily or weekly as well.
Following data processing are important to convert the data to data-frame and further time -series for future analysis.
Here, our data-set is now processed and we are ready to perform further analysis.
Most statistical models rely on a normal distribution, a distribution that is symmetric and has a characteristic bell shape. Probability distributions are usually (not entirely) represented in charts whose abscissa axis represents the possible values of the variable and whose ordinal axis represents the probability of occurrence.
The distribution plot comprising density and normal Q-Q plot above clearly shows that data distribution is not normal and reflects non-Gaussian characteristics.
Output from the above stationarity tests show non-stationary characteristics of data.
We normalize the data in the stock prediction scenario because of the values of random behavior. Here, min-max normalization is not advised because the stock values are not limited to a certain price range; the prices having brownian motion and can change dramatically. Therefore, advisable to normalize the stock prices with elementary standardization function:
z = (x — mean(X))/ std(X)
We see here that:
the stationary signal has very few lags exceeding the CI of the ACF .
The trend resulted in almost all lags exceeding the confidence interval.
Therefore, it can be concluded that the ACF signal is stationary. But, the trend signal is not stationary. The stationary series has a better variance around the mean level, and the peaks are evidence of the interventions in the original series. We will further decompose the time series which involves a combination of level, trend, seasonality, and noise components. Decomposition helps to provide a better understanding of problems during analysis and forecasting.
Decomposing a time series means separating it into its constituent components, which are often a trend component and a random component, and if the data is seasonal, a seasonal component.
Non-seasonal time series consist of a trend component and a random component. Decomposing the time series involves tying to separate the time series into these individual components. A seasonal time series, in addition to the trend and random components, also has a seasonal component. Decomposing a seasonal time series means separating the time series into these three components. Let's estimate the trend, seasonal, and random components of the Natural Gas TS.
plot(decompose(df1), yax.flip=TRUE) # Decompose time-series
The above plot clearly shows that trend is non-stationary which we can see from the trend signal plot in the normalization section. The seasonal component exists for a brief period of time every year most likely during the winter session. Overall, the price is highly volatile with random walk movement.
Unfortunately, there is no universally accepted notation for state space models. However, the most commonly used notations in statistical and econometric applications can be found in the work of Harrison & Stevens (1976) and West & Harrison (1997). The main focus is on Bayesian analysis is maximum likelihood (ML) estimation of unknown parameter. We will start with a local level model to introduce the methodology.
We will develop the prediction with a Base Level Model followed by ARIMA and Dynamic Linear Model to compare the prediction output.
State-space models are based on the idea that the time series (Yt) is an incomplete and noisy function of some underlying un-observable process (θt, t = 1, 2, . . .), called the state process.
A structural approach to time-series analysis is facilitated by the state space framework. In this approach, different unobserved components responsible for the dynamics of the series. These are trend, seasonal, cycle, and the effects of explanatory and intervention variables are identified separately before being put together in a state space model. Let us understand the mathematical intuition for local level model.
A time series is normally considered a set of observation y1,......, yn ordered in time. Here, the base model for representing the time series is additive model. The formula can be written as:
μt is a slowly varying component called the trend;
Υt is a periodic component of fixed period and called as seasonal;
εt is random noise or irregular component called error;
To develop suitable models and μt and Υt, we need the concept of random walk. We can consider a simple form of model in which μt = αt, where αt is random walk, no seasonal is present and all random variables are normally distributed. We assume that εt has constant variance σ2ε. This gives the model:
for t = 1,.....,n, where εt’s and ηt’s are mutually independent and independent on α1. There are two parameters, and σ2η. It is an ARIMA(0,1,1) model, but with restrictions on the parameter set. This is called local level model and provides the basis for our analysis. It exhibits the characteristics structure of state space models where there is a series of unobserved values α1,....,αn, which are states and represents the development over time of the system under study together with a set of observations y1,....,yn, which are related to the αt’s by the state space model. This is suitable for both Classical and Bayesian analysis. Where εt’s and ηt’s are not normally distributed we obtain equivalent results from the standpoint of minimum variance linear unbiased estimation.
Initially we assume that, α1 ~ N(α1, P1), where α1 and P1 are known and that σ2ε and σ2η are known. Since, random walk are non-stationary, the model is non-stationary and the distribution of random variables yt and αt depend on time t.
The local linear trend model, type = “trend”, has the same measurement equation, but with a time-varying slope in the dynamics. The basic structural model, type = “BSM”, is a local trend model with an additional seasonal component.
Call:StructTS(x = df1, type = "level")Variances:  level  epsilon     1.26     0.00
All significance tests in linear Gaussian state space models — and the construction of confidence intervals — are based on assumptions concerning the residuals of the analysis. The residuals should satisfy independence, homoscedasticity, and normality. The independence and normality of the residuals can be diagnosed using the Box-Ljung test statistic. Homoscedasticity can be checked by testing the variance of the standardized prediction errors.
Below plot shows the predicted Close price data for 12 months, together with 50% and 90% probability intervals.
plot(forecast::forecast(fit_level, level = c(50, 90), h = 12), xlim = c(2016, 2021))
pred <- predict(fit_level, n.ahead = 12)
Auto-regressive integrated moving average (ARIMA) model can be put into state space form.
In auto arima is able to decide whether to use or not a seasonal arima (SARIMA) model.
auto.arima(df1, trace = T, stepwise = F, approximation = F)
# Arima arima <- Arima(df1, order = c(0,1,0))checkresiduals(arima)
We can see from the first and last graphs that, the residuals don’t appear to be white noise. The ACF plot shows no significant correlation exists among the residuals.
# Forecast of 12 monthspred_arima <- forecast::forecast(arima, h=12)plot(pred_arima)
This prediction is similar to structural time series. However, we will check further Dynamic Linear Model in below section.
Dynamic linear models are a special case of state space models where the errors of the state and observed components are normally distributed. DLM can be specified by means of two equations
here, Gt and Ft are known matrices and the (vt) & (wt) are two independent white noise sequences (i.e., they are independent, both between them and within each of them), with mean zero and known co-variance matrices Vt and Wt respectively.
The general state space model for a uni-variate time series (Yt, t = 1, 2, . . .) is the so-called random walk plus noise model, defined by:
with arbitrary functions gt and ht which is more flexible. The error sequences (vt) and (wt) are independent. The linear state space models specify gt and ht as linear functions, and Gaussian linear models add the assumptions of Gaussian distributions.
A polynomial DLM (a local level model is a polynomial DLM of order 1) , a local linear trend is a polynomial DLM of order 2). You may find more details of dlm model here.
The arguments dV and dW are used to specify the diagonal of the observation and evolution covariance matrices respectively.
The main purpose of the Kalman filter is to obtain optimal estimates of the state at time point t, only considering the observations {y1, y2, . . . , yt−1}. The key property of the predicted state and its related estimates is therefore that they are only based on the historical values of the series. The objective of filtering is to update our knowledge of the system each time a new observation yt is brought in.
# applying kalman filtermodel_filter <- dlmFilter(df1, modelfit)plot(residuals(model_filter, sd = FALSE), type = “o”, ylab = “Standardized prediction error”)abline(h = 0)
The Kalman smoother allows to compute the densities of θt|DT , starting from t = T−1, in which case θT|DT ~ N(sT = mT , ST = CT ), and then proceeding backward for computing the densities of θt|DT for t = T−2, t = T−3, etc.
forecast$f
Here, we can see from the above plot that, the model has been able capture the trend quite well. Filtered and smoothed lines are moving together in the series and do not differ much from each other. The lines of forecast series for next 12 months has been picked up from where the original series ends
If we compare the three predicted results, we find that, DLM model has been able to capture the trend better than other two.
Similar approach can be applied to analyze and forecast for both uni-variate and multi-variate time-series. Here, the structural time series computes the maximum likelihood estimates of the variance parameters and the filtered means of the state vectors for a basic structural model, i.e., a constant DLM consisting of a random walk plus noise, a local linear trend. we have used the same data-set in all three models to maintain consistency. Interestingly ARIMA & Structural Models are not been able to pick up on the seasonal component like the dlm model. We have not considered the seasonal component in case of dlm modeling. However, the seasonal component can also be added for better prediction.
DLM is quite an interesting package and I would like to further use it for multi-variate analysis in future.
I can be reached here.
Notice: The programs described here are experimental and should be used with caution. All such use at your own risk.
References:
Petris, G., Petrone, S., & Campagnoli, P. (2009). Dynamic linear models. In Dynamic Linear Models with R (pp. 31–84). Springer, New York, NY.Giovanni Petris (2010). An R Package for Dynamic Linear Models. Journal of Statistical Software, 36(12), 1–16.Durbin J, Koopman SJ (2001). Time Series Analysis by State Space Methods. Oxford University Press, Oxford.Harrison PJ, Stevens CF (1976). “Bayesian Forecasting.” Journal of the Royal Statistical Society B, 38, 205–247.
Petris, G., Petrone, S., & Campagnoli, P. (2009). Dynamic linear models. In Dynamic Linear Models with R (pp. 31–84). Springer, New York, NY.
Giovanni Petris (2010). An R Package for Dynamic Linear Models. Journal of Statistical Software, 36(12), 1–16.
Durbin J, Koopman SJ (2001). Time Series Analysis by State Space Methods. Oxford University Press, Oxford.
Harrison PJ, Stevens CF (1976). “Bayesian Forecasting.” Journal of the Royal Statistical Society B, 38, 205–247.