Bayesian Time Series Forecasting with Change Point and Anomaly Detection

Abstract: Time series forecasting plays a crucial role in marketing, finance and many other quantitative fields. A large amount of methodologies has been developed on this topic, including ARIMA, Holt–Winters, etc. However, their performance is easily undermined by the existence of change points and anomaly points, two structures commonly observed in real data, but rarely considered in the aforementioned methods. In this paper, we propose a novel state space time series model, with the capability to capture the structure of change points and anomaly points, as well as trend and seasonality. To infer all the hidden variables, we develop a Bayesian framework, which is able to obtain distributions and forecasting intervals for time series forecasting, with provable theoretical properties. For implementation, an iterative algorithm with Markov chain Monte Carlo (MCMC), Kalman filter and Kalman smoothing is proposed. In both synthetic data and real data applications, our methodology yields a better performance in time series forecasting compared with existing methods, along with more accurate change point detection and anomaly detection.

Reviews and author responses:



Summary:

This paper develops a state space time series forecasting model in the Bayesian framework, jointly detects anomaly and change points. Integrated with an iterative MCMC method, the authors develop an efficient algorithm and use both synthetic and real data set to demonstrate that their algorithms outperform many other state-of-art algorithms. 

Major comments:
In the beginning of section 3, the authors assume that all the terms that characterize the change-points and anomaly points are normally distributed with mean zero and different variance. However, in classic formulation for change-point or anomaly detection, usually there is also a mean shift other than the variance change. For example, we might assume $r_t \sim N(\theta, \sigma_r^2)$ for some $\theta>0$ to demonstrate the positive mean shift. I believe that this kind of mean shift is more efficient to model the structure of change-point. 

My main concern is with the novelty. The work does not seem to be very novel.

Minor comments:

1. In the end of the page 2, the last panel is the residual, not the spikes. 

2. In page 12, the caption of figure 5 should be (left) and (right), not (top) and (bottom).

The paper introduces a Bayesian model for timeseries with anomaly and change points besides regular trend and seasonality. It develops algorithms for inference and forecasting. The performance is evaluated and compared against state-of-the-art methods on three data sets: 1) synthetic data obtained from the generative Bayesian model itself; 2) well-log data; 3) internet traffic data.

On the methodological side, this appears to be a solid and significant contribution, although I am not sure how well it is aligned with the scope of ICLR. The introduced model is elegant; the algorithms for inference are non-trivial.

From a practical perspective, one cannot expect this contribution to be ground-breaking, since there has been more than 40 years of work on time series forecasting, change point and anomaly detection. In some situations the methodology proposed here will work better than previous approaches (particularly in the situation where the data comes from the Bayesian model itself - in that case, there clearly is no better approach), in other cases (which the paper might have put less emphasis on), previous approaches will work better. To position this kind of work, I think it is important the authors discuss the limitations of their approach. Some guidelines on when or when not to use it would be valuable. Clearly, these days one cannot introduce methodology in this area and expect it to outperform existing methods under all circumstances (and hence practitioners to always choose it over any other existing method).

What is surprising is that relatively simple approaches like ETS or STL work pretty much equally well (in some cases even better in terms of MSE) than the proposed approach, while more recent approaches - like BSTS - dramatically fail. It would be good if the authors could comment on why this might be the case.

Summary:
+ Methodology appears to be a significant, solid contribution.
- Experiments are not conclusive as to when or when not to choose this approach over existing methods
- writing needs to be improved (large number of grammatical errors and typos, e.g. 'Mehtods')

Minor comments:
- page 3. “The observation equation and transition equations together (i.e., Equation (1,2,3)) together define “ - one “together” should be removed
- page 4. “From Figure 2, the joint distribution (i.e., the likelihood function ” - there should be additional bracket
- page 7. “We can further integral out αn “ -> integrate out

Major comments:
The paper is well-written. The paper considers structural time-series model with seasonal component and stochastic trend, which allow for change-points and structural breaks.

Such type of parametric models are widely considered in econometric literature, see e.g.
[1] Jalles, João Tovar, Structural Time Series Models and the Kalman Filter: A Concise Review (June 19, 2009). FEUNL Working Paper No. 541. Available at SSRN: https://ssrn.com/abstract=1496864 or http://dx.doi.org/10.2139/ssrn.1496864 
[2] Jacques J. F. Commandeur, Siem Jan Koopman, Marius Ooms. Statistical Software for State Space Methods // May 2011, Volume 41, Issue 1.
[3] Scott, Steven L. and Varian, Hal R., Predicting the Present with Bayesian Structural Time Series (June 28, 2013). Available at SSRN: https://ssrn.com/abstract=2304426 or http://dx.doi.org/10.2139/ssrn.2304426 
[4] Phillip G. Gould, Anne B. Koehler, J. Keith Ord, Ralph D. Snyder, Rob J. Hyndman, Farshid Vahid-Araghi, Forecasting time series with multiple seasonal patterns, In European Journal of Operational Research, Volume 191, Issue 1, 2008, Pages 207-222, ISSN 0377-2217, https://doi.org/10.1016/j.ejor.2007.08.024.
[5] A.C. Harvey, S. Peters. Estimation Procedures for structural time series models // Journal of Forecasting, Vol. 9, 89-108, 1990
[6] A. Harvey, S.J. Koopman, J. Penzer. Messy Time Series: A Unified approach // Advances in Econometrics, Vol. 13, pp. 103-143.

They also use Kalman filter and MCMC-based approaches to sample posterior to estimate hidden components.

There are also non-parametric approaches to extraction of components from quasi-periodic time-series, see e.g.
[7] Artemov A., Burnaev E. Detecting Performance Degradation of Software-Intensive Systems in the Presence of Trends and Long-Range Dependence // 16th International Conference on Data Mining Workshops (ICDMW), IEEE Conference Publications, pp. 29 - 36, 2016. DOI: 10.1109/ICDMW.2016.0013
[8] Alexey Artemov, Evgeny Burnaev and Andrey Lokot. Nonparametric Decomposition of Quasi-periodic Time Series for Change-point Detection // Proc. SPIE 9875, Eighth International Conference on Machine Vision, 987520 (December 8, 2015); 5 P. doi:10.1117/12.2228370;http://dx.doi.org/10.1117/12.2228370

In some of these papers models of structural brakes and change-points are also considered, see e.g. 
- page 118 in [6]
- papers [7, 8]

There were also Bayesian approaches for change-point detection, which are similar to the model of change-point, proposed in the considered paper, e.g.
[9] Ryan Prescott Adams, David J.C. MacKay. Bayesian Online Changepoint Detection // https://arxiv.org/abs/0710.3742
[10] Ryan Turner, Yunus Saatçi, and Carl Edward Rasmussen. Adaptive sequential Bayesian change point detection. In Zaïd Harchaoui, editor, NIPS Workshop on Temporal Segmentation, Whistler, BC, Canada, December 2009.

Thus,
- the paper does not provide comparison with relevant econometric literature on parametric structural time-series models,
- the paper does not provide comparison with relevant advanced change-point detection methods e.g. [7,8,9,10]. The comparison is provided only with very simple methods,
- the proposed model itself looks very similar to what can be found across econometric literature,
- the datasets, used for comparison, are very scarce. There are datasets for anomaly detection in time-series data, which should be used for extensive comparison, e.g. Numenta Anomaly Detection Benchmark.

Therefore, also the paper is well-written, 
- it lacks novelty,
- its topic does not perfectly fit topics of interest for ICLR,
So, I do not recommend this paper to be published.

Dear Reviewer,

Thank you for your comments. We have addressed them accordingly. Please see below for our response point by point.



Minor comments:
- page 3. “The observation equation and transition equations together (i.e., Equation (1,2,3)) together define “ - one “together” should be removed
- page 4. “From Figure 2, the joint distribution (i.e., the likelihood function ” - there should be additional bracket
- page 7. “We can further integral out αn “ -> integrate out

>> Thanks, we corrected the typos. 

Thus,
- the paper does not provide comparison with relevant econometric literature on parametric structural time-series models,

>> Econometric literature of refs [1, 2, 3, 4, 5]  do not properly consider and process the changing point and anomalies although they perform time-series forecasting.

- the paper does not provide comparison with relevant advanced change-point detection methods e.g. [7,8,9,10]. The comparison is provided only with very simple methods,

>> We compared state-of-the-art  Bayesian Structural Time Series (BSTS), Prophet R package by Taylor & Letham (2017),  , Exponential Smoothing State Space Model (ETS). The results are shown in Tables 2-6.  The idea of [9--10] are quite similar to them. 

- the proposed model itself looks very similar to what can be found across econometric literature,

>> The econometric literature ignores the proper treatment of changing point and anomalies. Bayesian modeling part is also different regarding estimation of posterior for hidden components given different prior distributions.
>>We add the “related work section” to illustrate the differences between our work and the existing works.

- the datasets, used for comparison, are very scarce. There are datasets for anomaly detection in time-series data, which should be used for extensive comparison, e.g. Numenta Anomaly Detection Benchmark.

>> The experimental study demonstrates that our method outperforms the other methods as well on other benchmarks for anomaly detection. Our ultimate goal is time series forecasting conditional on structure changes. It might not be that meaningful to compare in Numenta Anomaly Detection Benchmark since Anomaly Detection is kind of secondary endpoint.

Therefore, also the paper is well-written, 
- it lacks novelty,
- its topic does not perfectly fit topics of interest for ICLR,

>> There are three goals of our work: (1) time series forecasting; (2) change point detection; (3) anomalies detection;  these three goals are jointly put in one unified framework by modeling using state-space bayesian modeling.  The change point and anomalies are detected for better forecasting giving time series (structure) input.  Due to the strong description power of bayesian state-space model, the results of model prediction and abnormal and change points detection are mutually improved. Compared to the existing bayesian modeling, our work is novel by  sampling posterior to estimate hidden components given the individual Bernoulli prior of changing point and anomalies.
>> The paper is related to structure input representation and state-space modeling, which in fact is relevant to ICLR audience. We also highlighted the novelty of the work in “contribution of this work” on Page 2 in the updated version.  


Dear Reviewer,

Thank you for your time and effort on reviewing papers. Unfortunately it seems like you uploaded a WRONG review. This is possibly a review for some other paper titled "Deformation of Bregman divergence and its application", not for ours.

Dear Reviewer,

Thank you for reviewing our paper and thank you for appreciating our work. We have made changes following your suggestions. Please see below for our response point by point. Thank you.


The paper introduces a Bayesian model for timeseries with anomaly and change points besides regular trend and seasonality. It develops algorithms for inference and forecasting. The performance is evaluated and compared against state-of-the-art methods on three data sets: 1) synthetic data obtained from the generative Bayesian model itself; 2) well-log data; 3) internet traffic data.

>> Thanks for the appreciation of our work. 

From a practical perspective, one cannot expect this contribution to be ground-breaking, since there has been more than 40 years of work on time series forecasting, change point and anomaly detection. In some situations the methodology proposed here will work better than previous approaches (particularly in the situation where the data comes from the Bayesian model itself - in that case, there clearly is no better approach), in other cases (which the paper might have put less emphasis on), previous approaches will work better. To position this kind of work, I think it is important the authors discuss the limitations of their approach. 

>> As most (if not all) of the time series works, our method cannot work in every case. For example, when the time series does not have clear decomposition structure as modeled in Eqs.(1-3), the model may not correctly recover the hidden components and correspondingly perform forecasting.

Some guidelines on when or when not to use it would be valuable. Clearly, these days one cannot introduce methodology in this area and expect it to outperform existing methods under all circumstances (and hence practitioners to always choose it over any other existing method).

What is surprising is that relatively simple approaches like ETS or STL work pretty much equally well (in some cases even better in terms of MSE) than the proposed approach, while more recent approaches - like BSTS - dramatically fail. It would be good if the authors could comment on why this might be the case.

>>  BSTS fails in some cases due to the mismatch between model assumptions and actual data distribution and generation process. Usually more complicated a model is, more likely it will fail when the data structure does not satisfy its underlying assumptions. In those cases, simple approaches may achieve better performance, which is not surprising. Nevertheless, our proposed method obtains the best result.

Summary:
+ Methodology appears to be a significant, solid contribution.
- Experiments are not conclusive as to when or when not to choose this approach over existing methods
- writing needs to be improved (large number of grammatical errors and typos, e.g. 'Methods')

>> We will incorporate the discussions regarding model strength, application conditions, and the limitations in final version. 
>> We already fix the typos in updated version. 


Meta-review: Thank you for submitting you paper to ICLR.  The consensus from the reviewers is that this is not quite ready for publication. There is also concern about whether ICLR, with its focus on representational learning, is the right venue for this work.

One of the reviewers initially submitted an incorrect review, but this mistake has now been rectified. Apologies that this was not done sooner in order to allow you to address their concerns.

Decision: rejected-papers