SSMF: Shifting Seasonal Matrix Factorization

Abstract: Given taxi-ride counts information between departure and destination locations, how can we forecast their future demands? In general, given a data stream of events with seasonal patterns that innovate over time, how can we effectively and efficiently forecast future events? In this paper, we propose Shifting Seasonal Matrix Factorization approach, namely SSMF, that can adaptively learn multiple seasonal patterns (called regimes), as well as switching between them. Our proposed method has the following properties: (a) it accurately forecasts future events by detecting regime shifts in seasonal patterns as the data stream evolves; (b) it works in an online setting, i.e., processes each observation in constant time and memory; (c) it effectively realizes regime shifts without human intervention by using a lossless data compression scheme. We demonstrate that our algorithm outperforms state-of-the-art baseline methods by accurately forecasting upcoming events on three real-world data streams.


Reviews and author responses:

In this paper, the authors propose a matrix factorization method when the matrices evolve over time. For each matrix at time (t), the authors factorize it as a product of factors as well as a "seasonality tensor" that captures specifics about the time series at time (t). The proposed method is online, and can be employed in a streaming fashion. They use an MDL principle to determine whether the seasonality tensor needs to be updated. Experiments on 3 datasets show that the method outperforms baselines. 

Edit: R2 pointed out similarities to an existing work that I was unaware of. Given that and the authors not responding, I'm changing my score.   The paper is somewhat easy to follow, but there are a few places where it can be improved (mentioned in the comments). Overall, the authors consider a pretty important problem, and propose a principled way to solve it. The experimental results are compelling. There is a section on theory, but that just analyzes the computational and memory complexity, so I'd not really call those theoretical results.

Comments:
- in the definition of W, is s a user defined parameter?

- Problem 2: Why do you want to maintain a factor U, V per time period? What happens if the matrices are always factored into the same U and V, but only W captures the time variability? Presumably this is easier to train since you have fewer parameters to learn. It will be good to know experimentally or via a theoretical argument whether this would be suboptimal. 

- Line 129: this statement is not clear. What does error thresholds mean in this context?

- line 143: provide --> provided

- line 154: what does a smooth update mean?

- what is the loss function that leads to the gradient updates in (8) and (9)?

- line 178: is there a reason you want the parameters to be non negative?

- Figure 2 should have error bars. Hard to know if these results are significant.   No

both limitations and societal impact have not been addressed. 

Limitations: what is the reason for having nonnegative factors? is it merely interpretability? some of the baseline methods don't have that constraint. Similarly, does your method require a minimum number of samples observed per time instant? these questions should be answered. 

Impact: time series forecasting is a pretty broadly applicable problem, and has the potential for large impact in various important applications. The authors have not considered addressing this and should definitely do so. 

This paper proposes the Shifting Seasonal Matrix Factorization approach, SSMF, to adaptively learn multiple (dynamic) regimes and the model chooses the best regime automatically. The idea of remembering multiple regimes is very novel and can be effective especially if the temporal patterns change such as pattern A -> pattern B -> pattern A. The authors use the taxi case during Covid-19, which is a suitable case to show the effectiveness of their model.    
In general, I like the idea proposed in the paper, which is neat and really interesting, while there are still a lot of issues that may help improve the paper:

1) I have to say I am struggling to understand the benefits of the paper's main idea of remembering historical regimes (I prefer using a phrase like 'regime memory' to name the W tensor, which will help to understand the tensor's purpose easily). Apparently, if without remembering the historical regimes (and selecting the best of them), this paper has no advantage compared to SMF (Table 1). Actually, until I read the case study of Covid-19 NYC Taxi, I have not understood the purpose (This case study is really nice). I strongly suggest that the authors use the Covid-19 taxi as a motivation example to put at the beginning of the paper, so as to illustrate the necessity of using regime memory.

2) While the authors only increase the regime memory size g if the new regime performs better than all the previous ones, this still will always increase the regime memory. Hence, if the algorithm runs for a long time, g would still be large (I also would like to see the increase of g in the experiments). I suggest that authors discuss when certain regimes can be deleted from the memory W (e.g., some regimes not used for a very long time).

3) I understand that matrix factorization is the focus of the paper. However, deep learning (neural networks)  has been a very popular method for traffic prediction today. The authors should discuss the advantage/disadvantages of their proposed methods with deep learning-based methods in traffic forecasting. Besides, adding some representative deep learning methods as benchmark methods in the experiments is also necessary. I understand that SSMF has certain advantages that can deal with sparse data and better explanation in the regime, but the authors should explicitly discuss these issues in the paper.

4) Some experiment settings are not clear. What is the data sparsity in the experiments? If the data is not sparse, I think the authors should also add the experiments to add sparsity (e.g., randomly deleting some records from the historical records) to check the effectiveness of their methods in the sparse setting.

5) If real traffic prediction problems, context factors (weather, hour of day) play important roles. However, how to add these context factors into the current model is unclear.

6) Many typos and notation mistakes in the paper. For example, W^(i) in Eq. (6) should be W^(1)? W^(i)_ts in Eq. (13) should be W^(z)_ts? There are also many grammar errors. The authors please carefully proofread the paper. The paper should add a limitation discussion paragraph.

This paper proposes a matrix factorization approach for time-series forecasting when data arrives in streams. The key claimed distinguishing factor from the previous approaches is the ability to account for seasonal and regime changes. This is done by keeping multiple "seasonal" diagonal matrices diag(w_i) in the factorization X = U diag(w_i) V^T, and introducing an heuristic based on MDL to decide which w_i to use for each time step and when to introduce a new w_i. Based on 3 datasets, the authors show that this approach outperforms 3 other matrix-factorization-based forecasting methods from the literature.   Originality: This paper is heavy-based on the work in:
- Hooi, B., Shin, K., Liu, S., & Faloutsos, C. (2019, May). Smf: Drift-aware matrix factorization with seasonal patterns. In Proceedings of the 2019 SIAM International Conference on Data Mining (pp. 621-629). Society for Industrial and Applied Mathematics.
The methodology is very similar (essentially an extension of it), 2 of the 3 case studies considered are the same, even some of the figures are remarkably similar. Therefore, unfortunately, the novelty is rather low. The proposed approach can essentially be regarded as an extension of the work by Hooi et al. (2019) to include multiple matrices W_i in order to account for regime changes, and introducing an heuristic to determine which to use at given time and when to introduce a new matrix/regime. I recommend the author to make their contributions more clear in the paper and, in particular, the how the proposed approach differs from the work in Hooi et al. (2019).

Quality: The empirical results indeed show that the proposed approach outperforms 3 other approaches based on matrix factorization for time-series forecasting on the 3 case studies considered, however, I have a few concerns regarding the experimental analysis that I think the author should consider:
- The results are interesting but, at the same time, they are not that surprising. Essentially, 2 of the 3 datasets considered (Taxi demand and bike sharing demand in NYC) contain one massive regime change due to covid. It would be interesting to see whether this is the only regime change that the proposed algorithm is picking up on, and most of the improvements are simply due to a "lockdown/no lockdown" mode, or whether the proposed approach is identifying other more interesting regime changes that would less trivial to identify and account for. In other words, a detailed analysis of the regime changes detected is missing for the remaining case studies. 
- Related to the previous point, since this paper focuses on the problem of time series forecasting, simpler traditional forecasting methods should be included for comparison, and in order to understand the scale of the improvements. Additionally, the author should consider simple baselines that, for example, just copy the values of the previous week and use them as the prediction.
- The authors should provide error bars (or ideally a statistical significance analysis) for the results in Figure 2, as done in Hooi et al. (2019). Additionally, the authors should consider providing these results also in a table format, perhaps in appendix. 

Clarity: Overall, the paper is clear, but it contains several typos and grammatical mistakes that should be address. I encourage the authors to perform a careful revision, as well as to consider changing the titles of sections "3 Proposed Model" and "4 Proposed Algorithm" to something more concrete and informative, and Section "4.3 Theoretical Analysis" to "Complexity Analysis", since the former is misleading. 

Significance: Time-series forecasting is a topic with a very large audience, but, since the contribution is small in terms of methodology, I'm not sure whether this paper advances the state of the art in significant way. My previously mentioned concerns regarding the experimental analysis further aggravate this issue. I would encourage the authors to work on improving the experimental analysis in order to increase the significance of their work.  No, the authors have not addressed the limitations and potential negative societal impact of their work. I would encourage the authors to include a dedicated section in the paper for discussing these.

Meta-review: This paper proposes a matrix factorization approach for time-series forecasting for streaming data. The reviewers identified novelty as a main concern; indeed there is substantial overlap with a paper that has already been published. In addition, there were concerns about the experimental analysis.

Decision: accept