MambaTS: Improved Selective State Space Models for Long-term Time Series Forecasting

Authors: Xiuding Cai, Yaoyao Zhu, Xueyao Wang, Yu Yao
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Time Series Forcasting; State Space Model

Abstract:
In recent years, Transformers have become the de-facto architecture for long-term sequence forecasting (LTSF), yet they face challenges associated with the self-attention mechanism, including quadratic complexity and permutation invariant bias. This raises an important question: \emph{do we truly need the self-attention mechanism to establish long-range dependencies in LTSF?} Recognizing the significance of causal relationships in multivariate LTSF, we propose MambaTS, which leverages causal relationships to model global dependencies across time and variables through a single linear scan. However, causal graphs are often unknown. To address this, we introduce variable-aware scan along time (VAST), which dynamically discovers variable relationships during training and decodes the optimal variable scan order by solving the shortest path visiting all nodes problem during inference. MambaTS employs the latest Mamba model as its backbone. We suggest that the causal convolution in Mamba is unnecessary due to the presence of independent variables, leading to the development of the Temporal Mamba Block (TMB). To mitigate model overfitting, we further incorporate a dropout mechanism for selective parameters in TMB. Extensive experiments conducted on eight public datasets demonstrate that MambaTS achieves new state-of-the-art performance.

Decision: reject
Reviewer scores: 5, 6, 3, 8, 6

Reviews and discussion:
<thread>
creation date:
1730274887256

thread_type:
official_review

<thread_object>
creator: reviewer_patf

<content>
summary:
This paper presents MambaTS, an LTSF model addressing Transformers' self-attention complexity and bias by using causal relationships for global dependency modeling. The author designs variable-aware scan along time to get variable causal relationships and also Temporal Mamba Block to avoid causal convolution. The experimental results show that MambaTS outperforms several state-of-the-art models.

strengths:
1. It is interesting to see another new work on mamba for time series forecasting. In my view, some properties of mamba are fit for time series and it's an interesting direction to explore more. 
2. The authors propose several designs to tailor mamba for time series application, which has its merits.

weaknesses:
1. The clarity of the paper needs to be improved. In some parts, I cannot fully understand, such as the cost of a random walk without return. Also what is the cost from node i to node j, and how we can get this in the first iteration
2. Some claims have no support/evidence. For example, the authors mention that the random walk without return is a promising approach to estimate causal links. I would like to know the reason, e.g., any citations/proofs. 
3. The experiments seem not comprehensive. The authors only compare MambaTS with 7 baselines. There are a few more after iTransformer, which are worth to be compared. E.g., ModernTCN [1], UniTST [2], TSLANet [3]. 


References:

[1] ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis. 

[2] UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting. 

[3] TSLANet: Rethinking Transformers for Time Series Representation Learning

questions:
1. In proposition 1, the assumption is that the causal graph exists. What if it doesn't exist? And is there any support on the random walk without return is a promising approach to estimate causal links? 
2. What is the definition of cost C and how we can get/set it empirically (e.g., the cost from node i to node j)?
3. Proposition 2 indicates that theoretically the causal relationships can be estimated without infinite random walks with return. I would like to ask how many walks are required empirically. And also the time spent? 
4. In Eq (6), how we can get the p^{(0)}? 
5. In my view, another major difference between MambaTS and iTransformer is that MambaTS model the dependencies on both time and variables, while iTransformer mainly on variables. I would like to know how this contributes to eventual performance. Because UniTST [1] is also modeling the dependencies on both time and variables dimensions, but with Transformer architecture. How does MambaTS compare with UniTST? 

Reference:

[1] UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730517278315

thread_type:
official_review

<thread_object>
creator: reviewer_yzlz

<content>
summary:
The paper introduces MambaTS, an improved selective state space model for long-term time series forecasting. The model leverages a novel method for variable-aware scanning along time (VAST) to model global dependencies in a time series with variable missing rates and different intervals. By utilizing a combination of causal graphs and shortest path solutions, MambaTS addresses the limitations of previous Transformer-based models which often struggle with high computational costs and inefficient handling of long-range dependencies.

strengths:
1. The introduction of VAST and the use of causal graphs for modeling dependencies offers a unique solution to efficiently process long-term dependencies in time series data with linear complexity.
2. The model is tested across various public datasets, demonstrating superior performance compared to existing state-of-the-art models. This not only validates the efficacy of MambaTS but also showcases its versatility in handling different types of time series data.
3. MambaTS significantly reduces the computational cost traditionally associated with long-range forecasting models like Transformers by avoiding the quadratic complexity of the self-attention mechanism.

weaknesses:
1. The effectiveness of the model heavily depends on the accuracy of the causal graphs. Incorrect or incomplete causal relationships can lead to suboptimal forecasting results, which the paper does not extensively address in terms of robustness against poor graph structure
2. While the model shows high efficiency and effectiveness, the paper lacks a thorough discussion on scalability, especially in scenarios with exceedingly large datasets or highly complex variable relationships.
3. There is a need for a comparison of the model’s performance with other SOTA methods, such as Onefitsall, TimeLLM etc.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730540712973

thread_type:
official_review

<thread_object>
creator: reviewer_dr3c

<content>
summary:
This paper proposes MambaTS, a selective state-space model for long-term time series forecasting (LTSF) that addresses the computational limitations of Transformers by leveraging causal relationships across variables and time with a single linear scan.

strengths:
1. LTSF presents a compelling and complex challenge.
2. The experiments are thorough but still lack some essential details.

weaknesses:
1. **Lack of Experimental Details:** Important implementation details are missing, such as patch length, the value of beta in Equation 7, and whether the random walk on variables is conducted K-1 times per epoch (meaning  k-1 more training time cost than one epoch).
2. **Efficiency Concerns:** Theoretical complexity analysis in Table 5 lacks practical runtime comparisons. Given that MambaTS requires K-1 iterations to estimate causal relationships, its efficiency is questionable.
3. **Incomplete Ablation Studies:** The paper introduces the TMB (with dropout replacing the original convolution), but no ablation study compares TMB and the original Mamba block, leaving its impact on performance unclear.
4. **Limited Explanation in Variable-Aware Scanning:** Section 5.2 does not clearly explain whether K-1 transitions are sufficient to estimate all variable orders, or if consistency (e.g., v1 always preceding vk) is assumed.
5. **Limited Benchmarking:** Two commonly used datasets (ETTh1, ETTm1) are missing, which reduces the generalizability of the results.
6. **Code Availability:** No code is provided, limiting reproducibility.
7. **Unpersuasive SOTA Claims:** Results in Table 2 are questionable. For example, our reimplementation of PatchTST (using official configurations) achieved better results than the reported MambaTS performance on ETTm2 (input length 720). Specifically:
   - ETTm2_720_96.log: 0.1632, 0.2555
   - ETTm2_720_192.log: 0.2167, 0.2942
   - ETTm2_720_336.log: 0.2679, 0.3282
   - ETTm2_720_720.log: 0.3521, 0.3798

   These results suggest MambaTS may not definitively outperform all baselines, especially as no code is available for direct comparison.
8. **Notation Issue:** The meaning of \( I \) in Equation 7 is unclear.
9. **Inference Process Detail:** Section 5.2 lacks details on the inference process for Variable-Aware Scan Along Time.

questions:
1. How is the patch length chosen, and does it vary across datasets?
2. Could the authors clarify the random walk process and the number of epochs used in variable scanning?
3. Why were benchmarks ETTh1 and ETTm1 not included?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720000775

thread_type:
official_review

<thread_object>
creator: reviewer_zjn1

<content>
summary:
This paper introduces MambaTS, an architecture for long-term time series forecasting that models global dependencies efficiently with a linear scan, avoiding the computational challenges of self-attention. The Variable-Aware Scan along Time (VAST) mechanism dynamically infers causal relationships among variables using random walks and determines an optimal scanning order through heuristic path decoding. This design achieves scalability and adaptability, particularly for complex, high-dimensional datasets with unknown causal structures.

strengths:
- MambaTS reduces computational complexity from quadratic O(K^2) to linear O(K) by leveraging a topologically ordered linear scan, making it suitable for high-dimensional time series data.
- VAST enhances adaptability by inferring causal relationships in the absence of explicit causal graphs, using random walks to approximate dependencies and mitigate the need for exhaustive pairwise calculations.

weaknesses:
Reliance on heuristic optimization for scanning order yields sub-optimality:
- The variable-aware san along time (VAST) employs the asymmetric traveling salesman problem (ATSP) to determine the optimal scanning order, relying on heuristics like simulated annealing to address its NP-hard nature. Although heuristics provide feasible solutions, this dependency introduces inconsistency, as different approximations may affect the accuracy of variable ordering (in case of complex, dense inter-variable connections)
- Extra experiments on alternative heuristic approaches such as genetic algorithms (that are powerful in navigating NP-hard problems) could reveal a more stable and efficient approach. In the same vein, additional experiments measure how different heuristic methods affect the resulting scanning order and, subsequently, forecasting accuracy. This can help users determine if any heuristic consistently produces a favorable scanning order.

Convergence guarantee or confidence interval is not covered in causal estimation which lacks usability:
- Proposition 2 lacks formal guarantees for convergence speed, raising questions about the robustness of causality inference in finite settings. Without clear bounds on the number of walks required, the approach may yield only approximate estimates, especially when practical constraints limit the number of walks. This limitation affects the consistency and reproducibility of causal estimation results, as reliance on empirical averaging may not ensure reliable causal inference across varied dataset structures. 
- It might be helpful to introduce a stopping rule based on convergence metrics (e.g. average change in transition costs), or introduce confidence intervals on causality estimates to users to give insight into the stability of causal inferences under finite computational budgets, where both suggestions seem to be beyond the scope of this study. I hope the authors consider usability in the future works.

questions:
The paper offers a well-reasoned and innovative approach to time series forecasting, with theoretically sound propositions and a practical methodology that balances computational efficiency with modeling accuracy. While the heuristic reliance on VAST and scalability issues in dense graphs present limitations, the model’s strengths in efficiency, adaptability, and architectural design make it a valuable contribution. MambaTS is especially promising for high-dimensional and complex time series data, though further work is recommended to address heuristic dependency and enhance robustness in varied causal structures. Overall, it's a solid and innovative work on time-series forecasting, effectively incorporating causality in a computationally efficient and scalable manner.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731139124004

thread_type:
official_review

<thread_object>
creator: reviewer_8tnh

<content>
summary:
This paper introduce MambaTS, a new time series forecasting model based on selective state space models. In order to tackle multivariate forecasting, the timeseries patches of each variable is unrolled in a certain order to form a single sequence. One key innovation of the paper is a method for estimating the causal relationship between variables during training via random walk without return.

strengths:
The paper propose a strategy to apply Mamba to to multivariate ts forecasting and achieves empirical result comparable to SOTA.

weaknesses:
1. The proof in Proposition 2 does not make sense to me. I am not sure the whole concept of random walk on a casual graph with certain cost is well defined in the paper.
2. The proposed method claims to leverage the causal dependency between the variables and thus is more suitable in the multivariate setting. However, it does not seems to have a large advantage over chanel independent PatchTST, which is univariate forecasting method.

questions:
I don't see why a random permutation is equivalent to a random walk. Line 324 says "K − 1 transition tuples ${(v_1, v_2),(v_2, v_3), · · ·(v_{K−1}, v_K)}$ are derived". I wonder what prevent the authors from deriving K(K-1)/2 tuples, so that each $(v_i, v_j), i<j$ is included?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733226541263

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
We sincerely appreciate the insightful and constructive feedback from the reviewers, which has significantly contributed to improving the quality of our paper.

We are deeply encouraged that the reviewers recognized the key contributions of our paper, including **the reduction of quadratic complexity to linear complexity** for variable dependencies modeling, the novel random walk-based method for estimating causal relationships, and the innovative application of causal graphs for handling long-range dependencies. They also highlighted the **theoretical soundness** of our approach, and recognized MambaTS as a **well-reasoned, practical** solution to challenges in **high-dimensional time series data**. The empirical validation of MambaTS was well-received, with its consistent state-of-the-art performance across diverse datasets highlighting its versatility and effectiveness.

The reviewers also provided valuable insights and constructive concerns, all of which we have addressed in detail. The revisions we made are summarized as follows:

- **Proposition 2 Proof (Reviewer 8TNh, Reviewer pAtf)**: Both reviewers expressed concerns regarding the clarity of Proposition 2. In response, we have provided a more detailed formal proof and additional references to address these concerns. Furthermore, we have included a visualization of the learned variable dependency graph in Appendix Figure 5. 
- **Comparison with More SOTA Models (Reviewer YZLZ, Reviewer pAtf)**: Reviewer YZLZ suggested comparing MambaTS with large-scale models like Onefitsall and TimeLLM, while Reviewer pAtf recommended post-iTransformer models such as ModernTCN, UniTST, and TSLANet. We included these comparisons in Appendix Table 6. Our results show that MambaTS maintains a competitive edge, especially on large, complex datasets like Traffic.
- **Convergence Guarantee (Reviewer zjn1, Reviewer dR3c, Reviewer pAtf)**: In Proposition 2, we establish the theoretical convergence of the relationship matrix P as the random walk progresses towards infinity. While we did not provide a precise formalization for the exact number of iterations required for full convergence, our experiments revealed that by the end of training, the causal relationship estimation typically converges, as further validated by Figure 5 in the Appendix. We sincerely appreciate the constructive feedback from the reviewers and plan to explore this aspect in more detail in future work.
- **Implementation Discrepancies with PatchTST (Reviewer dR3c)**: We thank Reviewer dR3c for pointing out the discrepancies with PatchTST results. We ensured a fair comparison by adopting the training and evaluation protocol from TimesNet, which uses 10 epochs for training, widely adopted in recent works like iTransformer (ICLR'24), TSLANet (ICML'24), and TimeMixer (ICLR'24). In contrast, PatchTST typically trains for 100 epochs with a more carefully designed learning rate schedule. To address the reviewer’s concerns, we compared our results with the original PatchTST. Our experiments show that MambaTS outperforms PatchTST, even with 100 epochs of training, especially on complex datasets like Electricity and Traffic.
- **Additional Comparison with iTransformer**: iTransformer is a robust baseline for modeling multivariate dependencies in time series forecasting. 
  - **Reviewer dR3c**: The theoretical complexity of MambaTS is $\mathcal{O}(\frac{KL}{P})$, where $K$ represents the number of variables, $L$ is the sequence length, and $P$ is the patch size, while iTransformer has a complexity of $\mathcal{O}(K^2)$. To address runtime concerns, we provide empirical comparisons showing that MambaTS is approximately 3x faster than iTransformer, particularly on large, complex datasets like Traffic. This validates our theoretical analysis regarding the efficiency gains offered by MambaTS over traditional Transformer models.
  - **Reviewer pAtf**: We also introduced iMambaTS, which we compared with both MambaTS and iTransformer. Our results indicate that iMambaTS performs less effectively than MambaTS, thus emphasizing the benefits of our Variable Scan along Time (VST) method. Furthermore, iMambaTS outperforms iTransformer on most datasets, emphasizing the benefits of the MambaTS architecture and the VAST-based variable relationship estimation.
- **Additional Decoding Strategies (Reviewer zjn1)**: In response to suggestions for further decoding strategies, we implemented a genetic algorithm (GA) for decoding and conducted additional experiments. The results show that MambaTS+GA consistently achieves similar performance, confirming the robustness and stability of our method.

We also addressed specific writing and notation concerns raised by the reviewers, enhancing the clarity of the manuscript. Hope that our response can address reviewers' concerns.

We believe these revisions have significantly strengthened our work. Once again, we thank all the reviewers for their thoughtful and constructive comments.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces MambaTS, a new time series forecasting model based on selective state space models which leverages causal relationships to model global dependencies. Adapting Mamba-style SSMs to the Time Series Space is certainly an interesting direction that is worth investigating, and the paper proposes several interesting design choices to tailor mamba for time series. However several reviewers expressed concerns with the paper around presentation clarity (the propositions and proofs lack formal rigor) and more importantly around soundness of the experimental results. In particular, there were valid questions by reviewers (that the AC agrees with) around the lack of detailed model efficiency comparisons with MLP models, and around both the competitiveness of the model to PatchTST, and fidelity of the reported baseline metrics, especially when baseline models are not trained to convergence. Furthermore the lack of error bars and the limited number of datasets considered for evaluation makes it hard to judge the significance of the gains over SOTA baselines.

I would urge the authors to conduct a more extensive and through evaluation of the model and resubmit to a future venue.