MambaTS: Improved Selective State Space Models for Long-term Time Series Forecasting

## Abstract

Abstract In recent years, Transformers have become the de-facto architecture for long-term sequence forecasting (LTSF), but faces challenges such as quadratic complexity and permutation invariant bias. A recent model, Mamba, based on selective state space models (SSMs), has emerged as a competitive alternative to Transformer, offering comparable performance with higher throughput and linear complexity related to sequence length. In this study, we analyze the limitations of current Mamba in LTSF and propose four targeted improvements, leading to MambaTS. We first introduce variable scan along time to arrange the historical information of all the variables together. We suggest that causal convolution in Mamba is not necessary for LTSF and propose the Temporal Mamba Block (TMB). We further incorporate a dropout mechanism for selective parameters of TMB to mitigate model overfitting. Moreover, we tackle the issue of variable scan order sensitivity by introducing variable permutation training. We further propose variable-aware scan along time to dynamically discover variable relationships during training and decode the optimal variable scan order by solving the shortest path visiting all nodes problem during inference. Extensive experiments conducted on eight public datasets demonstrate that MambaTS achieves new state-of-the-art performance.

## 1 Introduction

Long-term time series forecasting (LTSF) has a wide range of applications in various fields, including weather, finance, healthcare, energy and transportation ; ; . With the rapid advancement of deep learning, the current methods for time series prediction have shifted from traditional statistical learning approaches to deep learning-based methods, such as recurrent neural networks (RNNs) and temporal convolutional neural networks (TCNs; ; ). Since the introduction of Transformer , Transformer-based methods have emerged as the mainstream LTSF approach ; , leveraging their self-attention mechanism to effectively capture long-term dependencies in time series data.

However, recent studies have highlighted challenges faced by Transformers in LTSF. Firstly, these methods suffer from the curse of quadratic complexity, where the computational cost rapidly increases with the length of the context ; ; . Additionally, some studies have observed that the performance of Transformer-based LTSF methods does not necessarily improve with the increasing look-back window ; . Notably, a recent study called DLinear  has questioned the effectiveness of Transformers for permutation invariant bias in LTSF, achieving surprising results and surpassing most state-of-the-art (SOTA) Transformer-based methods using a single-layer feed-forward network.

Figure: Table 1: Case study: Mamba for LTSF tasks.
Refer to caption: x1.png

The recent popularity of state space models (SSMs; ; ; ) has caught our attention. SSMs are principled sequential models that describe the evolution of states over time using ordinary differential equations, making them naturally suitable for time series modeling. Recently, Mamba  enhances traditional SSMs by introducing selection mechanisms to filter out irrelevant information and reset states along. It also incorporates hardware-aware design for efficient parallel training. Mamba has demonstrated competitive performance compared to Transformers across various domains ; ; , offering rapid inference and scalability with sequence length.

The inherent sequential and selective nature of Mamba has driven us to validate its performance in LTSF and compare it to Transformer. Initially, we replaced the Transformer Block in two SOTA Transformer-based methods, PatchTST  and iTransformer , with Mamba Block, resulting in corresponding Mamba versions(^1^11Following the practice of Mamba, we removed positional encoding as the nature of sequence modeling does not necessitate it.). Subsequently, we compared them with PatchTST and iTransformer, with the results presented in Table 1 and Figure 1. Our findings indicate that, compared to PatchTST and iTransformer, the Mamba versions exhibited a 1.3x faster training speed, with memory consumption reduced by 5.3x and 7.0x, respectively. However, despite these improvements, we did not observe a performance advantage of Mamba over Transformer on these LTSF datasets.

This seems to contradict the intuition that Mamba is naturally more suitable for time series modeling. We made four improvements to Mamba and introduced MambaTS, a framework specifically designed for LTSF based on improved selective SSMs. Firstly, we introduce variable scan along time (VST). Similar to PatchTST , we initially segment every variable into patches and linearly map them to tokens. However, unlike PatchTST, we adopt a variable mixing manner by alternately organizing the tokens of different variables at the same timestep, thus expanding them along time to form a global historical look-back window. *However, the lack of identified causality among variables results in the locally non-sequential spanning outcomes after VST.* In the case, causal convolution in the original Mamba Block may impact on model performance. In addition, some studies have argued that causal convolution for the look-back window is unnecessary and may restrict temporal feature extraction ; . To address these issues, we introduce the Temporal Mamba Block (TMB) by removing local convolution before SSM. Additionally, to mitigate overfitting, we incorporated a dropout mechanism  into selective parameters of TMB, building on findings that excessive information integration can lead to overfitting , as observed in our experiments with Mamba.

While TMB is tailored for temporal modeling, it remains sensitive to scan order, highlighting the significance of variable organization. To address this, we introduce variable permutation training (VPT). By shuffling the variable order in each iteration, we mitigate the impact of undefined variable order and boost the model’s local interaction capacity. *An ensuing question is how do we determine the optimal channel scan order?* Inspired by topological sorting ; in graph theory, we further propose a novel Variable-Aware Scan along Time (VAST). In VAST, we consider the position of nodes in a linear sequence, such as $a\to c,b\to c$, where a reasonable topological order might be $a\to b\to c$. Specifically, during training, we update a distance matrix that captures variable relationships based on forward propagation results following each permutation. This formulation transforms the optimal scan order problem into an instance of the asymmetric traveling salesman problem (ATSP) in a dense graph. To address this, we utilize a simulated annealing-based ATSP solver to derive the final channel scan order. To our knowledge, we are the first to investigate variable scan order in the context of SSMs for time series modeling.

Through the aforementioned four designs, MambaTS achieves efficient modeling of global dependencies across time and variables with linear complexity. Extensive experiments on eight popular public datasets demonstrate that MambaTS achieve SOTA performance in most LSTF tasks and settings.

In summary, our contributions are as follows:

- •
We introduce MambaTS, a novel time series forecasting model that builds upon improved selective SSMs. By incorporating VST, we effectively organize historical information of all variables to create a global retrospective sequence.
- •
We present TMB, which addresses the dispensability of causal convolutions in vanilla Mamba for LTSF. Furthermore, we incorporate a dropout mechanism for the selective parameters of TMB to mitigate model overfitting.
- •
We implement VPT strategy to eliminate the influence of undefined variable order and further enhance the model’s local context interaction capabilities.
- •
We propose VAST, which discover relationships between different variables during training and leverages an ATSP solver to determine the optimal variable scan order during inference.

## 2 Related Work

#### Long-term time series forecasting

Traditional LTSF methods leverage the statistical properties and patterns of time series data for prediction . In recent years, LTSF has shifted towards deep learning approaches, where various neural networks are utilized to capture complex patterns and dependencies, elevating LTSF performance. These methods can be broadly classified into two categories: variable-mixing and variable-independent. Variable-mixing methods employ diverse architectures to model dependencies across time and variables. RNNs ; ; were initially introduced to LTSF due to the nature of sequence modeling. TCNs, known for their local bias, are effective in capturing local patterns in time series data and have shown promising results in LTSF ; ; . Transformers are subsequently introduced to accomplish long-range dependency modeling through self-attention and have become a mainstream method in LSTF . However, due to the quadratic complexity, Transformer-based methods have struggled with optimization efficiency ; ; ; ; . Recently, significant improvements have been made in these methods with patch-based techniques ; . MLPs are also commonly used for LTSF and have achieved impressive results with their simple and direct architectures . Graph neural networks have been utilized to model relationships between variables . FourierGNN  represents the entire time series information as a hypervariate graph and employs Fourier Graph Neural Network for global dependency modeling. On the other hand, variable-independent methods focus solely on modeling temporal dependencies under the assumption of variable independence ; ; . These approaches are known for their simplicity and efficiency, often capable of mitigating model overfitting and achieving remarkable outcomes ; . Nevertheless, this assumption may oversimplify the problem and potentially lead to ill-posed scenarios .

#### State Space Models

Recently, some works  have combined SSMs  with deep learning and demonstrated significant potential in addressing the long-range dependencies problem. However, the prohibitive computation and memory requirements of state representations often hinder their practical applications . Several efficient variants of SSMs, such as S4 , H3 , Gated State Space , and RWKV , have been proposed to enhance model performance and efficiency in practical tasks. Mamba  addresses a key limitation of traditional SSMs methods by introducing a data-dependent selection mechanism based on S4 to efficiently filter specific inputs and capture long-range context that scales with sequence length. Mamba demonstrates linear-time efficiency in modeling long sequences and surpasses Transformer models in benchmark evaluations .

Mamba has also been successfully extended to non-sequential data such as image ; ; , point cloud , table  and graphs ; to enhance its capability in capturing long-range dependencies. To address the scan order sensitivity of Mamba, some studies have introduced bidirectional scanning , multi-directional scanning ; , and even automatic direction scanning . However, there is currently limited work considering the issue of variable scan order in temporal problems. To tackle this challenge, we introduce the VAST strategy to further enhance the expressive power of MambaTS. Similar to our approach, Graph-Mamba  also proposes a similar permutation strategy to extend context-aware reasoning on graphs. However, it is based on node prioritization, introducing a biased strategy specifically designed for graphs.

## 3 Preliminaries

#### State Space Models

SSMs are typically regarded as linear time-invariant (LTI) systems that map continuous input signals $x(t)$ to corresponding outputs $y(t)$ through a state representation $h(t)$. This state space describes the evolution of the state over time and can be represented using ordinary differential equations as follows:

$$ $\displaystyle h^{\prime}(t)$ $\displaystyle=\bm{A}h(t)+\bm{B}x(t)$ (1) $\displaystyle y(t)$ $\displaystyle=\bm{C}h(t)+\bm{D}x(t)$ $$

Here, $h^{\prime}(t)=\frac{dh(t)}{dt}$, and $\bm{A},\bm{B},\bm{C}$, and $\bm{D}$ are parameters of the time-independent SSMs.

#### Discretization

Finding analytical solutions for SSMs is highly challenging due to their continuous nature. Discretization is typically employed to facilitate analysis and solution in the discrete domain, which involves approximating the continuous-time state space model into a discrete-time representation. This is done by sampling the input signals at fixed time intervals to obtain their discrete-time counterparts. The resulting discrete-time state space model can be represented as:

$$ $\displaystyle h_{k}$ $\displaystyle=\overline{\bm{A}}h_{k-1}+\overline{\bm{B}}x_{k}$ (2) $\displaystyle y_{k}$ $\displaystyle=\overline{\bm{C}}h_{k}+\overline{\bm{D}}x_{k}$ $$

Here, $h_{k}$ represents the state vector at time instant $k$, and $x_{k}$ represents the input vector at time instant $k$. The matrices $\overline{A}$ and $\overline{B}$ are derived from the continuous-time matrices $A$ and $B$ using appropriate discretization techniques such as the Euler or ZOH (Zero-Order Hold) method. In this case, $\overline{\bm{A}}=\exp(\Delta\bm{A}),\overline{\bm{B}}=(\Delta\bm{A})^{-1}(
\exp(\Delta\bm{A})-\bm{I})\cdot\Delta\bm{B}$.

#### Selective Scan Mechanism

Mamba further introduces selective SSMs by allowing the parameters to influence the interactions along the sequence in a context-dependent manner. This selective mechanism enables Mamba to filter out irrelevant noise in time series tasks, while selectively propagating or forgetting information relevant to the current input. This differs from previous SSMs methods with static parameters, but it does break the LTI characteristics. Therefore, Mamba takes a hardware optimization approach and implements parallel scan training to address this challenge.

## 4 Model architecture

For the multivariate time series prediction problem, given an $L$-length look-back window time series data with $K$ variables $(\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{L})$, where $\mathbf{x}_{i}\in\mathbb{R}^{K}$ represents the values of $K$ variables at time step $i$. Our objective is to forecast the values of the future $T$ time steps, denoted as $(\mathbf{x}_{L+1},\cdots,\mathbf{x}_{L+2},\cdots,\mathbf{x}_{L+T})$.

### 4.1 Overall Architecture

Figure: Figure 2: The overall architecture of MambaTS.
Refer to caption: x2.png

The architecture of MambaTS is illustrated in Figure 2. It primarily consists of an embedding layer, an instance normalization layer, $N\times$ Temporal Mamba blocks, and a prediction head.

#### Patching and Tokenization

Semantic information at a single time point tends to be sparse. Following PatchTST , we segment each variable into about $M=L/s$ patches every $s$ time steps, aggregate information from neighboring points, and then map them to a $D$-dimensional token through linear mapping. This process efficiently reduces memory usage and eliminates redundant information.

#### Variable Scan along Time

By embedding $K$ variables, we derive $K\times M$ tokens. To fully leverage the linear complexity and selective advantages of Mamba while establishing a comprehensive representation of the look-back window, we introduce the variable scan along time mechanism. VST arranges tokens of variables at each time step in an alternating fashion temporally. This structured organization enables the model to more accurately capture long-term dependencies and dynamic changes in time series data. Subsequently, we feed the results of VST into the encoder.

#### Encoder

The encoder consists of $N$ stacked TMBs, each comprising two branches. The right SSM branch focuses on sequence modeling, while the left branch contains a gated non-linear layer. The computation process of the original Mamba block is as follows:

$$ $h_{t}=\text{SSM}(\text{Conv}(\text{Linear}(\mathbf{x_{t}})))+\sigma(\text{ Linear}(\mathbf{x_{t}})).$ (3) $$

Here, causal convolution, acting as a shift-SSM, is inserted before the SSM module to enhance connections between adjacent tokens, as depicted in Figure 2 (b). However, the results generated by VST may not be causal locally. Therefore, TMB (see Figure 2 (c)) removes this component. Additionally, to prevent overfitting, we introduce a dropout mechanism  for the selective parameters, as follows:

$$ $h_{t}=\text{SSM}(\text{Dropout}(\text{Linear}(\mathbf{x_{t}})))+\sigma(\text{ Linear}(\mathbf{x_{t}})).$ (4) $$

#### Prediction Head

Since the encoder is capable of capturing global dependencies, for efficiency reasons, similar to PatchTST , we adopt a channel-independent decoding approach in the decoding phase. Each channel is decoded individually using a simple linear head.

#### Instance Normalization

To mitigate distribution shift between training and test data, following RevIN , we standardize each input channel to zero mean and unit standard deviation, and keep track of these statistics for de-normalization of the model prediction.

#### Loss Function

We choose mean squared error (MSE) loss as the primary loss function, given by:

$$ $\mathcal{L}=\mathbb{E}_{\mathbf{x}}\frac{1}{M}\sum_{i=1}^{M}\left\|\hat{ \mathbf{x}}_{L+1:L+T}^{(i)}-\mathbf{x}_{L+1:L+T}^{(i)}\right\|_{2}^{2}.$ (5) $$

### 4.2 Variable Permutation Training

To mitigate the impact of undefined channel orders and augment local context interactions, we introduce the variable permutation training (VPT) strategy. Specifically, for the encoder input consisting of $K\times M$ tokens, VPT shuffle the $K$ tokens at each time step in a consistent order, and revert the shuffle state after decoding to ensure the correct output of the sequence.

### 4.3 Variable-Aware Scan along Time

To determine the optimal variable scan order, it is essential to consider the relationships between variables. However, since these relationships are unknown, we propose VAST, a simple yet effective method to estimate variable relationships, guiding the scan order during inference stage.

#### Training

For any $K$ variables, we maintain a directed graph adjacency matrix $\bm{P}\in\mathbb{R}^{K\times K}$, where $p_{i,j}$ represents the cost from node $i$ to node $j$. Notably, with the introduction of variable permutation during training, we can explore various combinations of scan orders and evaluate their effectiveness using Eq. (5). Following a permutation, a node index sequence $\mathbf{V}=\{v_{1},v_{2},\cdots,v_{K}\}$ is obtained, where $v_{k}$ denotes the new index in the shuffled sequence. Subsequently, $K-1$ transition tuples $\{(v_{1},v_{2}),(v_{2},v_{3}),\cdots(v_{K-1},v_{K})\}$ are derived. For each sample, a training loss ${l}^{(t)}$ is computed during the $t$-th iteration of the network. Therefore, we update $\bm{P}$ with exponential moving average:

$$ $p^{(t)}_{v_{k},v_{k+1}}=\beta p^{(t-1)}_{v_{k},v_{k+1}}+(1-\beta)l^{(t)},$ (6) $$

where $\beta$ is a hyperparameter that controls the rate of the moving average, determining the impact of new estimates on the variable relationships. To facilitate efficient training, we extend Eq. (6) to a batch version. By a simple centralization operation to eliminate the influence of different sample batches, i.e., $\bar{l}^{(t)}={l}^{(t)}-\mu({l}^{(t)})$, where $\mu$ is the mean function, the equation is modified to:

$$ $p^{(t)}_{v_{k},v_{k+1}}=\beta p^{(t-1)}_{v_{k},v_{k+1}}+(1-\beta)\bar{l}^{(t)}.$ (7) $$

#### Inference

Throughout training, $\bm{P}$ are leveraged to determine the optimal variable scan order. This involves solving the asymmetric traveling salesman problem, which seeks the shortest path visiting all nodes. Given the dense connectivity represented by $\bm{P}$, finding the optimal traversal path is NP-hard. Hence, we introduce a heuristic-based simulated annealing  algorithm for path decoding.

## 5 Experiments

Table: Table 2: Summary of Dataset Characteristics

#### Dataset

We conducted extensive experiments on eight public datasets, as shown in Table 2, including two ETT datasets , Weather, Electricity, Traffic , Solar , Covid-19 , and PEMS , covering domains such as electricity, energy, transportation, weather, and health.

#### Baselines and metrics

To demonstrate the effectiveness of MambaTS, we compared it against SOTA models of LTSF, including five popular Transformer-based methods: PatchTST , iTransformer , FEDformer , Autoformer , and three competitive non-Transformer-based methods: DLinear , MICN , and FourierGNN . Following PatchTST , we primarily evaluate the models using Mean Squared Error (MSE) and Mean Absolute Error (MAE).

#### Implementation Details

Experiments were performed on an NVIDIA RTX 3090 Ti 24 GB GPU using the Adam optimizer  with betas of (0.9, 0.999). Training ran for 10 epochs with early stopping implemented using a patience of 3 to avoid overfitting. The best parameter selection for all comparison models was carefully tuned on the validation set.

Table: Table 3: Multivariate long-term series forecasting results. All models employ a look-back window length of $L=96$ for the Covid-19 dataset and $L=720$ for the remaining datasets.

**Table 4: Ablations on components. VST: Variable Scan along Time. TMB: Temporal Mamba Block. VAST: Variable-Aware Scan along Time. The average results of all predicted lengths are listed here.**
| VST | TMB | VAST | ETTm2 | Traffic | Electricity | Solar |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |  |  |
| $\Circle$ | $\Circle$ | $\Circle$ | 0.285 | 0.342 | 0.400 | 0.273 | 0.167 | 0.260 | 0.192 | 0.261 |
| $\CIRCLE$ | $\Circle$ | $\Circle$ | 0.284 | 0.341 | 0.383 | 0.268 | 0.164 | 0.260 | 0.197 | 0.266 |
| $\Circle$ | $\CIRCLE$ | $\Circle$ | 0.264 | 0.329 | 0.389 | 0.268 | 0.160 | 0.255 | 0.191 | 0.253 |
| $\CIRCLE$ | $\CIRCLE$ | $\Circle$ | 0.263 | 0.327 | 0.376 | 0.267 | 0.161 | 0.257 | 0.193 | 0.261 |
| $\CIRCLE$ | $\CIRCLE$ | $\CIRCLE$ | 0.262 | 0.325 | 0.373 | 0.262 | 0.155 | 0.251 | 0.184 | 0.247 |

### 5.1 Main results

Table 3 displays the results of multivariate long-term forecasting. Overall, MambaTS achieved new SOTA results (highlighted in red bold) across various prediction horizons on most datasets. While DLinear and PatchTST assume variable independence and perform well on datasets with a small number of variables like ETTh2/m2 ($K=7$) and Weather ($K=21$), their performance diminishes on complex datasets with a larger number of variables such as Traffic ($K=862$) and Covid-19 ($K=948$), highlighting the limitations of the variable-independent assumption. Conversely, iTransformer exhibited contrasting performance, excelling on intricate datasets but underperforming on datasets with fewer variables. Other baselines demonstrated competitive results on specific datasets under certain prediction scenarios.

### 5.2 Ablation studies and analyses

To validate the rationality and effectiveness of the proposed components, we conducted extensive ablation experiments as shown in Table 4, Table 5, Figure 3 and Figure 4.

#### Component Ablation

Table 4 presents the ablation on components. Initially, integrating VST with the baseline Mamba-based PatchTST led to performance improvements across most datasets, showcasing the benefits of considering all variables. Substituting solely with TMB resulted in a notable performance boost, underscoring the efficacy of TMB for temporal modeling. Combining VST and TMB yielded performance superior to VST alone but slightly below the original TMB, attributed to VST including all variables while TMB removes local bias causal convolutions, making it more sensitive to variable order. However, this issue is addressed by introducing VAST (see Table 4, row 4). Through these components, MambaTS achieves optimal performance.

Figure: Figure 3: Dropout ablations of TMB. Left: TMB with varying dropout rates on Weather dataset. Right: Loss curves over training.
Refer to caption: x3.png

#### Dropout Ablation

We further analyzed the role of the dropout in TMB. Figure 3 (left) shows the results of MambaTS with different dropout rates (0.1-0.5) on the Weather dataset. Compared to no dropout, the reduction in MSE of MambaTS increased with the dropout rate, achieving the best performance at 0.2 and 0.3, and performance degradation beyond 0.4. Figure 3 (right) illustrates corresponding loss curves during training, indicating that the dropout in TMB helps prevent premature convergence and overfitting. Additionally, we observed that dropout facilitated lower validation loss.

#### VAST Ablation

In Table 5, we conducted extensive ablation studies on the VAST strategy, focusing on the design and selection of path decoding strategies. "W/o VPT" in Table 5 indicates that MambaTS was trained without VPT as the baseline. "Random (100x)" represents sampling 100 test runs after training with VPT and averaging the results. It can be observed that "Random (100x)" significantly outperformed "W/o VPT", which underscores the effectiveness of VPT. Further visual comparisons in Figure 4 show that even random variable scanning outperforms "W/o VPT" in most cases. We then explored different heuristic decoding strategies, including Greedy Strategy (GD), Local Search (LS), Lin and Lernighan (LK), and Simulated Annealing (SA). We defaulted to adopting SA as our solver. As a trade-off between efficiency and performance, we did not employ an exact ATSP solver due to its exponential complexity. As shown in Table 5 and Figure 4, SA consistently outperformed other solvers in terms of relative performance consistency.

Figure: Table 5: Ablations on VAST.
Refer to caption: x4.png

Figure: Figure 5: Performance of MambaTS in different datasets with varying length look-back windows.
Refer to caption: x5.png

Table: Table 6: Computational complexity analysis. SA: Self-attention. Conv: Convolution.

### 5.3 Model Analysis

#### Increasing Lookback Window

Previous studies have shown that Transformer-based methods may not necessarily benefit from a growing lookback window ; , possibly due to distracted attention over the long input. In Figure 5, we assess MambaTS’s performance in this context and compare it with several baselines. It can be observed that MambaTS consistently demonstrates the ability to benefit from the growing input sequence. iTransformer, PatchTST, and DLinear also show this benefit, but MambaTS’s overall curve is lower than PatchTST and DLinear. Compared to iTransformer, MambaTS benefits more from a longer lookback window. Additionally, we notice that iTransformer seems to exhibit discontinuous gains on individual dataset tasks.

#### Efficiency Analysis

MambaTS integrates historical information from all variables using VST and conducts global dependency modeling through TMB. The computational complexity of MambaTS is $\mathcal{O}(\frac{KL}{P})$, where $K$ represents the number of variables, $L$ denotes the length of the lookback window, and $P$ signifies the patch stride. Table 6 outlines the computational complexity of other models.

Among them, Autoformer and FEDformer employ traditional point-wise tokenization, with computational complexity dependent on $L$. MICN and FourierGNN, focusing on modeling dependencies between variables, entail higher complexity. DLinear achieves $\mathcal{O}(L)$ complexity but lacks variable mixing. PatchTST and iTransformer have complexities of $\mathcal{O}(\left(\frac{L}{P}\right)^{2})$ and $\mathcal{O}(K^{2})$, respectively. In contrast, MambaTS’s complexity of $\mathcal{O}(K\cdot\frac{L}{P})$ strikes a balance between these approaches, as illustrated in Table 3, highlighting the meaningful and efficient equilibrium.

## 6 Conclusion

In this work, we present MambaTS, a novel multivariate time series forecasting model built upon improved selective SSMs. We first introduce VST to organize the historical information of all variables, forming a global retrospective sequence. Recognizing the dispensability of causal convolution in Mamba for LTSF, we propose the Temporal Mamba Block. Furthermore, we enhance Mamba’s selective parameters with dropout regularization to prevent overfitting and enhance model performance. To mitigate the scan order sensitivity problem, we implement variable permutation training to counter the impact of undefined variables order. Lastly, we propose VAST to dynamically discover relationships between variables during training and solve the shortest path visiting problem using an ATSP solver to determine the optimal variable scann order. Through these designs, MambaTS achieves global dependency modeling with linear complexity and establishes new state-of-the-art results across multiple datasets and prediction settings.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: I don't see why a random permutation is equivalent to a random walk. Line 324 says "K − 1 transition tuples ${(v_1, v_2),(v_2, v_3), · · ·(v_{K−1}, v_K)}$ are derived". I wonder what prevent the authors from deriving K(K-1)/2 tuples, so that each $(v_i, v_j), i<j$ is included?
soundness: 2
strengths: The paper propose a strategy to apply Mamba to to multivariate ts forecasting and achieves empirical result comparable to SOTA.
summary: This paper introduce MambaTS, a new time series forecasting model based on selective state space models. In order to tackle multivariate forecasting, the timeseries patches of each variable is unrolled in a certain order to form a single sequence. One key innovation of the paper is a method for estimating the causal relationship between variables during training via random walk without return.
weaknesses: 1. The proof in Proposition 2 does not make sense to me. I am not sure the whole concept of random walk on a casual graph with certain cost is well defined in the paper.
2. The proposed method claims to leverage the causal dependency between the variables and thus is more suitable in the multivariate setting. However, it does not seems to have a large advantage over chanel independent PatchTST, which is univariate forecasting method.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: The paper offers a well-reasoned and innovative approach to time series forecasting, with theoretically sound propositions and a practical methodology that balances computational efficiency with modeling accuracy. While the heuristic reliance on VAST and scalability issues in dense graphs present limitations, the model’s strengths in efficiency, adaptability, and architectural design make it a valuable contribution. MambaTS is especially promising for high-dimensional and complex time series data, though further work is recommended to address heuristic dependency and enhance robustness in varied causal structures. Overall, it's a solid and innovative work on time-series forecasting, effectively incorporating causality in a computationally efficient and scalable manner.
soundness: 3
strengths: - MambaTS reduces computational complexity from quadratic O(K^2) to linear O(K) by leveraging a topologically ordered linear scan, making it suitable for high-dimensional time series data.
- VAST enhances adaptability by inferring causal relationships in the absence of explicit causal graphs, using random walks to approximate dependencies and mitigate the need for exhaustive pairwise calculations.
summary: This paper introduces MambaTS, an architecture for long-term time series forecasting that models global dependencies efficiently with a linear scan, avoiding the computational challenges of self-attention. The Variable-Aware Scan along Time (VAST) mechanism dynamically infers causal relationships among variables using random walks and determines an optimal scanning order through heuristic path decoding. This design achieves scalability and adaptability, particularly for complex, high-dimensional datasets with unknown causal structures.
weaknesses: Reliance on heuristic optimization for scanning order yields sub-optimality:
- The variable-aware san along time (VAST) employs the asymmetric traveling salesman problem (ATSP) to determine the optimal scanning order, relying on heuristics like simulated annealing to address its NP-hard nature. Although heuristics provide feasible solutions, this dependency introduces inconsistency, as different approximations may affect the accuracy of variable ordering (in case of complex, dense inter-variable connections)
- Extra experiments on alternative heuristic approaches such as genetic algorithms (that are powerful in navigating NP-hard problems) could reveal a more stable and efficient approach. In the same vein, additional experiments measure how different heuristic methods affect the resulting scanning order and, subsequently, forecasting accuracy. This can help users determine if any heuristic consistently produces a favorable scanning order.

Convergence guarantee or confidence interval is not covered in causal estimation which lacks usability:
- Proposition 2 lacks formal guarantees for convergence speed, raising questions about the robustness of causality inference in finite settings. Without clear bounds on the number of walks required, the approach may yield only approximate estimates, especially when practical constraints limit the number of walks. This limitation affects the consistency and reproducibility of causal estimation results, as reliance on empirical averaging may not ensure reliable causal inference across varied dataset structures. 
- It might be helpful to introduce a stopping rule based on convergence metrics (e.g. average change in transition costs), or introduce confidence intervals on causality estimates to users to give insight into the stability of causal inferences under finite computational budgets, where both suggestions seem to be beyond the scope of this study. I hope the authors consider usability in the future works.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. How is the patch length chosen, and does it vary across datasets?
2. Could the authors clarify the random walk process and the number of epochs used in variable scanning?
3. Why were benchmarks ETTh1 and ETTm1 not included?
soundness: 1
strengths: 1. LTSF presents a compelling and complex challenge.
2. The experiments are thorough but still lack some essential details.
summary: This paper proposes MambaTS, a selective state-space model for long-term time series forecasting (LTSF) that addresses the computational limitations of Transformers by leveraging causal relationships across variables and time with a single linear scan.
weaknesses: 1. **Lack of Experimental Details:** Important implementation details are missing, such as patch length, the value of beta in Equation 7, and whether the random walk on variables is conducted K-1 times per epoch (meaning  k-1 more training time cost than one epoch).
2. **Efficiency Concerns:** Theoretical complexity analysis in Table 5 lacks practical runtime comparisons. Given that MambaTS requires K-1 iterations to estimate causal relationships, its efficiency is questionable.
3. **Incomplete Ablation Studies:** The paper introduces the TMB (with dropout replacing the original convolution), but no ablation study compares TMB and the original Mamba block, leaving its impact on performance unclear.
4. **Limited Explanation in Variable-Aware Scanning:** Section 5.2 does not clearly explain whether K-1 transitions are sufficient to estimate all variable orders, or if consistency (e.g., v1 always preceding vk) is assumed.
5. **Limited Benchmarking:** Two commonly used datasets (ETTh1, ETTm1) are missing, which reduces the generalizability of the results.
6. **Code Availability:** No code is provided, limiting reproducibility.
7. **Unpersuasive SOTA Claims:** Results in Table 2 are questionable. For example, our reimplementation of PatchTST (using official configurations) achieved better results than the reported MambaTS performance on ETTm2 (input length 720). Specifically:
   - ETTm2_720_96.log: 0.1632, 0.2555
   - ETTm2_720_192.log: 0.2167, 0.2942
   - ETTm2_720_336.log: 0.2679, 0.3282
   - ETTm2_720_720.log: 0.3521, 0.3798

   These results suggest MambaTS may not definitively outperform all baselines, especially as no code is available for direct comparison.
8. **Notation Issue:** The meaning of \( I \) in Equation 7 is unclear.
9. **Inference Process Detail:** Section 5.2 lacks details on the inference process for Variable-Aware Scan Along Time.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: See weakness.
soundness: 3
strengths: 1. The introduction of VAST and the use of causal graphs for modeling dependencies offers a unique solution to efficiently process long-term dependencies in time series data with linear complexity.
2. The model is tested across various public datasets, demonstrating superior performance compared to existing state-of-the-art models. This not only validates the efficacy of MambaTS but also showcases its versatility in handling different types of time series data.
3. MambaTS significantly reduces the computational cost traditionally associated with long-range forecasting models like Transformers by avoiding the quadratic complexity of the self-attention mechanism.
summary: The paper introduces MambaTS, an improved selective state space model for long-term time series forecasting. The model leverages a novel method for variable-aware scanning along time (VAST) to model global dependencies in a time series with variable missing rates and different intervals. By utilizing a combination of causal graphs and shortest path solutions, MambaTS addresses the limitations of previous Transformer-based models which often struggle with high computational costs and inefficient handling of long-range dependencies.
weaknesses: 1. The effectiveness of the model heavily depends on the accuracy of the causal graphs. Incorrect or incomplete causal relationships can lead to suboptimal forecasting results, which the paper does not extensively address in terms of robustness against poor graph structure
2. While the model shows high efficiency and effectiveness, the paper lacks a thorough discussion on scalability, especially in scenarios with exceedingly large datasets or highly complex variable relationships.
3. There is a need for a comparison of the model’s performance with other SOTA methods, such as Onefitsall, TimeLLM etc.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 1
questions: 1. In proposition 1, the assumption is that the causal graph exists. What if it doesn't exist? And is there any support on the random walk without return is a promising approach to estimate causal links? 
2. What is the definition of cost C and how we can get/set it empirically (e.g., the cost from node i to node j)?
3. Proposition 2 indicates that theoretically the causal relationships can be estimated without infinite random walks with return. I would like to ask how many walks are required empirically. And also the time spent? 
4. In Eq (6), how we can get the p^{(0)}? 
5. In my view, another major difference between MambaTS and iTransformer is that MambaTS model the dependencies on both time and variables, while iTransformer mainly on variables. I would like to know how this contributes to eventual performance. Because UniTST [1] is also modeling the dependencies on both time and variables dimensions, but with Transformer architecture. How does MambaTS compare with UniTST? 

Reference:

[1] UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting.
soundness: 2
strengths: 1. It is interesting to see another new work on mamba for time series forecasting. In my view, some properties of mamba are fit for time series and it's an interesting direction to explore more. 
2. The authors propose several designs to tailor mamba for time series application, which has its merits.
summary: This paper presents MambaTS, an LTSF model addressing Transformers' self-attention complexity and bias by using causal relationships for global dependency modeling. The author designs variable-aware scan along time to get variable causal relationships and also Temporal Mamba Block to avoid causal convolution. The experimental results show that MambaTS outperforms several state-of-the-art models.
weaknesses: 1. The clarity of the paper needs to be improved. In some parts, I cannot fully understand, such as the cost of a random walk without return. Also what is the cost from node i to node j, and how we can get this in the first iteration
2. Some claims have no support/evidence. For example, the authors mention that the random walk without return is a promising approach to estimate causal links. I would like to know the reason, e.g., any citations/proofs. 
3. The experiments seem not comprehensive. The authors only compare MambaTS with 7 baselines. There are a few more after iTransformer, which are worth to be compared. E.g., ModernTCN [1], UniTST [2], TSLANet [3]. 


References:

[1] ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis. 

[2] UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting. 

[3] TSLANet: Rethinking Transformers for Time Series Representation Learning

Meta-review: The paper introduces MambaTS, a new time series forecasting model based on selective state space models which leverages causal relationships to model global dependencies. Adapting Mamba-style SSMs to the Time Series Space is certainly an interesting direction that is worth investigating, and the paper proposes several interesting design choices to tailor mamba for time series. However several reviewers expressed concerns with the paper around presentation clarity (the propositions and proofs lack formal rigor) and more importantly around soundness of the experimental results. In particular, there were valid questions by reviewers (that the AC agrees with) around the lack of detailed model efficiency comparisons with MLP models, and around both the competitiveness of the model to PatchTST, and fidelity of the reported baseline metrics, especially when baseline models are not trained to convergence. Furthermore the lack of error bars and the limited number of datasets considered for evaluation makes it hard to judge the significance of the gains over SOTA baselines.

I would urge the authors to conduct a more extensive and through evaluation of the model and resubmit to a future venue.

Decision: Reject