Sequential Order-Robust Mamba for Time Series Forecasting

Authors: Seunghan Lee, Juri Hong, Kibok Lee, Taeyoung Park
Primary area: learning on time series and dynamical systems
Keywords: Mamba, State-space Model, Time Series

Abstract:
Mamba has recently emerged as a promising alternative to Transformers, offering near-linear complexity in processing sequential data.
However, while channels in time series (TS) data have no specific order in general, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this issue, we propose SOR-Mamba, a TS forecasting method that 1) incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order, and 2) eliminates the 1D-convolution originally designed to capture local information in sequential data. Furthermore, we introduce channel correlation modeling (CCM), a pretraining task aimed at preserving correlations between channels from the data space to the latent space in order to enhance the ability to capture CD.
Extensive experiments demonstrate the efficacy of the proposed method across standard and transfer learning scenarios.

Decision: reject
Reviewer scores: 5, 5, 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730192045777

thread_type:
official_review

<thread_object>
creator: reviewer_cgt6

<content>
summary:
In this manuscript, the authors claim that channels in time series (TS) data have no specific order in general, recent studies have adopted
Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this issue, the authors propose SOR-Mamba, a TS forecasting method that 1) incorporates a regularization strategy to minimize the discrepancy between
two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order.

strengths:
1. The authors propose SOR-Mamba, a TS forecasting method that handles the sequential order bias by
1) regularizing the unidirectional Mamba to minimize the distance between two embedding vectors
generated from data with reversed channel orders for robustness to channel order.
2. The authors introduce CCM, a novel pretraining task that preserves the correlation between channels from
the data space to the latent space, thereby enhancing the model’s ability to capture CD.
3. With the exception of the power-related datasets (ETTm1 & m2, etc.), the experimental performance exceeds the current SOTA.

weaknesses:
1. It is puzzling why the authors mention 'robust' but does not give experiments to resist noise or PGD attacks. Or it may be necessary to modify the terminology in the manuscript to be more precise, after all, 13 different domains of time series prediction tasks have been tried.
2. With the exception of the power-related datasets (ETTm1 & m2, etc.), the experimental performance exceeds the current SOTA. Why is the performance of the author's scheme inferior to SOTA (i.e., PatchTST) on power-related datasets (ETTm1 & m2, etc.) with more significant global regularity?
3. Despite the computational complexity comparison, the reviewer is concerned about the actual training and inference time.
4. The description in Figure 1 seems to be the difference between different exogenous time series due to sampling frequency and periodic change rule, and the description of sequence order is prone to misunderstanding (i.e., covariables have orders).

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730364640668

thread_type:
official_review

<thread_object>
creator: reviewer_vpya

<content>
summary:
This paper introduce a new Mamba-based model for long term time series forecasting named SOR-Mamba which utilizes modified Mamba structure to capture channel dependencies. Furthermore, the authors also introduces channel correlation modeling as a pretraining task aimed at enhancing the ability to capture channel dependencies.

strengths:
1. The proposed SOR-Mamba uses a modified Mamba structure (also with a regularization strategy) to eliminate the sequential order bias of Mamba which is more useful in capture channel dependencies.
2. The authors also introduce a pretraining task named channel correlation modeling (CCM) to preserve correlations between channels from the data space to the latent space in order to enhance the ability to capture CD.

weaknesses:
1. Since Mamba is a sequential-based model, and the channels lake sequential order, why we need Mamba to extract channel dependencies? It is important to clarify the importance of capturing channel dependencies with Mamba.
2. The experimental comparisons appear unfair, as models like PatchTST exhibit better performance with longer lookback window. I strongly suggest the authors conduct additional experiments, such as setting $L=336$.

questions:
See Weaknesses above.
3. In line 369-370, I’m curious about how to calculate the global correlation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730465023718

thread_type:
official_review

<thread_object>
creator: reviewer_9yaz

<content>
summary:
This manuscript proposes an algorithm to regularize bidirectional Mamba channel vectors in order to solve the problem that there is no intrinsic order relationship between channels in time series data. The authors also design a new pre-training method (CCM) to help the model better establish the association between channels of multi-dimensional time series. Extensive experiments and ablation experiments are conducted, and the experimental results show the effectiveness of the relevant contributions.

strengths:
1. This study achieves a good improvement in results with only a few changes to the method. The proposed method is simple, novel and effective.
2. The experiments carried out by the authors are very detailed and the relevant parameters are well discussed. This is an experimentally rigorous and solid study.
3. The manuscript is well-written and the figures are clear and intuitive.

weaknesses:
1. The authors lack a complete description of the motivation for regularization. Why regularizing the bidirectional mamba channel vectors and minimizing the distance between them can help the model better capture the correlation between channels? This requires the author to give a more intuitive and reasonable motivation.
2. Although the authors have done a lot of experiments, they lack experiments on longer input lengths. Some widely used baselines in this field, such as PatchTST (arXiv preprint arXiv:2211.14730, 2022.), all use longer input sequences.
3. The authors use too many abbreviations in their manuscripts and experimental result tables, and lack clear and complete explanations of the abbreviations. This makes the relevant content very difficult to read and easily confusing.

questions:
1. Table I.1 seems to show that the hyperparameter lambda does not affect the results. Is this a reasonable phenomenon for the key contribution?
2. What are the principles and motivations for using parameter-sharing CD-Mamba block and removing 1D-conv? Can the authors provide additional details?
3. The description of the pre-training method used in this study is somewhat vague. Can the authors supplement the algorithm flow or provide detailed explanation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730619147255

thread_type:
official_review

<thread_object>
creator: reviewer_efur

<content>
summary:
This paper addresses a limitation in applying Mamba to time series forecasting: the sequential order bias when modelling channel dependencies, as channels typically lack inherent order. The authors propose SOR-Mamba, which introduces two main innovations: a regularization strategy that minimizes discrepancy between embedding vectors from reversed channel orders, and the removal of the unnecessary 1D-convolution layer. They also introduce Channel Correlation Modelling (CCM), a pretraining task that preserves channel correlations from data space to latent space. The method reported promising performance across different scenarios, including cases with missing data and varying channel orders, while maintaining computational efficiency in terms of memory and runtime.

strengths:
The proposed method addresses the sequential order bias in channel dependencies (CD). This bias can degrade performance when channels lack inherent order in multi-channel time series data. The SOR-Mamba method mitigates this bias by introducing a regularization technique that minimizes the discrepancy between embeddings generated from reversed channel orders, thus improving robustness. 

Additionally, the paper proposes Channel Correlation Modeling (CCM) as a pretraining task, which preserves channel correlations from data to latent space, enhancing CD capture. The architecture modifications, including removing the 1D-convolution and focusing on unidirectional Mamba, can reduce computational overhead, achieving efficiency gains in memory and parameter usage compared to other models like S-Mamba. This should be of interests to the group of readers working on multi channel time series data forecasting.

weaknesses:
The paper’s approach is relatively complex, involving several architectural changes (e.g., removing 1D-convolutions, applying specific regularization, and adding CCM pretraining) that may complicate replication and limit accessibility. 

While it emphasizes efficiency improvements, the paper could provide more detailed explanations regarding the trade-offs involved in removing the 1D-convolution, especially on datasets where channel order may have some inherent structure (e.g., traffic data). The study also lacks a comprehensive discussion of potential limitations in real-world deployment, such as sensitivity to hyperparameters (e.g., λ in regularization) and its impact across different datasets.

questions:
1. How does the choice of the distance metric d in the regularization term impact the model's performance across datasets with varying levels of channel correlation? - is the regularization term’s effectiveness dependent on the number of channels or dataset-specific factors etc.?

2. The paper uses CCM to preserve channel correlations in the latent space during pretraining. How does CCM compare to masked modelling and reconstruction pretraining across different dataset sizes and channel configurations? - and is there any limitation of CCM if there's only exist weak channel correlations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645734624

thread_type:
official_review

<thread_object>
creator: reviewer_wewk

<content>
summary:
This paper introduces SOR-Mamba, a time series forecasting method that addresses the sequential order bias  while capture channel dependencies in Mamba, through a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, and 1D-convolution removal, The authors also propose Channel Correlation Modeling (CCM) as a pretraining task.

strengths:
1. Clear problem identification regarding Mamba's limitations in handling unordered channels
2. Comprehensive experiments
3. Improved efficiency compared to other approaches

weaknesses:
**Limited Technical Novelty**:
- The regularization strategy is overly simplistic that minimizing the distance between embeddings from different channel orders
- The removal of 1D-conv lacks theoretical justification and appears to be an ad-hoc solution, As shown in Figure 5 of the paper, its removal may negatively impact datasets with ordered channels such as PEMS datasets

questions:
1. In the Manba architecture, are there other advances in order-invariant architectures that can be compared?
2. The definition of sequential order bias in Figure 4 and how it is calculated?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687571763

thread_type:
official_review

<thread_object>
creator: reviewer_pjuw

<content>
summary:
This paper proposes SOR-Mamba to solve the sequential order bias introduced by mamba capturing channel dependencies. Model SOR-Mamba incorporates a regularization strategy to minimize the distance between embeddings from data and reversed data and eliminates the 1D-convolution in the original Mamba block. And a pretraining task channel correlation modeling is introduced to preserve the channel correlation from the data space to the latent space. The effectiveness of the proposed method is demonstrated by the extensive experiment results.

strengths:
* This paper uses a single unidirectional Mamba with regularization and non-1D convolutions to capture channel correlations. By modifying the traditional Mamba module, it can better address the issue of sequential order bias. The research problem is clearly defined, and the solution is reasonable. And the pretraining task channel correlation modeling also enhances the model's generalization and performance.

* The details of the model are mostly clear, with each module explained and supported by equations. The experiments  include time series forecasting, transfer learning and many ablation studies to validate the effectiveness of each module.

weaknesses:
* The proposed method is mostly constructed on existing models -- reverse modeling has been widely used in literature and removing the 1d-conv is trival.
* The experimental results show that the model's performance improvement across various datasets is not significant, mostly in the thousandth.
* The removal of 1D-conv negatively impacts PEMS dataset in tale 7 and figure 5. The necessity to remove 1D-conv is uncertain.
* The model uses the Mamba backbone, but the baseline only includes a single Mamba model: S-Mamba. Providing more models from the Mamba framework for comparison would be better.

questions:
* When randomly shuffling the channel order, how is it done? Is a fixed shuffling method used for all sequences in a dataset, or is it shuffled randomly each time?
* In the last row of Table 1, is the calculation formula written incorrectly?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732366874554

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
### **7. Minor issues**
- **Calculation of global correlation (vpYA)**:  The global correlation refers to the correlation between channels across the entire training TS dataset, excluding the validation and test datasets, with details on the train-validation-test split protocol provided in **Appendix A**.

- **Simplicity of the proposed method: (eFUR, WEwK)**: The removal of the 1D-conv is straightforward, requiring no changes to the existing architecture (**Appendix D**), while the regularization uses existing embeddings, maintaining simplicity and minimizing computational overhead (**Appendix C**). Similarly, the CCM pretraining task only computes the correlation between output vectors, with no additional modules (**Appendix J**).

- **Training and inference time (Cgt6)**: We have already analyzed efficiency in terms of (1) parameters, (2) complexity and memory, and (3) computational time (training and inference) using the Traffic dataset ($C=862$), as shown in **Table 13**.

- **Lack of details regarding CCM (eFUR)**: We have included pseudocode for CCM in **Appendix J** and provided additional guidance for readers in **L244**.

&nbsp;

In what follows, we summarize additional changes in the manuscript we provided during the rebuttal period:
- **(Figure L.1; 9Yaz, vPYA)** Applying our method with various lookback windows $L$
- **(Algorithm 2; 9Yaz)** Pseudocode for CCM

&nbsp;

For your convenience, we have uploaded a PDF file that includes both the revised manuscript and appendices, with the major changes highlighted **in green**. 

We believe that the discussions and empirical results presented are valuable and will further enhance our contribution. If we have overlooked anything or if you have additional questions or suggestions, please feel free to share them with us; we would be happy to address them.

&nbsp;

Thank you very much.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732367022385

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# General Comment

&nbsp;

Dear AC and reviewers,

First of all, we deeply appreciate your time and effort in reviewing our paper. Our work introduces **SOR-Mamba**, a TS forecasting method to address the sequential order bias when capturing channel dependencies (CD) with Mamba. SOR-Mamba utilizes two strategies: **(1) regularization strategy** to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, and **(2) removal of 1D-convolution** originally designed to capture local information in sequential data.


As highlighted by the reviewers, our work effectively addresses the sequential order bias in Mamba (PJuw, eFUR, Cgt6) using novel regularization and the removal of 1D convolutions (PJuw, WEwK, vPYA). The introduction of Channel Correlation Modeling (CCM) improves the ability to capture channel dependencies (WEwK, eFUR, vPYA, Cgt6). Extensive experiments demonstrate the model's effectiveness and efficiency (PJuw, WEwK, eFUR, 9Yaz).


&nbsp;

In our responses, we addressed the concerns raised by all reviewers and supplemented our claims with additional analyses. Here we provide highlights to aid your post-rebuttal discussion (all sections/figures/tables labeled based on the revision):

&nbsp;

### **1. Motivation of Mamba for CD (vPYA)**

As noted in **L39–41**, our focus is on Mamba’s role in capturing CD, consistent with recent works advocating ***complex attention mechanisms for channel dependency (CD)*** and *simple MLPs for temporal dependency (TD)*. Similarly, **Table 11** confirms that complex architectures like attention or Mamba are better suited for CD than TD.

&nbsp;

### **2. Motivation of the proposed method**

 The reviewers expressed concerns about the motivation and necessity of the proposed methods.

- **Regularization method (9Yaz)**: The regularization was designed to address the sequential order bias, as previous methods using bidirectional Mamba still showed sensitivity to channel order (**Table 1**, **Figure 7**). Our approach provides a more consistent representation, as shown in **Table 10**, **Appendix H.1**, and **Figure 7**.

- **Removal of 1D-conv (9Yaz, PJuw, WEwK)**: The 1D-conv in Mamba was intended for sequential data (**L66--68**) but is not suitable for non-sequential channel data in TS. Previous works overlooked this, which motivated us to remove it, with details in **Appendix D**.

&nbsp;

### **3. Effect of performance by removal of 1D-convolution (PJuw, eFUR, WEwK)**

The 1D-conv is removed for **general** TS datasets where channels lack inherent sequential order. As shown in **Table 7** and **Figure 5**, removing it does not always improve performance in datasets with ordered channels, highlighting that the 1D-conv captures sequential relationships. Removing it may not be beneficial when channel order exists, as discussed in **L362--366**.

&nbsp;

### **4. Comparison with other Mamba-based models (PJuw,WEwK)**
As noted in **Section 2**, several studies have used Mamba to capture CDs. However, these works are concurrent to ours (e.g, TimeMachine (ECAI-2024), S-Mamba, CMamba, FMamba, SST, Bi-Mamba+, MambaTS (Arxiv 2024)) and, aside from our baseline, most have not released their code, making direct comparisons infeasible.

&nbsp;

### **5. Various lookback windows $L$ (9Yaz, vPYA)**
We followed the experimental protocols of previous works ($L=96$) for fair comparison, instead of utilizing different values of $L$. Nonetheless, we conducted an additional ablation study on $L$ across four datasets in **Appendix L** during the rebuttal. The results demonstrate that performance is robust to $L$ for some datasets and improves with a larger $L$ for others.

&nbsp;

### **6. Robustness to hyperparameters (eFUR)**

We have ***already*** discussed the robustness of our proposed method to hyperparameters across various datasets, as follows:

- Hyperparameter λ: **Appendix I.1**

- Distance metric (for regularization): **Appendix K.1** 

- Distance metric (for CCM): **Appendix K.2**
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies an issue when using the Mamba (instead of Transformer) model for time series forecasting. In particular, a sequential order bias was introduced in recent studies when using Mamba to capture channel dependencies in time series. A Mamba variant is proposed in this paper to make the model less dependent on the channel order by incorporating two techniques, namely, regularization strategy and removal of 1D-convolution.

Major strengths:
- The two techniques proposed to make Mamba less dependent on the channel order makes sense although they are quite simple.
- The paper is generally easy to read for people who are familiar with the related areas.

Major weaknesses:
- The performance gain is insignificant.
- Removal of 1D-convolution may hurt if the dataset has ordered channels. Whether or not a dataset has ordered channels may not be apparent in some real-world applications.
- It is not clear whether the proposed techniques also work for other Mamba variants.

The two proposed techniques are simple, but we do not think it is right to reject a paper simply because of the simplicity of the proposed method. A simple idea could lead to paper acceptance if its effectiveness is thoroughly justified beyond a reasonable doubt. However, from the experiments, we are not convinced that the proposed method can lead to significant performance gain. A more comprehensive study by incorporating the suggestions of some reviewers would make this paper more ready for publication.