Title: Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multivariate Time Series

Time Series Modeling

Time Series Forecasting

Time Series Classification
Abstract: Modeling multivariate time series is a well-established problem with a wide range of applications from healthcare to financial markets. It, however, is challenging as it requires methods to (1) have high expressive power of representing complicated dependencies along the time axis to capture both long-term progression and seasonal patterns, (2) capture the inter-variate dependencies when it is informative, (3) dynamically model the dependencies of variate and time dimensions, and (4) have efficient training and inference for very long sequences. Traditional State Space Models (SSMs) are classical approaches for univariate time series modeling due to their simplicity and expressive power to represent linear dependencies. They, however, have fundamentally limited expressive power to capture non-linear dependencies, are slow in practice, and fail to model the inter-variate information flow. Despite recent attempts to improve the expressive power of SSMs by using deep structured SSMs, the existing methods are either limited to univariate time series, fail to model complex patterns (e.g., seasonal patterns), fail to dynamically model the dependencies of variate and time dimensions, and/or are input-independent.  We present Chimera, an expressive variation of the 2-dimensional SSMs with careful design of parameters to maintain high expressive power while keeping the training complexity linear. Using two SSM heads with different discretization processes and input-dependent parameters, Chimera is provably able to learn long-term progression, seasonal patterns, and desirable dynamic autoregressive processes. To improve the efficiency of complex 2D recurrence, we present a fast training using a new 2-dimensional parallel selective scan. Our experimental evaluation shows the superior performance of Chimera on extensive and diverse benchmarks, including ECG and speech time series classification, long-term and short-term time series forecasting, and time series anomaly detection.

Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models

Abstract

Modeling multivariate time series is a well-established problem with a wide range of applications from healthcare to financial markets. It, however, is challenging as it requires methods to (1) have high expressive power of representing complicated dependencies along the time axis to capture both long-term progression and seasonal patterns, (2) capture the inter-variate dependencies when it is informative, (3) dynamically model the dependencies of variate and time dimensions, and (4) have efficient training and inference for very long sequences. Traditional State Space Models (SSMs) are classical approaches for univariate time series modeling due to their simplicity and expressive power to represent linear dependencies. They, however, have fundamentally limited expressive power to capture non-linear dependencies, are slow in practice, and fail to model the inter-variate information flow. Despite recent attempts to improve the expressive power of SSMs by using deep structured SSMs, the existing methods are either limited to univariate time series, fail to model complex patterns (e.g., seasonal patterns), fail to dynamically model the dependencies of variate and time dimensions, and/or are input-independent. We present Chimera, an expressive variation of the 2-dimensional SSMs with careful design of parameters to maintain high expressive power while keeping the training complexity linear. Using two SSM heads with different discretization processes and input-dependent parameters, Chimera is provably able to learn long-term progression, seasonal patterns, and desirable dynamic autoregressive processes. To improve the efficiency of complex 2D recurrence, we present a fast training using a new 2-dimensional parallel selective scan. Our experimental evaluation shows the superior performance of Chimera on extensive and diverse benchmarks, including ECG and speech time series classification, long-term and short-term time series forecasting, and time series anomaly detection.

1 Introduction

Modeling time series is a well-established problem with a wide range of applications from healthcare [1-3] to financial markets [4, 5] and energy management [6]. The complex nature of time series data, its diverse domains of applicability, and its broad range of tasks (e.g., classification [2, 7], imputation [8, 9], anomaly detection [2, 10], and forecasting [6]), however, raise fundamental challenges to design effective and generalizable models: (1) The higher-order, seasonal, and long-term patterns in time series require an effective model to be able to expressively capture complex and autoregressive dependencies; (2) In the presence of multiple variates of time series, an effective model need to capture the complex dynamics of the dependencies between time and variate axes. More specifically, most existing multivariate models seem to suffer from overfitting especially when the target time series is not correlated with other covariates [11]. Accordingly, an effective model needs to adaptively learn to select (resp. filter) informative (resp. irrelevant) variates; (3) The diverse set of domains and tasks requires effective models to be free of manual pre-processing and domain

Input

ExpressiveRepresentation

(@1

0

0

01

SeasonalPatterns

Avariate =

0

Q2

0

0

Learnable discretization

via ZOH

0

0

0

△slx = fs(x)

→

0

0

0

Qd

Output

r0

0

0

a1

0

0

3

Long-term Progression

ATime

0

1

0

Learnable discretization

via ZOH

0

adl

plx = fp(x)

→

BTime, CTime , Bvariate, Cvariate are time dependent.

Chimera

Efficient TrainingUsing 2DScan

Parallel Scans

↑

田

Re-

discretization

AVariate

ATime

↑

2D SSM

2D SSM

Long-term

Seasonal

Formulate the 2D selective recurrence

as 2D prefix sum problem.

Input↑

AT→V

Av→T

knowledge and instead adaptively learn them; and (4) Due to the processing of very long sequences, effective methods need efficient training and inference.

Classical methods (e.g., State Space Models [12, 13], ARIMA [14], SARIMA [15], Exponential Smoothing (ETS) [16]) require manual data preprocessing and model selection, and often are not able to capture complex non-linear dynamics. The raise of deep learning methods and more specifically Transformers [17] has led to significant research efforts to address the limitation of classical methods and develop effective deep models [18-27]. Unfortunately, most existing deep models struggle to achieve all the above four criteria. The main body of research in this direction has focused on designing attention modules that use the special traits of time series [21, 20]. However, the inherent permutation equivariance of attentions contradicts the causal nature of time series and often results in suboptimal performance compared to simple linear methods [11]. Moreover, they often either overlook difference of seasonal and long-term trend or use non-learnable methods to handle them [20].

A considerable subset of deep models overlook the importance of modeling the dependencies of variates [11, 28, 29]. These dependencies, however, are not always useful; specifically when the target time series is not correlated with other covariates [30]. Despite several studies exploring the importance of learning cross variate dependencies [26, 27, 30], there has been no universal standard and the conclusion has been different depending on the domain and benchmarks. Accordingly, we argue that an effective model need to adaptively learn to capture the dependencies of variates in a data-dependent manner. In this direction, recently, Liu et al. [27] argue that attention mechanisms are more effective when they are used across variates, showing the importance of modeling complex non-linear dependencies across the variate axis in a data-dependent manner. However, the quadratic complexity of Transformers challenges the model on multivariate time series with a large number of variates (e.g., brain activity signals [2] or traffic forecasting [6]), limiting the efficient training and inference (see Table 3, and Table 5).

The objective of this study is to develop a provably expressive model for multivariate time series that not only can model the dynamics of the depenendencies along both time and variates, but it also takes advantage of fast training and inference. To this end, we present a Chimera, a three-headed two-dimensional State Space Model (SSM) that is based on linear layers along (i) time, (ii) variates, (iii) time → variate, and (iv) variate → time. Chimera has a careful parameterization based on the pair of companion and diagonal matrices (see Figure 1), which is provably expressive to recover both classical methods [16, 14, 15], linear attentions, and recent SSM-based models [31, 32]. It further uses an adaptive module based on a 2D SSM with an especially designed discretization process to

capture seasonal patterns. While our theoretical results and design of Chimera guarantee the first three criteria of an effective model, due to its 2D recurrence, the naive implementation of Chimera results in slow training. To address this issue, we reformulate its 2D recurrence as the prefix sum problem with a 2-dimensional associative operators. This new formulation can be done in parallel and has hardware-friendly implementation, resulting in much faster training and inference.

In our experimental evaluation, we explore the performance of Chimera in a wide range of tasks: ECG and audio speech time series classification, long- and short-term time series forecasting, and anomaly detection tasks. We find that Chimera achieve superior or on par performance with state-of-the-art methods, while having faster training and less memory consumption. We perform a case study on the human brain activity signals [2] to show (1) the effectiveness of Chimera and (2) evaluate the importance of modeling the dynamics of the variates dependencies.

2 Preliminaries

Notations. In this paper we mainly focus on classification and forecasting tasks. Note that anomaly detection can be seen as a binary classification task, where 0 means 'normall' and 1 means 'anomaly'. We let X = { x 1 , . . . , x N } ∈ R N × T be the input sequences, where N is the number of variates and T is the time steps. We use x v,t to refer to the value of the series v at time t . In classification (anomaly detection) tasks, we aim to classify input sequences and for forecasting tasks, given an input sequence x i , we aim to predict ˜ x i ∈ R 1 × H , i.e., the next H time steps for variate x i , where H is called horizon. In 2D SSMs formulation, for a 2-dimensional vector x ∈ C 1 , we use x (1) and x (2) to refer to its real and imaginary components, respectively.

Multi-Dimensional State Space Models. We build our approach on the continuous State Space Model (SSM) but later we make each component of Chimera discrete by a designed discretization process. For additional discussion on 1D SSMs see Appendix A. Given parameters A τ 1 ∈ R N ( τ 1 ) × N ( τ 1 ) , B τ 2 ∈ C N ( τ 2 ) × 1 , and C ∈ C N 1 × N 2 for τ 1 ∈ { 1 , ..., 4 } and τ 2 ∈ { 1 , 2 } , the general form of the time-invariant 2D SSM is the map x ∈ C 1 ↦→ y ∈ C 1 defined by the linear Partial Differential Equation (PDE) with initial condition h (0 , 0) = 0 :

Contrary to the multi-dimensional SSMs discussed by Gu and Dao [33], Gu et al. [34], in which multi-dimension refers to the dimension of the input but with one time variable, the above formulation uses two variables, meaning that the mapping is from a 2D grid to a 2D grid.

(Seasonal) Autoregressive Process. Autoregressive process is a basic yet essential premise for time series modeling, which models the causal nature of time series. Given p ∈ N , x k ∈ R d , the simple linear autoregressive relationships between x k and its past samples x k -1 , x k -2 , . . . , x k -p can be modeled as x k = ϕ 1 x k -1 + ϕ 2 x k -2 + . . . , ϕ p x k -p , where ϕ 1 , . . . , ϕ p are coefficients. This is called AR ( p ) . Similarly, in the presence of seasonal patterns, the seasonal autoregressive process, SAR ( p, q, s ) , is:

where s is the frequency of seasonality, and ϕ 1 , . . . , ϕ p and η 1 , . . . , η q are coefficients. Note that one can simply extend the above formulation to multivariate time series by letting coefficients to be vectors and replace the product with element-wise product.

3 Chimera: A Three-headed 2-Dimensional State Space Model

In this section, we first present a mathematical model for multivariate time series data and then based on this model, we present a neural architecture that can satisfy all the criteria discussed in §1.

3.1 Motivations & Chimera Model

SSMs have been long-standing methods for modeling time series [12, 13], mainly due to their simplicity and expressive power to represent complicated and autoregressive dependencies. Their states, however, are the function of a single-variable (e.g., time). Multivariate time series, on the other hand, require capturing dependencies along both time and variate dimensions, requiring the current state of the model to be the function of both time and variate. Classical 2D SSMs [35-38], however, struggle to achieve good performance compared to recent advanced deep learning methods as they are : (1) only able to capture linear dependencies, (2) discrete by design, having a pre-determined resolution, and so cannot simply model seasonal patterns, (3) slow in practice for large datasets, (4) their update parameters are static and cannot capture the dynamics of dependencies. Deep learning-based methods [6, 30, 27], on the other hand, potentially are able to address a subset of the above limitations, while having their own drawbacks (discussed in §1). In this section, we start with continuous SSMs due to their connection to both classical methods [12, 13] and recent breakthrough in deep learning [34, 33]. We then discuss our contributions on how to take the advantages of the best of both worlds, addressing all the abovementioned limitations.

Discrete 2D SSM. We use 2-dimensional SSMs, introduced in Equation 1-3, to model multivariate time series, where the first axis corresponds to the time dimension and the second axis is the variates. Accordingly, each state is a function of both time and variates. The first stage is to transform the continuous form of 2D SSMs to discrete form. Given the step size ∆ 1 and ∆ 2 , which represent the resolution of the input along the axes, discrete form of the input is defined as x k,ℓ = x ( k ∆ 1 , ℓ ∆ 2 ) . Using Zero-Order Hold (ZOH) method, we can discretize the input as (see Appendix C for details):

( A -1 3 ( ¯ A 3 -I ) B (1) 2 A -1 4 ( ¯ A 4 -I ) B (2) 2 ) . Note that this formulation can also be viewed as the modification of the discrete Roesser's SSM model [35] when we add a lag of 1 in the inputs ( ¯ x i,j ¯ x i,j ) . This modification, however, misses the discretization step, which is an important step in our model. We later use the discretization step to (1) empower the model to select (resp. filter) relevant (resp. irrelevant) information, (2) adaptively adjust the resolution of the method, capturing seasonal patterns.

From now on, we use t (resp. v ) to refer to the index along the time (resp. variate) dimension. Therefore, for the sake of simplicity, we reformulate Equation 5 as follows:

where ¯ A 1 , ¯ A 2 , ¯ A 3 , ¯ A 4 ∈ R N × N , ¯ B 1 , ¯ B 2 ∈ R N × 1 , and C 1 , C 2 ∈ R 1 × N are parameters of the model, h (1) v,t , h (2) v,t ∈ R N × d are hidden states, and x v,t ∈ R 1 × d is the input. In this formulation, intuitively, h (1) v,t is the hidden state that carries cross-time information (each state depends on its previous time stamp but within the same variate), where ¯ A 1 and ¯ A 2 control the emphasis on past cross-time and cross-variate information, respectively. Similarly, h (2) v,t is the hidden state that carries cross-variate information (each state depends on other variates but with the same time stamp). Later in this section, we discuss to modify the model to bi-directional setting along the variate dimension, to enhance information flow along this non-causal dimension.

Interpretation of Discretization. Time series data are often sampled from an underlying continuous process [39, 40]. In these cases, variable ∆ 1 in the discretization of the time axis can be interpreted as resolution or the sampling rate from the underlying continuous data. However, discretization along

2D SSM For Time Series

Multivariate Closed-Loop

↑ yu,t

h(1)

v,t+1

C1

D1→→Φ↑D2

C2

↑+zan

uv+1,t

Convolution Form

A1

A2

B

A3

A4

B2

(Ck@ +Cak@)

yu,t =

o,t

xo,t

h

(2)

u,t

uv,t

(1)

b(2)

v,t

Uu,t

Data-DependentParameters

p*q=

P1

p2

p3

米

Q1

Q2

q3

q1 Op1q2Op2q1p3+q2p6+q3

B = LinearB;(x),

P4

P5

p6

q4

95

Q6

q4Op4q5p5q4p3+q5p6+Q6

= Softplus (Linear△(x))

(p*q)*r=p*(q*r)

Parallel Scan

the variate axis, which is discrete by its nature, or when working directly with discrete data [41] is an unintuitive process, and raise questions about its significance. The discretization step in 1D SSMs has deep connections to gating mechanisms of RNNs [42, 33], automatically ensures that the model is normalized [43], and results in desirable properties such as resolution invariance [32].

Proposition 3.1. The 2D discrete SSM introduced in Equation 6-8 with parameters ( { ¯ A i } , { ¯ B i } , { ¯ C i } , k ∆ 1 , ℓ ∆ 2 ) evolves at a rate k (resp. ℓ ) times as fast as the 2D discrete SSM with parameters ( { ¯ A i } , { ¯ B i } , { ¯ C i } , ∆ 1 , ℓ ∆ 2 ) (resp. ( { ¯ A i } , { ¯ B i } , { ¯ C i } , k ∆ 1 , ∆ 2 ) ).

Accordingly, parameters ∆ 1 can be viewed as the controller of the length of dependencies that the model captures. That is, based on the above result, we see the discretization along the time axis as the setting of the resolution or sampling rate: while small ∆ 1 can capture long-term progression, larger ∆ 1 captures seasonal patterns. For now, we see the discretization along the variate axis as a mechanism similar to gating in RNNs [44, 33], where ∆ 2 controls the length of the model context. Larger values of ∆ 2 means less context window, ignoring other variates, while smaller values of ∆ 2 means more emphesis on the dependencies of variates. Later, inspired by Gu and Dao [33], we discuss making ∆ 2 as the function of the input, resulting in a selection mechanism that filters irrelevant variates.

Structure of Transition Matrices. For Chimera to be expressive and able to recover autoregressive process, hidden states h (1) v,t should carry information about past time stamps. While making all the parameters in A i learnable allows the model to learn any arbitrary structure for A i , previous studies show that this is not possible unless the structure of transition matrices are restricted [45, 46]. To this end, inspired by Zhang et al. [28] that argue that companion matrices are effective to capture the dependencies along the time dimension, we restrict A 1 and A 2 matrices to have companion structure (see Appendix D for the details). Not only this formulation is shown to be effective for capturing dependencies along the time dimension [28] (also see Theorem 3.4), but it also can help us to compute the power of A 1 and A 2 faster in the convolutional form, as discussed by Zhang et al. [28]. Also, for A 3 and A 4 , we observe that even a simpler structure of diagonal matrices is effective to fuse information along the variate dimension. Not only these simple structured matrices make the training of the model faster, but they also are proven to be effective [45].

Bi-Directionality. The causal nature of the 2D SSM result in limited information flow along the variate dimension as variate are not ordered. To overcome this challenge, inspired by the bi-directional 1D SSMs [47, 48, 1], we use two different modules for forward and backward pass along the variate dimension. The details of formulation is in Appendix E.

Convolution Form. As discussed by Baron et al. [49], similar to 1D SSMs [34], the data-independent formulation of 2D-SSMs can be viewed as a convolution with a kernel K . This formulation not only results in faster training by providing the ability of parallel processing, but it also connect Chimera

with very recent studies of modern convolution-based architecture for time series [9]. Applying the recurrent rules in Equation 6-8, we can write the output as:

where kernels K ( τ ) ˆ v, ˆ t = ∑ ( z 1 ,...,z 5 ) ∈ P ( τ ) q i ¯ A p 1 1 ¯ A p 2 2 ¯ A p 3 3 ¯ A p 4 4 ¯ B p 5 , and P ( τ ) is the partitioning of the paths from the starting point to (ˆ v, ˆ t ) for τ ∈ { 1 , 2 } . As discussed by Baron et al. [49], if the power of ¯ A i s are given and cached, calculating the partitioning of all paths can be done very efficiently (near-linearly) as it the generalization of pascal triangle. To calculate the power of ¯ A i , note that we use diagonal matrices as the structure of ¯ A 3 , and ¯ A 4 , and so computing their powers is very fast. On the other hand, for ¯ A 1 and ¯ A 2 with companion structures, we can use sparse matrix multiplication, which results in linear complexity in terms of the sequence length.

Data-Dependent Parameters. As discussed earlier, parameters ¯ A 1 and ¯ A 2 controls the emphasis on past cross-time and cross-variate information. Similarly, parameters ∆ 1 and ¯ B 1 controls the emphasis on the current input and historical data. Since these parameters are data-independent, one can interpret them as a global feature of the system. In complex systems (e.g., human neural activity), however, the emphasis depends on the current input, requiring these parameters to be the function of the input (see §4.1). The input-dependency of parameters allows the model to select relevant and filter irrelevant information for each input data, providing a similar mechanism as transformers [33]. Additionally, as we argue earlier, depending on the data, the model needs to adaptively learn if mixing information along the variates is useful. Making parameters input-dependent further overcomes this challenge and lets our model to mix relevant and filter irrelevant variates for the modeling of a variate of interest. One of our main technical contributions is to let ¯ B i , ¯ C i , and ∆ i for i ∈ { 1 , 2 } be the function of the input x v,t . This input-dependent 2D SSM, unfortunately, does not have the convolution form, limiting the scalability and efficiency of the training. We overcome this challenge by computing the model recurrently with a new 2D scan.

2D Selective Scan. Inspired by the scanning in 1D SSMs [50, 33], we present an algorithm to decrease the sequential steps that are required to calculate hidden states. Given p, q , each of which with 6 elements, we first define operation ⋇ as: ( ⊙ is matrix-matrix and ⊗ is matrix-vector multiplication)

The proofs of the next two theorems are in Appendix G.

Theorem 3.2. Operator ⋇ is associative: Given p, q, and r , we have: ( p ⋇ q ) ⋇ r = p ⋇ ( q ⋇ r ) . Theorem 3.3. 2D SSM recurrence can be done in parallel using parallel prefix sum algorithms with associative operator ⋇ , when fixing the variate.

3.2 New Variant of 2D SSM: 2D Mamba

Figure 2 (Top-Left) shows the recurrence form of our 2D SSM. Each small square is a state of the system, i.e., the state of a variate at a certain time stamp. 2D SSM considers two hidden states for each state (represented by two colors: light red and blue), encoding the information along the time (red) and variate (blue), respectively. Furthermore, each arrow represents a transition matrix A i that decides to how information need to be fused. In this section, we discuss a spacial instance of our 2D SSM by limiting its parameters.

2D Mamba. We let A 2 = A 3 = 0 in our 2D SSM. The resulting model is equivalent to:

[ 0 A -1 4 ( ¯ A 4 -I ) B (2) 2 ] . This formulation with data-dependent parameters, is equivalent to using two

S6 blocks [33] each of which along a dimension. Notably, these two S6 blocks are not separate as the output y v,t is based on both hidden states h (1) v,t and h (2) v,t , capturing 2D inductive bias.

3.3 Chimera Neural Architecture

In this section, we use a stack of our 2D SSMs, with non-linearity in between, to enhance the expressive power and capabilities of the abovementioned 2D SSM. To this end, similar to deep SSM models [28], we allow all parameters to be learnable and in each layer we use multiple 2D SSMs, each of which with its own responsibility. Also, in the data-dependent variant of Chimera, we let parameters B i , C i , and ∆ i for i ∈ { 1 , 2 } be the function of the input x :

Chimera follows the commonly used decomposition of time series, and decomposes them into trend components and seasonal patterns. it, however, uses special traits of 2D SSM to capture these terms.

Seasonal Patterns. To capture the multi-resolution seasonal patterns, we take advantage of the discretization process. Proposition 3.1 states that if x ( v, t ) ↦→ y ( v, t ) with parameters ( { ¯ A i } , { ¯ B i } , { ¯ C i } , ∆ 1 , ∆ 2 ) then x ( v, kt ) ↦→ y ( v, kt ) with ( { ¯ A i } , { ¯ B i } , { ¯ C i } , k ∆ 1 , ∆ 2 ) . Accordingly, we use 2D-SSM ( . ) module with a separate learnable ∆ s that is responsible to learn the best resolution to capture seasonal patterns. Another interpretation for this module is based on SAR ( p, q, s ) (Equation 4). In this case, ∆ s aims to learn a proper parameter s to capture seasonal patterns. Since we expect the resolution before and after this module matches, we add additional re-discretization module (a simple linear layer), after this module.

Trend Components. The second module of Chimera, 2D-SSM t ( . ) simply uses a sequence of multiple 2D SSMs to learn trend components. Proper combination of the outputs of this and the previous modules can capture both seasonal and trend components.

Both Modules Together. We followed previous studies [51] and consider residual connection modeling for learning trend and seasonal patterns. Given input data ˜ X 0 = X , and ℓ = 0 , . . . , L , we have:

Figure 1 illustrate the architecture of Chimera. Due to the ability of our 2D SSM to recover smoothing techniques (see Theorem 3.4), this combination of modules for trend and seasonal patterns can be viewed as a generalization of traditional methods that use moving average with residual connection to model seasonality [51].

Gating with Linear Mapping. Inspired by the success of gated recurrent and SSM-based models [52, 33], we use a head of a fully connected layer with Swish [53], resulting in SwiGLU variant [54]. While we validate the significance of this head, this

Closed-Loop 2D SSM Decoder. To enhance the generalizability and the ability of our model for longer-horizon, we extend the closed-loop decoder module [28], which is similar to autoregression, to multivariate time series. We use distinct processes for the inputs and outputs, using additional matrices D 1 and D 2 in each decoder 2D SSM, we model future input time-steps explicitly:

where u v,t is the next input and y v,t is the output. Note that the other parts (recurrence) are the same as Equation 6. Figure 2 illustrate the architecture of closed-loop 2D SSM.

3.4 Theoretical Justification

In this section, we provide some theoretical evidences for the performance of Chimera. These results are mostly revisiting the theorems by Zhang et al. [28] and Baron et al. [49], and extending them

for Chimera. In the first theorem, we show that Chimera recovers several classic methods, and pre-processing steps as it can recover SpaceTime [28] and additionally because of its design, it can recover SARIMA [15]:

Theorem 3.4. Chimera can represent seasonal autoregressive process, SARIMA [15], SpaceTime [28], and so ARIMA [14], and exponential smoothing [16].

Theorem 3.5. Chimera can represent S4nd [32], TSM2 [31], and TSMixer [30].

Next theorem compares the expressiveness of Chimera with some existing 2D deep SSMs. Since Chimera can recover 2DSSM [49], it can express full-rank kernels with a constant number of parameters:

Theorem 3.6. Similar to 2DSSM [49], Chimera can express full-rank kernels with O (1) parameters, while existing deep SSMs [32, 31] require O ( N ) parameters to express N -rank kernels.

4 Experiments

Goals and Baselines. We evaluate Chimera on a wide range of time series tasks. In § 4.1 we compare Chimera with the state-of-the-art general multivariate time series models [8, 9, 18, 20, 21, 24, 26, 27, 31, 55-57] on long-term forecasting and classification tasks. In the next part, we test the performance of Chimera in short-term forecasting. In § 4.1 we perform a case study on human neural activity to classify seen images, which requires capturing complex dynamic dependencies of variates, to test the ability of Chimera in capturing cross-variate information and the significance of data-dependency. In § 4.2, we evaluate the significance of the Chimera's components by performing ablation studies. In § 4.2, we evaluate whether the superior performance of Chimera coincide with its efficiency. Finally, we test the Chimera's generalizability on unseen variates and further evaluate its ability to filter irrelevant context in § 4.3. The details and additional experiments are in Appendix I.

Chimera

TSM2

Simba

TCN

iTransformer

RLinear

PatchTST

Crossformer

TiDE

TimesNet

DLinear

(ours)

[2024]

[2024]

[2024]

[2024]

[2023]

[2023]

[2023]

[2023]

[2023]

[2023]

MSE MAE MSE MAE MSE MAE MSE MAE MSE

MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE

ETTm1

0.355

0.381

0.361

-

0.383 0.396

0.351 0.381

0.407

0.410

0.414 0.407 0.387 0.400 0.513 0.496

0.419 0.419 0.400 0.406 0.403 0.407

ETTm2

0.252

0.317 0.267

-

0.271 0.327 0.253

0.314

0.288

0.332

0.286 0.327 0.281 0.326 0.757 0.610

0.358 0.404 0.291 0.333 0.350 0.401

ETTh1

0.408 0.425

0.403

-

0.441 0.432 0.404

0.420

0.454

0.447

0.446 0.434 0.469 0.454 0.529 0.522

0.541 0.507 0.458 0.450 0.456 0.452

ETTh2

0.321 0.377

0.333

-

0.361 0.391 0.322 0.379 0.383

0.407

0.374 0.398 0.387 0.407 0.942 0.684

0.611 0.550 0.414 0.427 0.559 0.515

ECL

0.154 0.249

0.169

-

0.185 0.274 0.156 0.253 0.178

0.270

0.219 0.298 0.205 0.290 0.244 0.334

0.251 0.344 0.192 0.295 0.212 0.300

Exchange 0.311

0.358

0.443

-

-

-

0.302

0.366 0.360

0.403

0.378 0.417 0.367 0.404 0.940 0.707

0.370 0.413 0.416 0.443 0.354 0.414

Traffic

0.403 0.286 0.420

-

0.493 0.291

0.398 0.270

0.428

0.282

0.626 0.378 0.481 0.304 0.550 0.304

0.760 0.473 0.620 0.336 0.625 0.383

Weather

0.219 0.258

0.239

-

0.255 0.280 0.224 0.264 0.258

0.278

0.272 0.291 0.259 0.281 0.259 0.315

0.271 0.320 0.259 0.287 0.265 0.317

1

st

Count

4

5

1

-

0

0

3

3

0

0

0

0

0

0

0

0

0

0

0

0

0

0

4.1 Main Results: Classification and Forecasting

Long-Term Forecasting. We perform experiments in long-term forecasting task on benchmark datasets [6]. Table 1 reports the average of results over different horizons (for the results of each see Table 8). Chimera shows outstanding performance, achieving the best or the second best results in all the datasets and outperforms baselines in 5 out of 8 benchmarks. Notably, it surpasses extensively studied MLP-based and Transformer-based models while being more efficient (see Table 3, Figure 4, and Appendix I), providing a better balance of performance and efficiency. It further significantly outperforms recurrent models, including very recent Mamba-based architectures [31, 57], unleashing the potential of classical models, SSMs, when are carefully designed in deep learning settings.

Classification and Anomaly Detection. We evaluate the performance of Chimera in ECG classification on PTB-XL dataset [7] (see Table 2), speech classification [39] (Table 3), 10 multivariate datasets from UEA Time Series Classification Archive [60] (see Figure 3 and Table 10), and anomaly detection tasks on five widely-used benchmarks: SMD [10], SWaT [61], PSM [62] and SMAP [63] (see Figure 3 and Table 11). For each benchmark, we use the state-of-the-art methods that are

applicable to the task as the baselines. Table 2 reports the performance of Chimera and baselines on ECG classification tasks. Chimera outperforms all the baselines in 4/6 tasks, while achieving the second best results on the remaining tasks. Since these tasks are univariate time series, we attribute the outstanding performance of Chimera, specifically compared to SpaceTime [28], to its ability of capturing seasonal patterns and its input-dependent parameters, resulting in dynamically learn dependencies.

Table 3 reports the results on speech audio classification task, which require long-range modeling of time series. Due to the length of the sequence (16K), LSSL [34] and Transformer [17] has out of memory (OOM) issue, showing the efficiency of Chimera compared to alternative backbones.

Finally, we report the summary of the results in multivariate time series classification and anomaly detection tasks in Figure 3. The full list of results can be found in Table 10 and Table 11. Chimera shows outstanding performance, achieving highest average accuracy and F1 score in classification and anomaly detection tasks even compared to very recent and state-of-the-art methods [8, 9].

Short-Term Forecasting. Our evaluation on short-term forecasting tasks on M4 benchmark datasets [64] reports in Table 4 (Full list in Table 9), which also shows the superior performance of Chimera compared to baselines.

Case Study of Brain Activity. Input dependency is a must to capture the dynamic of dependencies. To support this claim, we use BVFC [2] (multivariate time series only), which aim to classify seen images by its corresponding brain activity response. This task, requires focusing more on the dependencies of brain units and their responses rather than the actual time series. Also, since each window corresponds to a specific image, the model needs to capture the dependencies based on the current window, requiring to be input-dependent. Results are reported in Table 5. Chimera significantly outperforms all the baselines including our Chimera but without data-dependent parameters (convolution form). Due to the large number of brain units, i.e., 9K, in the first dataset, transformer-based methods face OOMissue. However, they are also data-dependent and so shows the second best results in second and third datasets. This results support the significance of data-dependency in Chimera.

4.2 Ablation Study and Efficiency

To evaluate the significance of the Chimera's design, we perform ablation studies and remove one of the components at each time, keeping other parts unchanged. Table 6 reports the results. The first row reports the Chimera's performance, while row 2 uses unidirectional recurrence along the variate dimension, row 3 removes the gating mechanism, row 4 uses convolution form (data-independent), and row 5 removes the module for seasonal patterns. The results show that all the components of Chimera contributes to its performance.

Method

Chimera

Chimera (ind.)

SpaceTime

iTrans.

Trans.

DLinear

(ours)

(ours)

[2023]

[2022]

[2024]

[2017]

[2023]

BVFC

(9K)

69.41

62.36

41.20

40.89

OOM

OOM

39.74

BVFC

(1K)

58.99

50.25

34.31

35.19

54.18

43.60

33.09

BVFC

(400)

51.08

45.17

33.58

33.76

48.22

38.05

32.73

Method

ETTh1

ETTm1

ETTh2

MSE

MAE

MSE

MAE

MSE

MAE

Chimera

0.408

0.425

0.355

0.381

0.321

0.377

Uni.-directional

0.416

0.430

0.368

0.397

0.334

0.386

w/o Gating

0.424

0.438

0.367

0.395

0.331

0.382

Input-independent

0.473

0.501

0.433

0.412

0.380

0.414

w/o seasonal

0.436

0.435

0.366

0.391

0.343

0.395

ClassificationPerformance

AnomalyDetectionPerformance

Chimera

ModernTCN

TimesNet

CrossFormer

FlowFormer

ETSFormer

LSSL

LSTM

Accuracy (%)

70

80

90

AverageF1Score

70

80

90

Transformers

Linear

Recurrent

Convolutional

350

Chimera (Scan)

sec/epoch)

300

Chimera (w/o Scan)

Chimera (Conv)

250

Transformer

200

LSTM

150

SpaceTime

ime

100

三

50

0

250

500

750

1000

1250

1500

1750

2000

SequenceLength

Length of Time Series. Weperform experiments on the effect of the sequence length on the efficiency of Chimera and baselines. The results are reported in Figure 4. Chimera scales linearly with respect to the sequence length and has smoother scaling than S4 [34] and Transformers [17]. These results also highlight the significance of our algorithm that uses 2D parallel scans for training Chimera. This algorithm results in ≈ × 4 faster training, which is very closed to the convolutional format without data dependency. Chimera also has a close running time to SpaceTime [28], which has 1D recurrent.

4.3 Selection Mechanism Along Time and Variate

Variate Generalization. We argue that the data-dependency with discretization allows the model to filter the irrelevant context based on the input, resulting in more generalizability. Inspired by Liu et al. [27], we train our model (and baseline) on 20% of variates and evaluate its generalizability to unseen variates. The results are reported in Figure 5. Chimera has on par generalizability compared to Transformers (when applied along the variate dimension), which we attributes to its data-dependent parameters as Chimera with convolution form performs poorly on unseen variates.

Context Filtering. Increasing the lookback length does not necessarily result in better performance for Transformers [27]. Due to the selection mechanism of Chimera, we expect it to filter irrelevant information and monotonically performs better. Figure 6 reports the Chimera's performance (w/ and w/o data-dependency) and transformer-based baselines [6, 22] while varying the lookback length. Chimera due to its selection mechanism monotonically performs better with increasing the lookback.

0.029

Traffic

ECL

1610

Traffic

ECL

0.8

100%

20%

0.4

0.137

20%

100%

0.8

0.35

MSB

0.7

0.6

0.5

0.4

0.005

0.152

0.004

MSE

0.3

0.2

0.05

0.06

E

S

M

0.6

0.5

0.7

E

S

M

0.30

0.25

0.20

0.3

0.4

0.15

0.2

0.1

48

96

192

336720

48

96

192

336

720

0.1

Chimera

Chimera (ind.)

FlowFormer

Informer

0.0

FlowFormer

iTransformer

Chimera (ind.)

Chimera

0.0

FlowFormer

iTransformer

Chimera (ind.)

Chimera

5 Conclusion

This paper presents Chimera, a three-headed 2-dimensional SSM model with provably high expressive power. Chimera is based on 2D SSMs with careful design of parameters that allows it to dynamically and simultaneously capture the dependencies along both time and variate dimensions. We provide different views of our 2D SSM for efficient training, and present a data-dependent formulation with a fast implementation using 2D scans. Our experimental and theoretical results support the effectiveness and efficiency of Chimera in a wide range of tasks.

Acknowledgments and Disclosure of Funding

This material is based upon work supported by the National Science Foundation under Award No. 2306556. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

References

Ali Behrouz and Farnoosh Hashemi. Brain-mamba: Encoding brain activity via selective state space models. In Conference on Health, Inference, and Learning , pages 233-250. PMLR, 2024.

Ali Behrouz, Parsa Delavari, and Farnoosh Hashemi. Unsupervised representation learning of brain activity via bridging voxel activity and functional connectivity. In International conference on machine learning (ICML) , 2024.

Plamen Ch Ivanov, Luis A Nunes Amaral, Ary L Goldberger, Shlomo Havlin, Michael G Rosenblum, Zbigniew R Struzik, and H Eugene Stanley. Multifractality in human heartbeat dynamics. Nature , 399(6735):461-465, 1999.

Kelum Gajamannage, Yonggi Park, and Dilhani I Jayathilake. Real-time forecasting of time series in financial markets using sequentially trained dual-lstms. Expert Systems with Applications , 223:119879, 2023.

Steve Pincus and Rudolf E Kalman. Irregularity, volatility, risk, and financial market time series. Proceedings of the National Academy of Sciences , 101(38):13709-13714, 2004.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11106-11115, 2021.

Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojciech Samek, and Tobias Schaeffter. Ptb-xl, a large publicly available electrocardiography dataset. Scientific data , 7(1):1-15, 2020.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=ju_Uqw384Oq .

Donghao Luo and Xue Wang. ModernTCN: A modern pure convolution structure for general time series analysis. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=vpJMJerXHU .

Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2828-2837, 2019.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI , 2023.

Andrew C Harvey. Forecasting, structural time series models and the kalman filter. Cambridge university press , 1990.

Masanao Aoki. State space modeling of time series . Springer Science & Business Media, 2013.

David J Bartholomew. Time series analysis forecasting and control., 1971.

Michael Bender and Slobodan Simonovic. Time-series modeling for long-range stream-flow forecasting. Journal of Water Resources Planning and Management , 120(6):857-870, 1994.

Peter R Winters. Forecasting sales by exponentially weighted moving averages. Management science , 6(3):324-342, 1960.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A , 379(2194):20200209, 2021.

Zonglei Chen, Minbo Ma, Tianrui Li, Hongjun Wang, and Chongshou Li. Long sequence time-series forecasting with deep learning: A survey. Information Fusion , 97:101819, 2023.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381 , 2022.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems , 34:22419-22430, 2021.

Haixu Wu, Jialong Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Flowformer: Linearizing transformers with conservation flows. In ICML , 2022.

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations , 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning , pages 27268-27286. PMLR, 2022.

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR , 2020.

Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The eleventh international conference on learning representations , 2023.

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview. net/forum?id=JePfAI8fah .

Michael Zhang, Khaled Kamal Saab, Michael Poli, Tri Dao, Karan Goel, and Christopher Re. Effectively modeling time series with simple discrete state spaces. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=2EpjkjzdCAa .

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=Jbdc0vTOcol .

Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O Arik, and Tomas Pfister. Tsmixer: An all-mlp architecture for time series forecasting. arXiv preprint arXiv:2303.06053 , 2023.

Ali Behrouz, Michele Santacatterina, and Ramin Zabih. Mambamixer: Efficient selective state space models with dual token and channel selection. arXiv preprint arXiv:2403.19888 , 2024.

Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. Advances in neural information processing systems , 35:2846-2861, 2022.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023.

Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations , 2022. URL https: //openreview.net/forum?id=uYLFoz1vlAC .

Sun-Yuan Kung, Bernard C Levy, Martin Morf, and Thomas Kailath. New results in 2-d systems theory, part ii: 2-d state-space models-realization and the notions of controllability, observability, and minimality. Proceedings of the IEEE , 65(6):945-961, 1977.

Ettore Fornasini and Giovanni Marchesini. Doubly-indexed dynamical systems: State-space models and structural properties. Mathematical systems theory , 12(1):59-72, 1978.

Rikus Eising. Realization and stabilization of 2-d systems. IEEE Transactions on Automatic Control , 23(5):793-799, 1978.

Ts Hinamoto. Realizations of a state-space model from two-dimensional input-output map. IEEE Transactions on Circuits and Systems , 27(1):36-44, 1980.

Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209 , 2018.

Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, and Chris I Baker. Things-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. Elife , 12:e82580, 2023.

Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data , 10(1):1, 2023.

Corentin Tallec and Yann Ollivier. Can recurrent neural networks warp time? In International Conference on Learning Representations , 2018. URL https://openreview.net/forum? id=SJcKhk-Ab .

Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Re. How to train your HIPPO: State space models with generalized orthogonal basis projections. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=klK17OQ3KB .

Albert Gu, Caglar Gulcehre, Thomas Paine, Matt Hoffman, and Razvan Pascanu. Improving the gating mechanism of recurrent neural networks. In International Conference on Machine Learning , pages 3800-3809. PMLR, 2020.

Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=yJE7iQSAep .

Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems , 34:572-585, 2021.

Junxiong Wang, Jing Nathan Yan, Albert Gu, and Alexander Rush. Pretraining without attention. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 58-69, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.5. URL https://aclanthology.org/2023.findings-emnlp.5 .

Ali Behrouz and Farnoosh Hashemi. Graph mamba: Towards learning on graphs with state space models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 119-130, 2024.

Ethan Baron, Itamar Zimerman, and Lior Wolf. A 2-dimensional state space layer for spatial inductive bias. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=BGkqypmGvm .

Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=Ai8Hw3AXqks .

William Toner and Luke Darlow. An analysis of linear time series forecasting models. International conference on machine learning (ICML) , 2024.

Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. Advances in Neural Information Processing Systems , 36, 2023.

Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941 , 2017.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan K Mathur, Rajat Sen, and Rose Yu. Longterm forecasting with tiDE: Time-series dense encoder. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=pCbC3aQB5W .

Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems , 35:5816-5828, 2022.

Badri N. Patro and Vijay S. Agneeswaran. Simba: Simplified mamba-based architecture for vision and multivariate time series, 2024.

Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv preprint arXiv:2305.10721 , 2023.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11121-11128, 2023.

Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 , 2018.

Aditya P Mathur and Nils Ole Tippenhauer. Swat: A water treatment testbed for research and training on ics security. In 2016 international workshop on cyber-physical systems for smart water networks (CySWater) , pages 31-36. IEEE, 2016.

Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages 2485-2494, 2021.

Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 387-395, 2018.

Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I Webb, Rob J Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive. arXiv preprint arXiv:2105.06643 , 2021.

Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201.12886 , 2022.

Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. ICLR , 2019.

T. Zhang, Yizhuo Zhang, Wei Cao, J. Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186 , 2022.

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time series forecasting. In NeurIPS , 2022.

S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Comput. , 1997.

Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems , 33: 1474-1487, 2020.

Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. In International Conference on Learning Representations , 2018. URL https://openreview. net/forum?id=HyUNwulC-.

George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics) , 17(2):91-109, 1968.

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014.

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International journal of forecasting , 36(3): 1181-1191, 2020.

Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In International conference on machine learning (ICML) , 2024.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning , pages 5156-5165. PMLR, 2020.

Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning , pages 9355-9366. PMLR, 2021.

Jun Ma, Feifei Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722 , 2024.

Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166 , 2024.

Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234 , 2024.

Yuxin Jia, Youfang Lin, Xinyan Hao, Yan Lin, Shengnan Guo, and Huaiyu Wan. WITRAN: Water-wave information transmission and recurrent acceleration network for long-range time series forecasting. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=y08bkEtNBK .

Romain Ilbert, Ambroise Odonnat, Vasilii Feofanov, Aladin Virmaux, Giuseppe Paolo, Themis Palpanas, and Ievgen Redko. Unlocking the potential of transformers in time series forecasting with sharpness-aware minimization and channel-wise attention. arXiv preprint arXiv:2402.10198 , 2024.

Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems , 35:12677-12690, 2022.

Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. Koopa: Learning non-stationary time series dynamics with koopman predictors. Advances in Neural Information Processing Systems , 36, 2023.

Rui Wang, Yihe Dong, Sercan O Arik, and Rose Yu. Koopman neural operator forecaster for time-series with temporal distributional shifts. In The Eleventh International Conference on Learning Representations , 2023.

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems , 35:9881-9893, 2022.

Chi-Tsong Chen. Linear system theory and design . Saunders college publishing, 1984.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR , 2018.

Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurIPS , 2019.

Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In NeurIPS , 2019.

Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. In ICLR , 2021.

A Background

A.1 1D Space State Models

1D Space State Models (SSMs) are linear time-invariant systems that map input sequence x ( t ) ∈ R L ↦→ y ( t ) ∈ R L [13]. SSMs use a latent state h ( t ) ∈ R N × L , transition parameter A ∈ R N × N , and projection parameters B ∈ R N × 1 , C ∈ R 1 × N to model the input and output as:

Most existing SSMs [34, 33, 31], first discretize the signals A , B , and C . That is, using a parameter ∆ and zero-order hold, the discretized formulation is defined as:

where ¯ A = exp( ∆A ) and ¯ B = ( ∆A ) -1 (exp ( ∆A -I )) . ∆B . [70] show that discrete SSMs can be interpreted as both convolutions and recurrent networks: i.e.,

which makes their training and inference very efficient as a convolution and recurrent model, respectively.

A.2 Data Dependency

Above discrete SSMs are based on data-independent parameters. That is, parameters ∆ , ¯ A , ¯ B , and C are time invariant and are the same for any input. Gu and Dao [33] argue that this time invariance has the cost of limiting SSMs effectiveness in compressing context into a smaller state [33]. To overcome this challenge, they present a selective SSMs (S6) block that effectively selects relevant context by enabling dependence of the parameters ¯ B , ¯ C , and ∆ on the input x t , i.e.:

where Linear ( . ) is a linear projection and Softplus ( . ) = log(1 + exp( . )) . This data dependency comes at the cost of efficiency as the model cannot be trained as a convolution. To overcome this challenge, Gu and Dao [33] show that the linear recurrence in Equation 1 can be formulated as an associative scan [71], which accepts efficient parallel algorithms.

B Additional Related Work

Classical Approach. Modeling time series data is a long-standing problem and has attracted much attention during the past 60 years. There have been several mathematical models to capture the time series traits like exponential smoothing[16], autoregressive integrated moving average (ARIMA) [14], SARIMA [15], Box-Jenkins method [72], and more recently state-space models [12, 13]. Despite their more interpretability, these methods usually fail to capture non-linear dependencies and also often require manually analyzing time series features (e.g., trend or seasonality), resulting in lack of generalizability.

Recurrent and Deep State Space Models. Another group of relevant studies to ours is deep sequence models. A common class of architectures for sequence modeling are recurrent neural networks such as like GRUs [73], DeepAR [74], LSTMs [69]. The main drawback of RNNs is their potential for vanishing/exploding gradients and also their slow training. Recently, linear attention methods with fast training attracted attention [75-77]. Katharopoulos et al. [76] show that these methods have recurrent formulation and can be fast in inference.

Recently, deep state space models have attracted much attention as the alternative of Transformers [17], due to their fast training and inference [70]. These methods are the combination of traditional SSMs

with deep neural networks by directly parameterizing the layers of a neural network with multiple linear SSMs, and overcome common recurrent training drawbacks by leveraging the convolutional view of SSMs [70, 45, 46, 34, 50]. Recently, Gu and Dao [33] present a new formulation of deep SSMs by allowing the parameters to be the function of inputs. This architecture shows promissing potential in various domains like NLP [33], vision [78, 79, 31], graphs [48], DNA modeling [33, 80].

All the above methods are design for 1D data, meaning that the states depends on one variable. There are, however, a few studies that uses 2D SSMs in deep learning settings. S4ND [32] uses continuous signals to model images. These methods not only consider two separate SSM for the axes, but it also directly treat the system as a continuous system without discretization step. Furthermore, S4ND has data-independent parameters. Another similar approach is 2DSSM [49], that models images as discrete signals. That is, the initial SSM model is discrete and again there is a lack of discretization step, which is important for time series as we discussed earlier. Also, their method again is based on data-independent parameters. Both S4ND and 2DSSM can be computed as a convolution. We, however, present a new scanning technique for fast training of 2D SSMs, even with input-dependent parameters.

WITRAN [81] is a 2D RNN approach for univariate time series, which is different from our approaches from different aspects: (1) Goal: The main goal of WITRAN is to use 2D RNN to model univariate time series data. That is, the first dimension corresponds to short-term, and the second dimension corresponds to long-term patterns. On the other hand, our 2D SSM aims to model multivariate time series where the first dimension corresponds to time, and the second dimension corresponds to variates. (2) Backbone: WITRAN is based on GSC (LSTM-like cell), which uses non-linear transition. Our 2D SSM is based on state space models, is simpler, and uses linear transitions. (3) Data-dependency: WITRAN is based on data-independent parameters, meaning that it uses the same transition process for all time steps. 2D SSM, however, is based on data-dependent parameters, which allows it to filter irrelevant/noisy time steps. (4) Recurrence, Hidden States, and Training: Although both models have recurrent form, WITRAN's recurrence is over extracted short-term and long-term patterns. Our 2D SSM's recurrence, however, is over time and variate dimensions. WITRAN uses only one hidden state for each state of the system, while 2D SSM uses 2 different hidden states for each state of the system, allowing more flexibility to capture cross-variate and cross-time information.

Other methods. Transformer-based models have attracted much attention over recent years for multivariate time series forecasting, when modeling the complex relationships of co-variates or along the time dimension is required [24-26, 11, 6, 23, 21, 82, 29]. Several studies have focused on designing more efficient and effective attentions with using special traits of time series [20]. Some other studies have focused on extracting long-term information for better forecasting [29, 83]. In addition to transformers, linear models also have shown promising results [8, 30]. For example, Chen et al. [30] present TSMixer, an all-MLP architecture for time series forecasting, with promising performance. Due to the expressive power of our 2D SSM, these linear methods sometimes can be viewed as a special case of 2D SSMs. Recently, convolution-based models for time series have shown promising results [9]. These methods by using global kernels enhance the global receptive field. Our data-independent formulation of Chimera is connected to this line of work as it can be written as a global convolution.

Recently, Koopman-based deep models [84, 85] are designed to capture non-linear dependencies in the data. Existing methods, however, does not consider dynamics in different variates and might miss complex dependencies of variates. Moreover, despite their promising results in the time series forecasting, these approaches can be hard to be parallelizable and so might have longer training for large models.

C Details of the Discretization

Given PDE with initial condition h (0 , 0) = 0 :

over the sampling intervals [ k ∆ t (1) , ( k +1)∆ t (1) ] and [ ℓ ∆ t (2) , ( ℓ +1)∆ t (2) ] we have:

and so:

Similarly, for the second equation we have:

and so:

Next, the integrals can be simplified as:

and

and similarly for the third and fourth equations we have:

and

Using ZOH assumption, we have:

Accordingly, the discretized form is as follows:

(47)

D Details of the Structure of Transition Matrices

Definition D.1 (Companion Matrix) . A matrix A ∈ R N × N has companion form if it can be written as:

which means that:

and

These matrices can be decomposed into a shift and a low-rank matrix. That is:

This formulation can help us to compute the power of A faster in the convolutional form, as discussed by Zhang et al. [28].

E Bidirectionality Across Variates

The causal nature of the 2D SSM result in limited information flow along the variate dimension as variate are not ordered. To overcome this challenge, inspired by the bi-directional 1D SSMs [47, 48, 1], we use two different modules for forward and backward pass along the variate dimension:

(1)

v,t

h

f

+1

= ¯

A

f

1

h

(1)

v,t

+ ¯

A

f

2

h

(2)

v,t

+ ¯

B

f

1

x

v,t

+1

,

(2)

v

h

f

+1

,t

= ¯

A

f

3

h

(1)

v,t

+ ¯

A

f

4

h

(2)

v,t

+ ¯

B

f

2

x

v

+1

,t

,

where ¯ A τ 1 , ¯ A τ 2 , ¯ A τ 3 , ¯ A τ 4 ∈ R N × N , ¯ B τ 1 , ¯ B τ 2 ∈ R N × 1 , and C τ 1 , C τ 2 ∈ R 1 × N are parameters of the model, h (1) τ v,t , h (2) τ v,t ∈ R N × d are hidden states, x v,t ∈ R 1 × d is the input, and τ ∈ { f, b } . Figure 2 illustrates the bi-directional recurrence process in Chimera. For the sake of simplicity, we continue with unidirectional pass, but adapting them for bi-directional setting is simple as we use two separate blocks, each of which for a direction.

F Time Complexity of 2D-SSM

Let T be the length of the time series and V be the number of variates. Since for each variate we have O ( T ) recurrence and we have V variates, the total number of recurrence is O ( TV ) . In each step, we have the matrix multiplication of transition matrices A i with the hidden states and also B i s with the input. Accordingly, if we use d as the state dimension, the complexity is O ( d 2 TV ) . Given the fact that d is usually a small number, the overall complexity is linear with respect to each of V and T .

G Theoretical Results

G.1 Proof of Theorem 3.2

In this part, we want to prove that ⋇ is associative. This operator is defined as:

Accordingly, we have:

f

f

re-using the definition of ⋇ , we have:

Using the fact that ⊙ and ⊗ are associative, we have:

which proves the theorem.

G.2 Proof of Theorem 3.3

For each v, t , we can pre-compute B 1 x v,t and B 2 x v,t +1 . Accordingly, all the following parameters are pre-computed:

for all inputs x v,t and i, j, k, ℓ ∈ { 0 , 1 } . Now, starting from ( S (1) 0 , 0 S (2) 0 , 0 ) = ( I I 0 I I 0 ) , we have:

Re-using operator ⋇ , we have:

Fixing the variate by putting 0 as the initial points of the operator, and then looking at the third element of each row, these elements are calculating the hidden states of the recurrent (it can be shown by a straightforward induction). Accordingly, using this operation, we can recursively calculate the outputs of 2D SSM.

However, using Theorem 3.2, we know that this is an associative operation, so instead of calculating in the recurrent form, we can use parallel pre-fix sum make this computation parallel, decreasing the sequential operations required to calculate the hidden states. Note that since our above operation can model the problem as an parallel prefix, all the algorithms for this problem can be used to enhance the efficiency.

G.3 Proof of Theorem 3.4

To prove this theorem, we need to (1) show that Chimera can recover SpaceTime. Given this, since SpaceTime is capable of recovering ARIMA [14], exponential smoothing [16], and controllable linear time-invariant systems [87], we can conclude that Chimera can also recover these methods. Then, (2) we need to prove that Chimera can recover SARIMA. This is the model that SpaceTime is not capable of recovering due to the additional seasonal terms.

Note that using A 2 = A 3 = A 4 = 0 , results in a 1D SSM, with companion matrix as the structure of A 1 , which is SpaceTime. Accordingly, SpaceTime is a special case of Chimera when the recurrence only happen along the time direction.

Note that as discussed in Proposition 3.1, multiplying the discretization parameter ∆ results in multiplying the steps. Accordingly, using s as the ∆ in our seasonal module and also letting A 2 = A 3 = A 4 = 0 for the seasonal module, we can model the seasonal terms in the formulation of SAR ( p, q, s ) , meaning that Chimera can also recover SARIMA which is ARIMA with seasonal terms. Note that the reason that Chimera is capable of such modeling is that it uses two heads separately for trend and seasonal terms. Therefore, using different discretization parameters, each can model their own corresponding terms in SAR ( p, q, s ) .

G.4 Proof of Theorem 3.5

Similar to the above, using A 2 = A 3 = 0 , our formulation is equivalent to S4D, while we use diagonal matrices as the structure of A 1 . Similarly, as discussed by Behrouz et al. [31], MambaMixer is equivalent to S4ND but on patched data. Using our Theorem 5, we can recover linear layers, resulting in recovering TSMixer by setting A 2 = A 3 = 0 .

G.5 Proof of Theorem 3.6

We in fact will show that restricting Chimera results in recovering 2DSSM [49]. As discussed earlier, this method do not use discretization and initially starts from a discrete system. Also, it uses inputindependent parameters. Therefore, we use Linear ∆ 1 ( . ) = Linear ∆ 2 ( . ) as broadcast function, and restrict Chimera to have input-independent parameters, then Chimera can recover 2DSSM [49].

H Experimental Settings

We provide the description of datasets in Table 7.

H.1 Baselines

In our experiments, we use the following baselines:

Table 8: TSM2 [31], Simba [57], TCN [9], iTransformer [27], RLinear [58], PatchTST [29], Crossformer [26], TiDE [55], TimesNet [8], DLinear [59], SCINet [56], FEDformer [24], Stationary [86], Autoformer [21].

Table 9: ModernTCN [9], PatchTST [29], TimesNet [8], N-HiTS [65], N-BEATS ∗ [66], ETSformer [20], LightTS [67], DLinear [11], FEDformer [24], Stationary [68], Autoformer [21], Pyraformer [23], Informer [6], Reformer [25], LSTM [69].

Table 10: LSTM [69], LSTNet [88], LSSL [34], Trans.former [17], Reformer [25], Informer [6], Pyraformer [23], Autoformer [21], Station. [68], FEDformer [24], ETSformer [20], Flowformer [22], DLinear [11], LightTS. [67], TimesNet [8], PatchTST [29], MTCN [9].

For the results of the baselines, we re-use the results reported by Wu et al. [8], or from the original cited papers.

I Additional Experimental Results

I.1 Long Term Forecasting Full Results

The complete results of long term forecasting are reported in Table 8.

I.2 Short-Term Forecasting

The complete results of short term forecasting are reported in Table 9.

I.3 Classification

The complete results of time series classification are reported in Table 10.

I.4 Anomaly Detection

The complete results of anomaly detection tasks are reported in Table 11.

J Additional Ablation Study

K Comparison with Koopman-based Deep Models

We also compare the results of Chimera with Koopman-based deep models [84]. In ECL dataset, Chimera achieves 0.132, 0.141, 0.144 for ( h = 96 , 144 , 192 ), while Koopa [84] achieves 0.136, 0.149, 0.156. Similarly, in ETTh2, Chimera achieves 0.262, 0.309, 0.320 for ( h = 96 , 144 , 192 ), while Koopa achieves 0.297, 0.333, 0.356. Similarly, in Exchange, Chimera achieves 0.077, 0.126, 0.159 for ( h = 96 , 144 , 192 ), while Koopa achieves 0.083, 0.130, 0.184. Since Liu et al. [84] has focused on these datasets and values for the horizon, we use the same setting for Chimera. The results for Koopa are from its original paper.

L Limitations

The main goal of this study is to enhance the time series modeling with a wide range of impact on society, from improving the healthcare system using developing deep learning models for analysing medical health records to forecasting stock. We, however, emphasis that our work is a proof-ofconcept, meaning that it has error modes.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: It is available in this link.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See above question.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See question 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer:[NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Chimera, a novel two-dimensional State Space Model (SSM) that can effectively model multivariate time series data. The model is designed to address several key challenges in multivariate time series modeling, including nonlinear dynamics along the temporal dimension, inter-variate dependencies, and efficient training and inference. The proposed discretization is original and suitable for the time series modality. Comprehensive experiments are provided to validate the effectiveness of multiple time series tasks.\n\nSTRENGTHS:\n* The motivations are well-summarized and the paper's writing is easy to follow.\n* The method exhibits promising results in handling multiple time series tasks.\n* The proposed 2D SSM is novel and the bi-directional modification for variate relations makes sense.\n\nWEAKNESSES:\n* About several obscure writing, which is detailed as follows:\n\n    (1) The second paragraph of Section 2 introduces Multi-Dimensional State Space Models, which are stated to be different from typical multi-dimensional SSMs like Mamba. For better clarification, the authors could elaborate more on the 2D grid spanning by the (seemingly) temporal variables $t^{(1)}$ and $t^{(2)}$. I suggest avoiding the plausible notation since \"each state is a function of both time and variates\" (Line 129)\", not both temporal variables.\n\n    (2) Would the author explain the meaning of $N_i$, $\\tau_i$ in Line 93?\n\n    (3) Some overloaded notations: (1) $x$ in Line 89 (complex number) and Line 217 (input time series, real number vector); $\\mathbf{x}$  in Line 85 (real number vector) and Line 95 (complex number).\n\n    (4) Unify the usage of \"variable\" and \"variate\".\n\n    (5) Equation 8: $x \\to \\mathbf{x}$.\n\n* About the soundness of 2D SSM. The author uses 2-dimensional SSMs, where \"the first axis corresponds to the time dimension and the second axis is the variates\" (Line 128) and can be \"viewed as the modification of the discrete Roesser’s SSM\" (Line 134). I am uncertain about the soundness, which further incorporates the variable dimension. It is well-acknowledged to depict the variable relationship by correlations instead of transitions (Equation 6, Left). Even though the authors make sensible bi-directional modifications, the non-causal transitions can not directly reveal the relationship between any two variables. It is also influenced by the variable permutation. However, I think it can be addressed by providing experiments on shuffled variates.\n\n* As the author stated the Classical 2D SSMs are only able to capture linear dependencies, It could be better to compare with Koopman-based deep models [1]\\[2], which are similar to SSMs and excel at tackling nonlinear dynamics. I also encourage the author to discuss this branch in related works.\n\n* Concern about the experimens: (1) Whether the classification (anomaly detection) is performed at the series level or the point level? (2) It can be observed from Table 6 that input-independent has a great influence on the performance, could the authors provide more explanation here? (3) About Figure 6, can the author provide a comparison with PatchTST? (4) Could the author provide theoretical complexity analysis on Chimera and other SSMs/Transformers?\n\n[1] Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors.\n\n[2] Koopman Neural Operator Forecaster for Time-series with Temporal Distributional Shifts.\n\nQUESTIONS:\nWhat is the lookback length in long-term forecasting (Table 1)?\n\nLIMITATIONS:\nThe paper does not explicitly discuss limitations. Discussing the limitations of the model would be useful to improve the paper. I also suggest the authors provide the code implementations for a better illustration of the method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed to use a state space model for time series modelling. Instead of using the SSM along the time dimension, the authors also have the space updated along the variables dimension, which makes the established both inter- and intra- variable dependencies.\n\nSTRENGTHS:\n- The motivation is clear and the focus is valid for time series forecasting.\n- The results show that the model holds value in providing better results over different time series tasks with competitive memory consumption.\n\nWEAKNESSES:\n- For forecasting tasks, the way the authors conducted evaluation metrics does not reflect the model performance for the target horizon. See detail in Questions.\n- The method does not present a consistent improvement over prior work.\n- Typos:\n    - The notation does not match the content. For example in Table 8, the first, second and third best results are all not highlighted with the provided notation.\n    - [minor] Checklist missing guidelines.\n\nQUESTIONS:\n- The evaluation is conducted on the forecasting sequence instead of only on the targeting time steps. How is it valid to tell whether the model performs better in the targeting time step or is it only better because it has superior performance in the more recent ones? It is necessary to also add the performance of different methods only on the targeting time step.\n- What is the advantage of using a state space model to establish intervariable dependencies in other neural network structures (such as CNNs or feedforward networks), given that there are no state variations between variables?\n\n\nAdditional\n- The proposed model looks very similar to WITRAN, which conducted 2D RNN. It would be good to have Chimera distinguished from WITRAN.\n\nLIMITATIONS:\nThe limitation of the paper is mainly about how the method may not apply to real-world applications. The technical limitations are not mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of multivariate time series modeling using a neural architecture based on a variation of two-dimensional state-space models (SSMs), referred to as Chimera. This approach features a stack of 2D SSMs combined with nonlinearities, a decomposition of time series into trend and seasonal components, various discretization processes, time-variant parameters, and a novel two-dimensional parallel selective scan for rapid training with linear complexity. The authors demonstrate that Chimera can encompass multiple existing time series models, including SARIMA, SpaceTime, and S4nd. Experiments across various tasks, such as time series forecasting, classification, and anomaly detection, validate the effectiveness of Chimera.\n\nSTRENGTHS:\n- State-space models (SSMs) have proven to be very effective and efficient for modeling temporal data compared to Transformers. Designing new efficient SSM architectures for multivariate time series is an important and challenging problem.\n\n- Chimera allows the state space to vary in two dimensions (time and variate). It also combines multiple components to build a specific architecture for multivariate time series.\n\n- Chimera is compared to multiple baselines across various datasets for tasks such as time series forecasting, classification, and anomaly detection.\n\nWEAKNESSES:\n- **Motivation**\n  - Two-dimensional state-space models (2D-SSMs) have been considered for multivariate time series with specific structures, such as S4ND and 2DSSM. The authors discuss the discretization process and data-independent parameters as new challenges. However, I found lines 113-128 unconvincing regarding the need for a new 2D-SSM for multivariate time series.\n\n  I believe this paper proposes yet another deep model for time series, while it is known that after removing strong seasonality, multivariate time series from real-world phenomena often exhibit weak time and variate dependencies/signals, which do not require such complex models. This is very different from text and image data.\n\n- **Contributions**\n  - The paper essentially combines multiple existing ideas: 2D-SSM, bi-directional models [46], selection mechanisms [32], companion structures [27], convolutional forms [33], etc. There are no substantial or specific contributions. The authors claim that one of their main technical contributions is having input-dependent parameters, which is a relatively weak contribution.\n\n- **Experiments**\n  - Even considering the experimental results, many (simpler) methods achieve similar results to the proposed approach (see Tables 8-11).\n\n  - I also found that the paper tends to overstate its claims, using phrases like \"outstanding performance\" and \"significantly outperforms recurrent models, including very recent Mamba-based architectures.\"\n\n  - I question whether a fair comparison was made between the methods. For example, Chimera involves a time series decomposition (lines 218-219, 236-237), whereas other baselines do not. Can the authors comment on this?\n\n  Additionally, there is a lack of naive baselines, such as methods that treat the multivariate series as multiple univariate series.\n\n  It would also be useful to visualize some time series decompositions of Chimera to validate its accuracy.\n\n  - The authors write, \"We attribute the outstanding performance of Chimera, specifically compared to SpaceTime [27], to its ability to capture seasonal patterns and its input-dependent parameters, resulting in dynamically learned dependencies.\" Why didn't you confirm this by analyzing the results?\n\n  - Given that the authors combine multiple existing ideas/components, I found the ablation study relatively weak.\n    - \"The results show that all the components of Chimera contribute to its performance.\" I do not agree with this statement. Table 6 only shows that input-independence reduces accuracy (significantly).\n    - Why did you only choose ETT datasets?\n\n  - For the forecasting problems, it would be useful to provide the results per horizon or group multiple horizons together to analyze performance over the forecast horizon. Does the improvement come from the first few horizons?\n\n\n\n- **Data**\n  - I question the validity of the benchmark datasets used.\n    - For time series forecasting, there is the Monash Time Series Forecasting Repository (https://forecastingdata.org/). I think the multivariate time series datasets considered in the paper have been overly used and overfitted by the community.\n    - For anomaly detection, I encourage the authors to check the following paper: Multivariate Time Series Anomaly Detection: Fancy Algorithms and Flawed Evaluation Methodology (https://arxiv.org/abs/2308.13068)\n\n- **Other comments**\n  - I think Theorems 1-5 should be labeled as Propositions.\n  - Seasonal autoregressive process: The authors introduce the AR(p) model with a multivariate time series (x_k in R^d, not R^1). This is confusing as there is also the VAR(p) model with a different formulation. The AR(p) presented uses the same coefficients for each dimension.\n  - Table 7: What do you mean by \"series length\"? M4 series do not have six observations. Also, it is not long-term vs short-term but rather multivariate vs univariate time series. This should also be changed in the main text. Does this mean that Chimera has been used with both univariate and multivariate time series? What are the implications? Do you still need the second dimension in that case?\n  - The paper uses different terminologies: \"intervariate information flow,\" \"inter-variate dependencies,\" \"complicated dependencies,\" \"dynamically model the dependencies of variate and time dimensions,\" \"cannot capture the dynamics of dependencies.\" Please be consistent and define these terms clearly. There are a lot of \"buzz\" words without clear definitions.\n\n  - The authors write: \"Note that anomaly detection can be seen as a binary classification task.\" This is supervised anomaly detection, which is rare as labeled anomalies are hard to find. How did you handle the imbalance during training?\n\n- **Typos**\n  - Line 665: \"These matrices can be decompose\"\n  - Line 695: \"recursively calculate the the outputs\"\n\n- Please provide standard errors in your tables.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe limitations have not been discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper introduces Chimera, a novel 2-dimensional State Space Model (SSM) for multivariate time series modeling, addressing key challenges such as capturing complex temporal and inter-variate dependencies, and efficient training. Chimera uses two SSM heads with different discretization processes and time-variant parameters to learn long-term progression, seasonal patterns, and dynamic autoregressive processes. A new 2-dimensional parallel selective scan improves training efficiency. Experimental results show Chimera's superior performance in various benchmarks, including ECG and speech classification, time series forecasting, and anomaly detection, marking a significant advancement in the field.

STRENGTHS:
1. This paper is well written. The notations are clear and the literature review is sufficient.
2. By using SSMs, the proposed Chimera Neural Architecture can achieve faster training and less memory consumption.
3. Experimental results cover a wide range of applications, including Long-Term Forecasting, Anomaly Detection, Short-Term Forecasting, etc. The results are very solid and demonstrate the advantages of adopting the proposed method well.

WEAKNESSES:
1. the writing and expression may need correction: Line 308, it is hard to understand the message:
'they are also data-dependent and so shows the second best results in second and third datasets.'
2. It would be great t know what kind of market dynamics are captured by with visual impact.

QUESTIONS:
What could be the interpretations for Structure of Transition Matrices? For example,  it is observed that a simpler structure of diagonal matrices is effective fuse information along the variate dimension.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13