Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models
Authors: Ali Behrouz
Michele Santacatterina
Ramin Zabih
Venue: NeurIPS 2024 poster (2024)
Keywords: Multivariate Time Series
Time Series Modeling
Time Series Forecasting
Time Series Classification
Primary area: deep_learning_architectures

Abstract: Modeling multivariate time series is a well-established problem with a wide range of applications from healthcare to financial markets. It, however, is challenging as it requires methods to (1) have high expressive power of representing complicated dependencies along the time axis to capture both long-term progression and seasonal patterns, (2) capture the inter-variate dependencies when it is informative, (3) dynamically model the dependencies of variate and time dimensions, and (4) have efficient training and inference for very long sequences. Traditional State Space Models (SSMs) are classical approaches for univariate time series modeling due to their simplicity and expressive power to represent linear dependencies. They, however, have fundamentally limited expressive power to capture non-linear dependencies, are slow in practice, and fail to model the inter-variate information flow. Despite recent attempts to improve the expressive power of SSMs by using deep structured SSMs, the existing methods are either limited to univariate time series, fail to model complex patterns (e.g., seasonal patterns), fail to dynamically model the dependencies of variate and time dimensions, and/or are input-independent.  We present Chimera, an expressive variation of the 2-dimensional SSMs with careful design of parameters to maintain high expressive power while keeping the training complexity linear. Using two SSM heads with different discretization processes and input-dependent parameters, Chimera is provably able to learn long-term progression, seasonal patterns, and desirable dynamic autoregressive processes. To improve the efficiency of complex 2D recurrence, we present a fast training using a new 2-dimensional parallel selective scan. Our experimental evaluation shows the superior performance of Chimera on extensive and diverse benchmarks, including ECG and speech time series classification, long-term and short-term time series forecasting, and time series anomaly detection.

Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models
Ali Behrouz Cornell University ab2947@cornel1.ced
Michele Santacatterina New York University cantam13@nyu. edu
Ramin Zabih Cornell University rdzθcs cornell.....m edu
Abstract
Modeling multivariate time series is well-established problem with wide range of applications from healthcare to financial markets. It, however, challenging as it requires methods (I) have high expressive power representing complicated dependencies along the time axis to capture both long-term progression and sea- sonal patterns, (2) capture the inter-variate dependencies when it is informative, (3) dynamically model the dependencies of variate and time dimensions, and (4) have efficient training and inference for very long sequences. Traditional State Space Models (SSMs) are classical approaches for univariate time series modeling due to their simplicity and expressive power represent linear dependencies. They, however, have fundamentally limited expressive power to capture non-linear depen- dencies, are slow practice, and fail model the inter-variate information flow. Despite recent attempts to improve the expressive power of SSMs by using deep structured SSMs, the existing methods are either limited to univariate time series, fail model complex patterns (e.g., seasonal patterns), fail to dynamically model the dependencies of variate and time dimensions, and/or are present Chimera, expressive variation of the 2-dimensional SSMs with careful design of parameters to maintain high expressive power while keeping the training complexity linear. Using two SSM heads with different discretization processes and input-dependent parameters, Chimera is provably able to learn long-term pro- gression, seasonal patterns, and desirable dynamic autoregressive processes. To improve the efficiency of complex 2D recurrence, we present fast training using a new 2-dimensional parallel selective scan. Our experimental evaluation shows the superior performance of Chimera on extensive and diverse benchmarks, including ECG and speech time series classification, long-term and short-term time series forecasting, and time series anomaly detection.
We
Introduction
Modeling time series is a well-established problem with wide range of applications from health- care [1-3] financial markets [4, 5] and energy management [6]. The complex nature of time series data, its diverse domains of applicability, and its broad range of tasks (e.g., classification [2, 7]. imputation [8, 9], anomaly detection [2, 10]. and forecasting [6]). however, raise fundamental chal- lenges to design effective and generalizable models: (1) The higher-order, seasonal, and long-term patterns in time series require an effective model be able expressively capture complex and autoregressive dependencies; (2) In the presence of multiple variates of time series, effective model need to capture the complex dynamics of the dependencies between time and variate axes. More specifically, most existing multivariate models seem to suffer from overfitting especially when the target time series not correlated with other covariates [11]. Accordingly, an effective model needs to adaptively learn to select (resp. filter) informative (resp. irrelevant) variates; (3) The diverse set of domains and tasks requires effective models to be free of manual pre-processing and domain
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Inpus
Expressive Representation
Complex Patterns
Seasonal Patterna
Learnable diseretization via ZOHL
2D SSM
Ast = f(x) X; x(k
Output
Long-term Progression
A Time
0
Learnable discretization via ZOH
Aple = fx(x) 24 = x(K(ppp)))
Brime Crime Baariato. Citrioa are time dependent.
Chimera
Efficient Training Using 2D Scan
Parallel Souns
dilersstiaation
Variae
ATme
2D SSM
2D SSM
Long Soasonal
Formulatte the 2D selective recurrence as 2D prefix sum problem.
Input
AT-V
Av--T
The Overview Contributions and Architecture Chimera. We present a 2-dimensional SSM with careful expressive paramelerization. uses different learnable discretization processes learn seasonal and long-term progression patterns, and leverages parallelizable and fast training process by re-formulating the 2D input dependent recurrence as 2D prefix sum problem.
Figure I:
knowledge and instead adaptively learn them; and (4) Due to the processing of very long sequences, effective methods need efficient training and inference.
Classical methods (e.g., State Space Models [12, 13], ARIMA [14], SARIMA [15], Exponential Smoothing (ETS) [16]) require manual data preprocessing and model selection, and often are not able to capture complex non-linear dynamics. The raise of deep learning methods and more specifically Transformers [17] has led to significant research efforts to address the limitation of classical methods and develop effective deep models [18-27]. Unfortunately, most existing deep models struggle to achieve all the above four criteria. The main body of research in this direction has focused on designing attention modules that use the special traits of time series [21, 20]. However, the inherent permutation equivariance of attentions contradicts the causal nature of time series and often results in suboptimal performance compared to simple linear methods [11]. Moreover, they often either overlook difference of seasonal and long-term trend or use non-learnable methods to handle them [20].
considerable subset of deep models overlook the importance of modeling the dependencies of variates [11, 28, 29]. These dependencies, however, are not always useful; specifically when the target time series not correlated with other covariates [30]. Despite several studies exploring the importance of learning cross variate dependencies [26, 27, 30], there has been no universal standard and the conclusion has been different depending on the domain and benchmarks. Accordingly, we argue that an effective model need to adaptively learn to capture the dependencies of variates in data-dependent manner. In this direction, recently, Liu et al. [27] argue that attention mechanisms are more effective when they are used across variates, showing the importance of modeling complex non-linear dependencies across the variate axis in data-dependent manner. However, the quadratic complexity of Transformers challenges the model on multivariate time series with large number of variates (c.g., brain activity signals [2] or traffic forecasting [6]), limiting the efficient training and inference (see Table 3, and Table 5).
The objective of this study to develop a provably expressive model for multivariate time series that not only can model the dynamics of the depenendencies along both time and variates, but it also takes advantage of fast training and inference. To this end, we present Chimera, a headed two-dimensional State Space Model (SSM) that is based on linear layers along (i) time, (ii) variates, (iii) viri-evai and (iv) variate-xtime Chimera has a careful parameterization based on the pair of companion and diagonal matrices (see Figure 1), which is provably expressive to recover both classical methods [16, 14, 15], linear attentions, and recent SSM-based models [31, 32). It further uses adaptive module based on 2D SSM with an especially designed discretization process to

---

capture seasonal patterns. While our theoretical results and design of Chimera guarantee the first three criteria of an effective model, due to its 2D recurrence, the naive implementation of Chimera results in slow training. To address this issue, we reformulate its 2D recurrence as the prefix sum problem with 2-dimensional associative operators. This new formulation can be done parallel and has hardware-friendly implementation, resulting in much faster training and inference.
In our experimental evaluation, we explore the performance of Chimera in a wide range of tasks: ECG and audio speech time series classification, long- and short-term time series forecasting, and anomaly detection tasks. We find that Chimera achieve superior or on par performance with state-of satte-of-the-rre art methods, while having faster training and less memory consumption. We perform a case study on the human brain activity signals [2] show (1) the effectiveness of Chimera and (2) evaluate the importance of modeling the dynamics of the variates dependencies.
2 Preliminaries
Notations. In this paper we mainly focus on classification and forecasting tasks. Note that anomaly detection can be seen as a binary classification task, where means "normall" and means "anomaly". We let X ( XN) } E RNT be the input sequences, where N is the number of variates and T' is the time steps. We use X to refer to the value of the series v at time t. In classification (anomaly detection) tasks, we aim to classify input sequences and for forecasting tasks, given an input sequence X,, we aim to predict &; E RRIH i.e., the next H time steps for variate x,, where H called horizon. In 2D SSMs formulation, for a 2-dimensional vector x E Cl, we use x!(1) and x(2) to refer to its real and imaginary components, respectively.
Multi-Dimensioe Dimensional State Space Models. We build our approach on the continuous State Space Model (SSM) but later we make each component of Chimera discrete by designed discretization pro- cess, For additional discussion on ID SSMs see Appendix A. Given parameters A,, E ( x
Br E (2((2) x1 and C E C X.2 for тi { .... 4} and 72 {1,2), the general form of the time-invariant 2D SSM is the map x E C + E C defined by the linear Partial Differential Equation (PDE) with initial condition h(0,0) = 0:
h (t0)) ((2) (1) (()) (())
12h (10))()) (() + B,x (6()), (())))),
(I)
01 h (10),12)) = (Aah(") (11), (22) (())
Aaah (10), (2) + B2x (10) ((1)((2)
(2)
y (10) (00) (C,x ((()) 1(21)
Contrary to the multi-dimensional SSMs discussed by Gu and Dao [33], Gu et al. [34], in which multi-dimension refers to the dimension of the input but with one time variable, the above formulation uses two variables, meaning that the mapping is from 2D grid to 2D grid.
(Seasonal) Autoregressive Process. Autoregressive process is a basic yet essential premise for time series modeling, which models the causal nature of time series. Given p E N. X2 E Rd the simple linear autoregressive relationships between X2 and its past samples 1;- Xg-2............ Xxpp can be modeled as X₁ Φ1 X;-1 + 02Xx-2 + +..., 0pXx-ps where di...., &p are coefficients. This is called AR(p). Similarly, in the presence of seasonal patterns, the seasonal autoregressive process, SAR(p, 4, s). is:
4 where s is the frequency of seasonality, and $1..... Op and . my are coefficients. Note that one can simply extend the above formulation multivariate time series by letting coefficients to be vectors and replace the product with clement-wise product.
X 01x 62x;-2 +.......k------.. 71 12xx-22
3 Chimera: A Three-headed 2-Dimensional State Space Model
In this section, we first present a mathematical model for multivariate time series data and then based on this model, present a neural architecture that can satisfy all the criteria discussed in $1.
3

---

3.1 Motivations & Chimera Model
SSMs have been Long-standing methods for modeling time series [12, 13], mainly due to their simplicity and expressive power to represent complicated and autoregressive dependencies. Their states, however, are the function of a single-variable (e.g., time). Multivariate time series, on the other hand, require capturing dependencies along both time and variate dimensions, requiring the current state of the model to be the function of both time and variate. Classical 2D SSMs [35-38], however, struggle to achieve good performance compared to recent advanced deep learning methods as they are (1) only able to capture linear dependencies, (2) discrete by design, having a pre-determined resolution, and so cannot simply model seasonal patterns, (3) slow in practice for large datasets, (4) their update parameters are static and cannot capture the dynamics of dependencies. Deep learning- based methods [6, 30, 27]. the other hand, potentially are able address subset of the above limitations, while having their own draw backs (discussed in $1). ). In this section, we start with continuous SSMs due to their connection to both classical methods [12, 13] and recent breakthrough in deep learning [34, 33). We then discuss our contributions on how to take the advantages of the best of both worlds, addressing all the abovementioned limitations.
Discrete 2D SSM. We use 2-dimensional SSMs, introduced in Equation 1-3, to model multivariate time series, where the first axis corresponds to the time dimension and the second axis is the variates. Accordingly, each state is a function of both time and variates. The first stage is transform the continuous form of 2D SSMs to discrete form. Given the step size A and 42, which represent the resolution of the imput along the axes, discrete form of the input is defined as X1, x x(kA1, (A2). Using Zero-Order Hold (ZOH) method, we can discretize the input as (see Appendix C for details):
( (2) 1KttLe))
A1
A2
B1 B2
Xx,e+1
=
A3
A
hig
(2)
+
xx+1,e
(5)
AAi' (A1 - 1())
where A; exp A 111 A.) for i = 1,2,3,4, B
A2 (A2 - 1) B1 and B₂
Ag 1 (Ag (1) A (A1 1 B!
2
Note that this formulation can also be viewed as the modification of
X x
the discrete Roesser's SSM model [35] when we add lag of I in the inputs
This modifi-
cation, however, misses the discretization step, which is an important step in our model. We later use the discretization step to (1) empower the model to select (resp. filter) relevant (resp. irrelevant) information, (2) adaptively adjust the resolution of the method, capturing seasonal patterns.
From now on, we use (resp. v) to refer to the index along the time (resp. variate) dimension. Therefore, for the sake of simplicity, we reformulate Equation 5 as follows:
hu,41 A + Agh Bixo,141,
(6)
h (2) Agh + + B2x,+1,6
141.1
(7)
Yu, C. C2hee
(8)
where A₁, A2, Ag, A B, E RNI, and C₁, C2 E RIXN are parameters of the model, (1) (()) E Nxd are hidden states, and x E 1Xddd xd is the input. In this formulation, intuitively, (. is the hidden state that carries cross-time information (each state depends on its previous time stamp but within the same variate), where A and Ag control the emphasis past cross-time and cross-variate information, respectively. Similarly, h is the hidden state that carries cross-variate information (each state depends on other variates but with the same time stamp). Later in this section, we discuss to modify the model to bi-directional setting along the variate dimension, to enhance information flow along this non-causal dimension.
Interpretation of Discretization. Time series data are often sampled from an underlying continuous process [39, 40]. In these cases, variable A the discretization of the time axis can be interpreted as resolution or the sampling rate from the underlying continuous data. However, discretization along

---

2D SSM For Time Series
Multivariate Closed-Loop
Ash B
Yu,t
(2)
Borla
Agh Ach Bax
D
D2
Up
A
AgAne AA B
Convolution Form
A
A
B
Aa
A
B
ΣΣ (c [ + C
You
u..e
Data-Dependee Parameters
P2
Ps
Q1
@ 96
0: 0p
0
B, Linearp, (ar),
Linearc, (x)
0
PB
(44 0
O
1
8 06)
A Softplus (Lineara, (x))
* 0)
* (a r) Parallel Scan
Different forms of Chimera. (Top-Leff) Chimera has recurrence form (bi-directional along the variates), which also be computed global convolution in training. (Top-Right) In forecasting, we present the multivariate closed- loop improve the performance for long horizons. (Bottom) Using data-dependent parameters, Chimera training can be done as parallel 2D scan.
Figure 2:
the variate axis, which is discrete by its nature, or when working directly with discrete data [41] is an unintuitive process, and raise questions about its significance. The discretization step in 1D SSMs has deep connections to gating mechanisms of RNNS [42, 33), automatically ensures that the model normalized [43], and results in desirable properties such resolution invariance [32].
Proposition 3.1. The 2D discrete SSM introduced in Equation 6-8 with parameters ([A:), (B.), (C.). (C.,,AA (A2) evolves rate k (resp. E) times as fast the 2D discrete SSM with parameters (1 (At), [B.), (C,),A, Sh,eA, (22) (resp. ((Ai), (B:), (A,),kA,,Aa))
Accordingly, parameters A can be viewed as the controller of the length of dependencies that the model captures. That is, based on the above result, we see the discretization along the time axis as the setting of the resolution sampling rate: while small A₁ can capture long-term progression, larger Å captures seasonal patterns. For now, we see the discretization along the variate axis as a mechanism similar to gating in RNNs [44, 33], where A₂ controls the length of the model context. Larger values of A2 means less context window, ignoring other variates, while smaller values of A₂ means more emphesis we discuss making A₂ as the function of the input, resulting in a selection mechanism that filters irrelevant variates.
the dependencies of variates. Later, inspired by Gu and Dao [33],
Structure of Transition Matrices. For Chimera to be expressive and able to recover autoregressive
process, hidden states (1) should carry information about past time stamps. While making all the parameters in A; learnable allows the model to learn any arbitrary structure for A,, previous studies show that this is not possible unless the structure of transition matrices are restricted [45, 46). To this end, inspired by Zhang et al. [28] that argue that companion matrices are effective capture the dependencies along the time dimension, we restrict A₃ and A matrices to have companion structure (see Appendix D for the details). Not only this formulation is shown to be effective for capturing dependencies along the time dimension [28] (also see Theorem 3.4), but also can help us to compute the power of A₁ and Ag faster in the convolutional form, discussed by Zhang et al. [28). Also, for A3 and A4, we observe that even a simpler structure of diagonal matrices effective to fuse information along the variate dimension. Not only these simple structured matrices make the training of the model faster, but they also are proven to be effective [45].
Bi-Directionalityy The causal nature the 2D SSM result limited information flow along the variate dimension variate are not ordered. To overcome this challenge, inspired by the bi-directional ID SSMs [47, 48, 1]. we use two different modules for forward and backward pass along the variate dimension. The details of formulation is in Appendix E.
Convolution Form. As discussed by Baron et al. [49], similar to ID SSMs [34], the data-independent formulation of 2D-SSMs can be viewed as a convolution with a kernel K. This formulation not only results in faster training by providing the ability of parallel processing, but it also connect Chimera
5

---

with very recent studies of modern ianolution-basee architecture for time series [9]. Applying the recurrent rules in Equation 6-8, we can write the output as:
Yott Σ Σ
(C,K Ri + C2K (, Xz,
(9)
where kernels K (7)
- (E di A A5 A Bpp and P(r) is the partitioning of the 8,4
paths from the starting point to (ě, f) for 7 E (1.2). As discussed by Baron et al. [49], if the power of A,S are given and cached, calculating the partitioning of all paths can be done very efficiently (near-linearly) as it the generalization of pascal triangle. To calculate the power of A,, note that we diagonal matrices as the structure of A3, and A4, and so computing their powers is very fast. On the other hand, for A and A₂ with companion structures, we can use sparse matrix multiplication, which results in linear complexity in terms of the sequence length.
Data-Dependent Parameters. As discussed earlier, parameters Å₃ and A controls the emphasis on past cross-time and cross-variate information. Similarly, parameters Å₁ and B₁ controls the emphasis the current input and historical data. Since these parameters are data-independent, one can interpret them as a global feature of the system. In complex systems (e.g., human neural activity), however, the emphasis depends on the current input, requiring these parameters to be the function of the input (see $4.1). The input-dependency of parameters allows the model to select relevant and filter irrelevant information for each input data, providing similar mechanism transformers [33]. Additionally, as we argue earlier, depending on the data, the model needs to adaptively learn if mixing information along the variates useful. Making parameters input-dependent further overcomes this challenge and lets our model to mix relevant and filter irrelevant variates for the modeling of variate of interest. One of our main technical contributions is to let B,, C₁, and A, for i {1.2) be the function of the input x,,t- This input- pept-deeendent 2D SSM, unfortunately, does not have the convolution form, limiting the scalability and efficiency of the training. We overcome this challenge by computing the model recurrently with a new 2D scan.
2D Selective Scan. Inspired by the scanning in ID SSMs [50, 33J, we present an algorithm to decrease the sequential steps that are required to calculate hidden states. Given p. q, each of which with elements, we first define operation * as: (0 is matrix-matrix and @ is matrix-vector multiplication)
p * g = P1
P2
P3
91
Q1 ① O p4
Q2 ① P2
41 2 P3 +
42 @
P6 + 93
P4
P5
P6
94
46
qa
q4
Ps
+ 45
P6 96
The proofs of the next two theorems are in Appendix G.
Theorem 3.2. Operator & is associative: Given p, q, and r, we have: (p q) * p (q r).
Theorem 3.3. 2D SSM recurrence can be done in parallel using parallel prefix sum algorithms with associative operator *, when fixing the variate.
3.2 New Variant of 2D SSM: 2D Mamba
Figure 2 (Top-Left) shows the recurrence form of our 2D SSM. Each small square is state of the system, i.e., the state of a variate at a certain time stamp. 2D SSM considers two hidden states for each state (represented by colors: light red and blue), encoding the information along the time (red) and variate (blue), respectively. Furthermore, each arrow represents a transition matrix A, that decides to how information need to be fused. In this section, discuss spacial instance of our 2D SSM by limiting its parameters.
2D Mamba. We let A₂ = As = 0 in our 2D SSM. The resulting model equivalent to:
h = Anh! + Bix,,
(10)
( = - B2x++1.
(11)
Ve.: = C + Cyh
(12)
[A] [AT' (A, (A) (I) and B₂
where A
exp (A A1),), Ag exp (A)))) B
0
0
A (A4 1) B (2)
This formulation with data-dependent parameters, is equivalent to using two

---

S6 blocks [33] each of which along dimension. Notably, these two S6 blocks are not separate as the output Yv,e based on both hidden states hig and he capturing 2D inductive bias.
3.3 Chimera Neural Architecture
In this section, we use a stack of our 2D SSMs, with non-linearity in between, to enhance the expressive power and capabilities of the abovementioned 2D SSM. To this end, similar to deep SSM models [28], we allow all parameters to be learnable and in each layer we use multiple 2D SSMs, each of which with its own responsibility, Also, in the data-dependent variant of Chimera, we let parameters B; C, and A, for i E [1,2] be the function of the input x:
(13) Chimera follows the commonly used decomposition time series, and decomposes them into trend components and seasonal patterns. it, however, uses special traits of 2D SSM to capture these terms.
B: = Linearp (z), C; = Linearc, (x), A Softplus (Linear (x)).
Seasonal Patterns. To capture the multi-resolution seasonal patterns, we take advantage of the discretization process. Proposition 3.1 states that if x(u.t) H y(e,t) with parameters ([A,). [B,}, A, cordingly, we use 2D-SSM(.) module with a separate learnable A, that responsible to learn best resolution to capture seasonal patterns. Another interpretation for this module is based on SAR(p, 9.8) (Equation 4). this case, A, aims to learn a proper parameter s to capture seasonal patterns. Since we expect the resolution before and after this module matches, we add additional re-discretization module (a simple linear layer), after this module.
then x(v, kt) + y(u, kt) with
(C.) kA,, Ac-
Trend Components. The second module of Chimera, 2D-SSM, (.) simply uses a sequence of multiple 2D SSMs to learn trend components, Proper combination of the outputs of this and the previous modules can capture both seasonal and trend components.
Both Modules Together. We followed previous studies [51] and consider residual connection modeling for learning trend and seasonal patterns. Given input data Xo X, and & 0,... C.
have:
X 2D- (x:).
(14)
X4+1 Re- DeeDiscretizztton
2DD-SSM, (X, (--xet1))
(15)
Figure illustrate the architecture of Chimera. Due to the ability of our 2D SSM to recover smoothing techniques (see Theorem 3.4), this combination of modules for trend and seasonal patterns can be
viewed as generalization of traditional methods that use moving average with residual connection to model seasonality [51].
Gating with Linear Mapping. Inspired by the success of gated recurrent and SSM-based models [52, 331, head of fully connected layer with Swish [53], resulting in SwiGLU variant [54]. While we validate the significance of this head, this
Closed-Loop 2D SSM Decoder. To enhance the generalizability and the ability of our model for longer-horizon, we extend the closed-loop decoder module [28], which is similar to autoregression, to multivariate time series. We use distinct processes for the inputs and outputs, using additional matrices D and D in each decoder 2D SSM, model future input time-steps explicitly:
No,t = h
(1) + C2
(16)
Up,t = Dj D + (2)
(17)
where Up is the next input and Yo,t is the output. Note that the other parts (recurrence) are the same Equation Figure 2 illustrate the architecture of closed-loop 2D SSM.
3.4 Theoretical Justification
In this section, we provide some theoretical evidences for the performance of Chimera. These results mostly revisiting the theorems by Zhang et al. [28] and Baron et al. [49], and extending them
7

---

for Chimera. In the first theorem, we show that Chimera recovers several classic methods, and pre-processing steps as it can recover SpaceTime [28] and additionally because of its design, it can recover SARIMA [15]:
Theorem 3.4. Chimera can represent seasonal autoregressive process, SARIMA [15], Space- Time [28], and so ARIMA [14], and exponential smoothing [16].
Theorem 3.5. Chimera can represent S4nd [32], TSM2 [31], and TSMixer [30].
Next theorem compares the expressiveness of Chimera with some existing 2D deep SSMs. Since Chimera can recover 2DSSM [49], it can express full-rank kernels with a constant number of parameters:
Theorem 3.6. Similar to 2DSSM [49], Chimera can express full-rank kernels with O (1) parameters, while existing deep SSMs [32, 31] require (N) parameters to express N -rank kernels.
Experiments
Goals and Baselines. We evaluate Chimera on a wide range of time series tasks. In $ 4.1 we compare Chimera with the state-of-the ahr general multivariate time series models [8, 9, 18, 20, 21, 24, 26, 27, 31, 55-57] on long-term forecasting and classification tasks. In the next part, we test the performance of Chimera in short-term forecasting. In & 4.1 perform classify seen images, which requires capturing complex dynamic dependencies of variates, to test the ability of Chimera in capturing cross-variate information and the significance of data-dependency. In
study human neural activity to
$ 4.2, we evaluate the significance of the Chimera's 's components by performing ablation studies. In 4.2, we evaluate whether the superior performance Chimera coincide with its efficiency. Finally, we test the Chimera's is generalizability on unseen variates and further evaluate its ability to filter irrelevant context in $ 4.3. The details and additional experiments are in Appendix I.
Table Average Performance on long-term forecasting task. The first and second results are highlighted red (bold) and orange (underline). Full results are reported in Appendix I.
Chimera
TSM2
Simba [2024]
TCN
iTranstommer
RLinear
PatchTST
Crossformer
TDDE
TimesNet
DLineur [2023]
(ours)
[2024]
[2024]
[2024]
[2023]
[2023]
[2023]
[2023]
[2023]
MSE MAR MSE MAP MSE MAF
MSE MAF MSF MAR
MSE MAR
MSR MAR MSE MAR MSR MAR MSR MAE MSE MAR
ETTml
0.355
0.381
0.36L
0.383 0.396
0.396/0.351
0,381
[0:407
0.410
00:414
0.407
00387
0.400
0,513 0.496
0.419
0.419/0.400 0 0.406/0.405 0.407
ETTm2
0.252
0.267
0.271
0.253
0.286
0.291
0.350
ETTh1
0.408 0:425
0.403
0.441 0.432
0.404
0.420
(0.454
0.447
0.446
0.434
0.469
0.454 0.529 0.522
0.541
0.507 0.458
0.456
0.456 0.452
TT
0.321 0.377
0.333
0,361 0.391
0.332
0.379
0.,83
0.407
0.374 0.398 0..387 0.407 0.942 0.684
0.611
0.550
0.414 0.427 0.559 0.515
ECL
0.154 0.249
0.169
0.274 0.185
0.156 0.253
10.178
0.270
0.219 0.298 00.205 0.290 0.244 0.334
0.251
0.344
0.199
0.295
0,212 0.300
Exchange
0.311
0.31. 0.358
0335880.44
0.443
00,303
0.3665
(0.360
0.403
0.378 0.417 00.477 0.404 0.940 0.707 0.370
0.416
0.41. 0.416
0.44
0.354 0.414
Traflic
0,403
0.420
0.493
0.291
00.398
0.270
0.4288
0,282
0.66
0.626
0.378 00.48
0.304
0.760
0.620
0.625
Weather
0.219
0.258...339 0.239
0.258
0.255 0.280
0
0.224
0.2641
0.258
0,278
00.773
0.291
(0.259 0.281
0.259
0.315
0.271
0.320
0.259
0.287
0.265 0.317
Cowot
0
0
0
0
0
4.1 Main Results: Classification and Forecasting
Long- Term Forecasting. We perform experiments in long- long-term term forecasting task on benchmark datasets [6]. Table reports the average of results over different horizons (for the results of each see Table 8). Chimera shows outstanding performance, achieving the best or the second best results in all the datasets and outperforms baselines in 5 out of 8 benchmarks. Notably, it surpasses extensively studied MLP-based and Transformer-hase models while being more efficient (see Table 3, Figure and Appendix 1), providing a better balance of performance and efficiency. It further significantly outperforms recurrent models, including very recent Mamba-bases architectures [31, 57], unleashing the potential of classical models, SSMs, when are carefully designed in deep learning settings.
Classification and Anomaly Detection, We evaluate the performance of Chimera in ECG classi- fication on PTB-XL dataset [7] (see Table 2), speech classification [39] (Table 3). 10 multivariate datasets from UEA Time Series Classification Archive [60] (see Figure 3 and Table 10), and anomaly detection tasks on five widely-used benchmarks: SMD [10], SWaT [61], PSM [62] and SMAP [63] (see Figure 3 and Table 11). For each benchmark, we use the state-of-the-ee methods that are
8

---

applicable to the task as the baselines. Table 2 reports the performance of Chimera and baselines on ECG classification tasks. Chimera outperforms all the baselines in 4/6 tasks, while achieving the second best results on the remaining tasks. Since these tasks are univariate time series, we attribute the outstanding performance of Chimera, specifically compared to SpaceTime [28], to its ability of capturing seasonal patterns and its input-dependent parameters, resulting in dynamically learn dependencies.
Table 3 reports the results on speech audio classification task, which require long-range modeling of time series. Due to the length of the sequence (16K), LSSL [34] and Transformer [17] has out of memory (0OM) issue, showing the efficiency of Chimera compared to alternative backbones.
Finally, we report the summary of the results in multivariate time series classification and anomaly detection tasks in Figure 3. The full list of results can be found in Table 10 and Table 11. Chimera shows outstanding performance, achieving highest average accuracy and F1 score in classification and anomaly detection tasks even compared to very recent and state-of-the-art methods [8, 9].
Table 2: ECG statement classification on PTB-XI BTB (100 Hz version).
Table 3: Speech classification.
Tasks
All
Diag
Sub-diag
Super-diag
Form
Rhythm
Method
Acc. (%)
Chimera
0.941
0.947
0.935
0.930
0.901
0.975
Chimera
98.40
SpuceTime [28]
0.936
0.941
0.933
0.929
0.883
0.967
Space'Time
97.29
S4 [34]
0.938
0.939
0,929
0,931
0.895
0,977
Inception
0.925
0.931
0.930
0.921
0.899
0.953
S4
98.32
XRN-101
0.925
0.937
0,929
0.928
0.896
0.957
L.SS..
OOM
LSTM
0.907
0.927
0,928
0.927
0.851
0.953
WaveGan-D
96.25
Transformer
0.857
0.876
0.882
0.887
0.771
0.831
Transformer
OOM
Short-Term Forecasting, Our evaluation on short-term forecasting tasks on M4 benchmark datasets [64] reports in Table 4 (Full list in Table 9), which also shows the superior performance of Chimera compared to baselines.
Table 4: Short-term forecasting task on the M4 dataset. Full results reported in Appendix I.
Models
Chimerra ModemTTNN PaichTST TimaNNet N-HITS x BEATS
N-HTS
ETS.
LighTTS DLinear FED- Statinary Autux Pyn Ins
Res
LSTM
(ours)
[2024]
[2023]
[2023]
[2022]
[2019]
[2022] [2022] [2023] [2022] [2022] (2021) (2021) [2021] [2020] [1997]
Weightd
Andtago
SMAPE
11.618
11,698
11,807
11,829
11.927
11885
14.718
13.525
13,639
12,840
12,780
12,909
16.987 14.086 18 160.00033
1.556
1,585
1.599
2,108
1.701
1.756
25.788
OWA
0.827
0.838
0.851
0.851
0.861
0.855
1.122
1051
1,051
0.918
0.930
0.939
1,480
1,230
1.775 12.642
Case Study of Brain Activity. Input dependency is a must to capture the dynamic of dependencies. To support this claim, we use BVFC [2] (multivariate time series only), which aim to classify seen images by its corresponding brain activity response. This task, requires focusing more on the dependencies of brain units and their responses rather than the actual time series. Also, since each window corresponds to specific image, the model needs to capture the dependencies based the current window, requiring to be input-dependent. Results are reported in Table 5. Chimera significantly outperforms all the baselines including Chimera but without data-dependent parameters (convolution form). methods face
Due to the large number of brain units, i.e., 9K, in the first dataset, transformer- OOM issue. However, they are also data-dependent and so shows the second best results in second and third datasets. This results support the significance of data-dependency in Chimera.
4.2 Ablation Study and Efficiency
To evaluate the significance of the Chimera's 's design, we perform ablation studies and remove one of the components at each time, keeping other parts unchanged. Table 6 reports the results. The first row reports the Chimera's performance, while row uses unidirectional recurrence along the variate dimension, row 3 removes the gating mechanism, row 4 uses convolution form (data-independent) and row removes the module for seasonal patterns. The results show that all the components of Chimera contributes to its performance.

---

Table 6: Ablation study the Chimera's design.
Table 5: Image classification by brain activity (Acc. %).
Method
ETThi
ETTml
ETT12
Methoo
Chineera
(surs)
Chimera (Ind.) (muns)
SpaceTime [2023]
[3022]
S4
ITnss
[2024]
Trans. [2017]
DLane [3023]
Chimare
0,408
MSE
0.425
MAE
0.355
MSE
0.383
MAE
0.321
MSE
0.377
MAE
BVPC (MK)
DOM
Uni dinectional
0.416
0.430
0.368
0.397
0.334
0.386
BVPC (1K)
18.18
Garag
0.424
0.395
0,382
(400)
Inpu: indepndent
Classification Performance
Anomaly Detection Performance
350
300
Transfomee
200
secrts
150
SpaceTime
Fara/ormars
Accumey
Linser
Hecummant
Convolational
Scare
70
Dura
250
500
705
Sequence Length
1005
1250
1200
1755
1000
Figure 3:
Classification and anomaly detection performance. Full list with additional baselines is Appendix
Figure 4: Wall-clock scaling,
Length of Time Series. We perform experiments on the effect of the sequence length the efficiency of Chimera and baselines. The results are reported in Figure 4. Chimera scales linearly with respect the sequence length and has smoother scaling than S4 [34] and Transformers [17]. These results also highlight the significance of our algorithm that uses 2D parallel scans for training Chimera. This algorithm results x4 faster training, which very closed the convolutional format without data dependency. Chimera also has close running time to SpaceTTime [28], which has 1D recurrent.
4.3 Selection Mechanism Along Time and Variate
Variate Generalization. We argue that the data-dependency with discretization allows the model filter the irrelevant context based the input, resulting in more generalizability. Inspired by Liu et al. [27], we train our model (and baseline) on 20% of variates and evaluate its generalizability to unseen variates. The results reported in Figure 5. Chimera has par generalizability compared to Transformers (when applied along the variate dimension), which we attributes to its data-dependent parameters Chimera with convolution form performs poorly on unseen variates.
Context Filtering. Increasing the lookback length does not necessarily result in better performance for Transformers [27]. Due to the selection mechanism of Chimera, we expect it to filter irrelevant information and monotonically performs better. Figure 6 reports the Chimera's 's performance (w/ and w/o data-dependency) and transformer-basse baselines [6. 22] while varying the lookback length. Chimera due its selection mechanism monotonically performs better with increasing the lookback.
Taffcc
C10
Trffic
36
V10
0.0
94 A/12
11
2
0.6
0.25
0.30
0.5
0.4
0.15
48
95
192
335
720
336
720
00
Figure Effect of lookback length.
Figure 5: Selection results in generalization variates.
5
Conclusion
This paper presents Chimera, three-headed 2 2-dimensional SSM model with provably high expressive power. Chimera is based on 2D SSMs with careful design of parameters that allows it to dynamically and simultaneously capture the dependencies along both time and variate dimensions. We provide different views of our 2D SSM for efficient training, and present data-dependent formulation with a fast implementation using 2D seans. Our experimental and theoretical results support the effectiveness and efficiency of Chimera in a wide range of tasks.
10

---

Acknowledgments and Disclosure of Funding
This material is based upon work supported by the National Science Foundation under Award No. 2306556. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
References
[1] Ali Behrouz and Farnoosh Hashemi. Brain-mambar Encoding brain activity via selective state space models. Conference on Health, Inference, and Learning, pages 233-250. PMLR, 2024,
[2] Ali Behrouz, Parsa Delavari, and Farnoosh Hashemi. Unsupervised representation learning of brain activity via bridging voxel activity and functional connectivity. In International conference on machine learning (ICML), 2024.
[3] Plamen Ch Ivanov, Luis A Nunes Amaral, Ary L Goldberger, Shlomo Havlin, Michael G Rosenblum, Zbigniew R Struzik, and H Eugene Stanley. Multifractality in human heartbeat dynamics. Nature, 399(6735):461-465 1999.
[4] Kelum Gajamannage, Yonggi Park, and Dilhani Jayathilake. Real-time forecasting of time se- ries in financial markets using sequentially trained dual-Istms. Expert Systems with Applications, 223: 19879, 2023.
[5] Steve Pineus and Rudolf E Kalman. Irregularity, volatility, risk, and financial market time series. Proceedings of the National Academy of Sciences, 101(38):13709-13777 -17711999911144--- 2004.
[6] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11 11106-11115, 2021.
[7] Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojeiech Samek, and Tobias Schaeffter. Ptb-xl, large publicly available clectrocardiography dataset. Scientific data, 7(1):1-15, 2020.
[8] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023. URL httpss//openrevion net/forum?
id Ju3334400
[9] Donghao Luo and Xue Wang. ModernTNN: A modern pure convolution structure for general time series analysis. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttpps://ppnttt /ttp:///ppeeeeeeeeeeeeeeee net/ formmordmideo @rvuniidvvpJMeerXHHH
[10] Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2828 2837, 2019.
[11] Ailing Zeng. Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI, 2023.
[12] Andrew C Harvey. Forecasting, structural time series models and the kalman filter. Cambridge university press, 1990.
[13] Masanao Aoki. State space modeling of time series. Springer Science & Business Media, 2013.
[14] David Bartholomew. Time series analysis forecasting and control., 1971.
[15] Michael Bender and Slobodan Simonovic. Time-series modeling for long-range stream-flow forecasting, Journal of Water Resources Planning and Management, 82066)857-870, 1994.
[16] Peter R Winters. Forecasting sales by exponentially weighted moving averages. Management science, 6(3):324-344 1960.

---

[17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[18) Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: survey, Philosoph- ical Transactions of the Royal Society A, 37902194))222000009, 2021.
[19] Zonglei Chen, Minbo Ma, Tianrui Li, Hongjun Wang, and Chongshou Li. Long sequence time-series forecasting with deep learning: A survey. Information Fusion, 97:101819, 2023.
[20] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Essfrrmrr: Ex- ponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01388, 2022.
[21] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34:22419 224300, 2021.
[22] Haixu Wu, Jialong Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Flowformer: Lineariz- ing transformers with conservation flows. In ICML, 2022.
[23] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations, 2021.
[24] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting, In International conference on machine learning, pages 27268-27286. PMLR, 2022.
[25] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2020.
[26] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The eleventh international conference on learning representations, 2023.
[27] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreevie.. net//forum?id-JePAAAB
[28] Michael Zhang, Khaled Kamal Saab, Michael Poli, Tri Dao, Karan Goel, and Christopher Re. Effectively modeling time series with simple discrete state spaces. The Eleventh International Conference on Learning Representations, 2023. URL htttps::/openreeeeee wwwprrviiieeeeeeorrrneeeee id=2EpjkjzdCAa.
[29] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023. URL id=JbdcOvTOcol.
wtwpeeiiwwweeeeeenneeeeeee net/forum?
[30] Si-An Chen, Chun-Liang Li, Nate Yoder, Serean O Arik, and Tomas Pfister. Tsmixer: An all-mlp architecture for time series forecasting, arXiv preprint arXiv:2303.060S. 2023.
[31] Ali Behrouz, Michele Santacatterina, and Ramin Zabih. Mambamixer: Efficient selective state space models with dual token and channel selection. arXiv preprint 2.0.::24433.98888 2024.
[32] Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. Advances in neural information processing systems, 35:2846-2220 -2661, 2022.
[33] Albert Gu and Tri Dao. Mamba: I Linear-time sequence modeling with selective state spaces. arXiv preprint ar arXiv:2312.007522 2023.
12

---

[34] Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations, 2022. URL https:
/ //openreview. wttprrrviieeeee/ffrrroe
[35] Sun-Yuan Kung, Bernard C Levy, Martin Morf, and Thomas Kailath. New results in 2-d systems theory, part 2-d state-space reaiisorralization and the notions of controllability, observability, and minimality. Proceedings of the IEEE, 65(6):945-99) 1977.
[36] Ettore Fornasini and Giovanni Marchesini. Doubly-indexed dynamical systems: State-space models and structural properties. Mathematical systems theory, 12(1):59-72, 1978.
[37] Rikus Eising. Realization and stabilization of 2-d systems. IEEE Transactions on Automatic Control, 23(5):793-799, 1978.
[38] Ts Hinamoto. Realizations of a state-space model from two-dimensional input-output map. IEEE Transactions on Circuits and Systems, 27(1):36-44, 1980.
[39] Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv: arXiv:1804.03209 2018.
[40] Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri- PasriiPashhkmm, and Chris I Baker. Things-data, multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. Elife, 12:802 2023.
[41] Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
[42] Corentin Tallec and Yann Ollivier. Can recurrent neural networks warp time? In International Conference on Learning Representations, 2018. URL https://openreviae net/forum? id=$JcKhk-Ab.
[43] Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Re. How to train your HIPPO: State space models with generalized orthogonal basis projections. In International Conference on Learning Representations, 2023. URL httttp:://ppereveee wwweeanrrveeeeee/fforr.rr net/forum? id=k1K170Q3KBB
[44] Albert Gu, Caglar Gulcehre, Thomas Paine, Matt Hoffman, and Razvan Pascanu. Improving the gating mechanism of recurrent neural networks. In International Conference on Machine Learning, pages 3800-3809. PMLR, 2020.
[45] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreral nwt met:/oorumiddyyJJ711...
[46] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572-585 585, 2021.
[47] Junxiong Wang. Jing Nathan Yan, Albert Gu, and Alexander Rush. Pretraining without at- tention. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguisties: EMNLP 2023, pages 58 69, Singapore, December 2023. As- sociation for Computational Linguistics. doi: 10.18653/v1/202. wwtp://..//////iiii....... findings- e.0gs...mmpp.....
URL
[48] Ali Behrouz and Farnoosh Hashemi. Graph mamba: Towards learning graphs with state space models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 119-130, 2024.
[49] Ethan Baron, Itamar Zimerman, and Lior Wolf. A 2-dimensiooal state space layer for spatial inductive bias. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttpnr/ve///eeeeiieeeee www.://ooummdddicigggoaaoae
13

---

[50] Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations. 2023. URL https: /ttp////ppe/eeeeee//////// wtwppeeewie...ee/effrr//o/
[51] William Toner and Luke Darlow. An analysis of linear time series forecasting models. Interna- lional conference on machine learning (ICML), 2024.
[52] Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. Advances in Neural Information Processing Systems, 36, 2023.
[53] Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions, arXiv preprint arXiv: 1710.05941 2017.
[54] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[55] Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan K Mathur, Rajat Sen, and Rose Yu. Long- term forecasting with tiDE: Time-series dense encoder. Transactions on Machine Learning Re- search, 2023. ISSN 2835-8856. URL https: http:::///ppereee /pwprevepreeeeeeeeee net/ htt///oonidddddaaaaaaaa
[56] Minhao Liu, Ailing Zeng. Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems, 35:5816-5822 2022.
[57] Badri N. Patro and Vijay S. Agneeswaran. Simba: Simplified mamba-base architecture for vision and multivariate time series, 2024.
[58) Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv preprint arXiv:2305. 10721, 2023.
[59] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128, 2023.
[60] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv: 1810000075 2018.
[61] Aditya P Mathur and Nils Ole Tippenhauer. Swat: A water treatment testbed for research and training on ics security. In 2016 international workshop on cyber-physical systems for smart water networks (CySWater), pages 31-36. IEEE, 2016.
[62] Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 2485-22822 2494, 2021.
[63] Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. Detecting spacecraft anomalies using Istms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 387-395, 2018
[64] Rakshitha Godahewa, Christoph Bergmeir, Geoffrey Webb, Rob Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive. arXiv preprint arXiv:2 2021.
[65] Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201. /2886, 2022.
[66] Boris Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting, ICLR, 2019.
14

---

[67] T. Zhang, Yizhuo Zhang, Wei Cao, J. Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling sampling-oriented -rreented mip structures, arXiv preprini arXiv:2207.01 186, 2022.
[68] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time series forecasting. In NeurIPS, 2022.
[69] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Comput., 1997.
[70] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474 1474-1487, 2020.
[71] Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. In International Conference on Learning Representations, 2018. URL htttps://opeerreiiie net/forum wwwooorrmmiiiduduu HyNNuul1C-
[72] George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series (Applied Statisties), 17(2):91-09, 1968.
[73] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555,
2014.
[74] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International journal of forecasting, 36(3): 1181-1191, 2020.
[75] Songlin Yang. Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In International conference on machine learning (ICML), 2024.
[76] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. In International conference on machine learming, pages 5156-5165. PMLR, 2020.
[77] Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers secretly fast weight programmers. In International Conference on Machine Learning, pages 9355-9366. PMLR, 2021.
[78] Jun Ma, Feifei Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722 2024.
[79] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu, Vmamba: Visual state space model. arXiv preprint 2rXiv:2401.10166 2024.
[80] Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234 2024.
[81] Yuxin Jia, Youfang Lin, Xinyan Hao, Yan Lin, Shengnan Guo, and Huaiyu Wan. WITRAN: Water-wave information transmission and recurrent acceleration network for long-range time series forecasting. In Thirty Conference on Neural Information Processing Systems, 2023. URL https::///// http:://ppeervvieeee net/ wwt.:/nee/t//ommemeeeeee /ttp://mmmddyyobb/tttaaaaa
[82] Romain Ilbert, Ambroise Odonnat, Vasilii Feofanov, Aladin Virmaux, Giuseppe Paolo. Themis Palpanas, and levgen Redko. Unlocking the potential of transformers time series fore- casting with sharpness-aware minimization and channel-wise attention. arXiv preprint arXiv:2402. 10/98, 2024.
[83] Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems, 35:12677-12699 2022.
15

---

[84] Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long, Koopa: Learning non non-stationary time series dynamics with koopman predictors. Advances Neural Information Processing Systems, 36, 2023.
[85] Rui Wang, Yihe Dong. Sercan O Arik, and Rose Yu. Koopman neural operator forecaster for time-series with temporal distributional shifts. In The Eleventh International Conference on Learning Representations, 2023.
[86] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 35:9881-958 2022.
[87] Chi-Tsong Chen. Linear system theory and design. Saunders college publishing, 1984.
[88] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR, 2018.
[89] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang. and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, In NeurlPS, 2019.
[90] Jven-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable represen- tation learning for multivariate time series. In NeurIPS, 2019.
[91] Jiehui Xu, Haixu Wa, Jianmin Wang, and Mingsheng Long, Anomaly transformer: Time series anomaly detection with association discrepancy. In ICLR, 2021.
16

---

A Background
A.1 1D Space State Models
1D Space State Models (SSMs) are linear time-invariant systems that map input sequence x(t) RL >> y(t) E Th' [13]. SSMs use a latent state h(t) TR transition parameter A E R and
projection parameters B E I Nx1, C E RIxN to model the input and output as:
(1(1)= = A h(O) + BB((),
v(() = C ((().
(18)
Most existing SSMs [34, 33, 31], first discretize the signals A, B, and C. That is, using a parameter A and zero-order hold, the discretized formulation is defined
h A hb B Et,
1/- C hts
(19)
where = exp (AA) and = (AA)- (exp ( n) AB. [70] show that discrete SSMs can be interpreted both convolutions and recurrent networks: i...
K = (CB, CAB, .............. CB)
y == x * K, which makes their training and inference very efficient as a convolution and recurrent model, respec- tively.
(20)
A.2 Data Dependency
Above discrete SSMs are based on data-independent parameters. That is, parameters A, A, B, and C' are time invariant and are the same for any input. Gu and Dao [33] argue that this time invariance has the cost of limiting SSMs effectiveness in compressing context into a smaller state [33]. To overcome this challenge, they present selective SSMs (S6) block that effectively selects relevant context by enabling dependence of the parameters B, C, and A on the input Tz, i.e.:
B, Linearg
(21)
C Linearc(a;)
(22)
A Softplus (Linear (x:))
(23)
where Linear(-) a linear projection and Softplus(.) = log(1 + exp(-)). This data dependency comes at the cost of efficiency the model cannot be trained convolution. To overcome this challenge, Gu and Dao [33] show that the linear recurrence in Equation can be formulated as an associative scan [71], which accepts efficient parallel algorithms.
B
Additional Related Work
Classical Approach. Modeling time series data a long-standing problem and has attracted much attention during the past 60 years. There have been several mathematical models to capture the time series traits like exponential smoothing[ 16], autoregressive integrated moving average (ARIMA) [14], SARIMA [15], Box-Jenkin method [72], and more recently sate-space models [12. 13]. Despite their more interpretability, these methods usually fail to capture non-linear dependencies and also often require manually analyzing time series features (e.g., trend seasonality), resulting in lack of generalizability.
Recurrent and Deep State Space Models. Another group of relevant studies to ours is deep sequence models. A common class of architectures for sequence modeling are recurrent neural networks such as like GRUs [73], DeepAR [74], LSTMs [69]. The main drawback of RNNs their potential for vanishing/exploding gradients and also their slow training. Recently, linear attention methods with fast training attracted attention [75-77]. Katharopoulos et al. [76] show that these methods have recurrent formulation and can be fast in inference.
Recently, deep state space models have attracted much attention as the alternative of Transformers [17]. due to their fast training and inference [70]. These methods are the combination of traditional SSMs
17

---

with deep neural networks by directly parameterizing the layers of a neural network with multiple linear SSMs, and overcome common recurrent training drawbacks by leveraging the convolutional view of SSMs [70, 45, 46, 34, 50]. Recently, Gu and Dao [33] present new formulation of deep SSMs by allowing the parameters to be the function of inputs. This architecture shows promissing potential in various domains like NLP [33], vision [78, 79, 31], graphs [48], DNA modeling [33, 80].
All the above methods are design for ID data, meaning that the states depends on one variable. There are, however, a few studies that uses 2D SSMs in deep learning settings. S4ND [32] uses continuous signals to model images. These methods not only consider two separate SSM for the axes, but also directly treat the system as a continuous system without discretization step. Furthermore, S4ND has data-inder indeeenddenn parameters. Another similar approach is 2DSSM [49], that models images as discrete signals. That is, the initial SSM model is discrete and again there is a lack of discretization step, which important for time series as we discussed earlier. Also, their method again is based on data-independent parameters. Both $4ND) and 2DSSM can be computed as a convolution. We, however, present new scanning technique for fast training of 2D SSMs, even with input-dependent parameters.
WITRAN [81] 2D RNN approach for univariate time series, which is different from our approaches from different aspects: (I) Goal: The main goal of WITRAN is to use 2D RNN to model univariate time series data. That is, the first dimension corresponds to short-term, and the second dimension corresponds to long-term patterns. On the other hand, our 2D SSM aims to model multivariate time series where the first dimension corresponds time, and the second dimension corresponds to variates. (2) Backbone: WITRAN is based on GSC (LSTM-like cell), which uses non-linear transition. Our 2D SSM based on state space models, is simpler, and uses linear transitions. (3) Data- Data-dependency: WITRAN is based on data-independent parameters, meaning that it uses the same transition process for all time steps. 2D SSM, however, based on data-dependent parameters, which allows it to filter irrelevan/nooisy time steps. (4) Recurrence, Hidden States, and Training: Although both models have recurrent form, WITRAN's recurrence is over extracted short-term and long-term patterns. Our 2D SSM's recurrence, however, is over time and variate dimensions. WITRAN uses only one hidden state for each state of the system, while 2D SSM uses different hidden states for each state of the system, allowing more flexibility to capture cross-vait cross-variate and cross time information.
Other methods. Transformer-basete models have attracted much attention over recent years for multivariate time series forecasting, when modeling the complex relationships of co-variates or along the time dimension is required [24-26, 11, 6, 23, 21, 82, 29J. Several studies have focused on designing more efficient and effective attentions with using special traits of time series [20]. Some other studies have focused on extracting long-term information for better forecasting [29, 83]. In addition transformers. linear models also have shown promising results [8, 30]. For example, Chen et al. [30] present TSMixer, an alI-MLP architecture for time series forecasting, with promising performance. Due to the expressive power of our 2D SSM, these linear methods sometimes can be viewed as a special case of 2D SSMs. Recently, convolution-
models for time series have shown promising results [9]. These methods by using global kernels enhance the global receptive field. Our data-independent formulation of Chimera is connected to this line of work as can be written as a global convolution.
Recently, Koopman-based deep models [84, 85] are designed to capture non-linear dependencies in the data. Existing methods, however, does not consider dynamics in different variates and might miss complex dependencies of variates. Moreover, despite their promising results in the time series forecasting, these approaches can be hard to be parallelizable and might have longer training for large models.
C Details of the Discretization
Given PDE with initial condition /(00, 0) 0:
18

---

0
(1)
(1(0) (10) = (A) (1(0) (10) .22h12) (1)) (() )) + +Bix (10) (11))))))
(24)
0 (2)
(1()) ((2)
(A) (11)) ((1) .A2hla)
(0)))0) + B (10)) ((2))
(25)
0 (1)
(1(0) (2))
(Aghi) (1() (de)) .Aah!
(10 + B2x (100) {(2)
(26)
0
(2)
(1()) x22) (Azh()) (+()) ((2)) .Agh! (1)))() + B2x (10)) E(2)
(27)
2)
over the sampling intervals (, (A: + 1) and [(Ar(2) (e + 1) we have:
(K+1)A
a h()) (10,)(0) de(2)
Oil)
(K+1)A
(Anha) (10), ((0) + +B x(1) x(1) (10), (10) de
(28)
and So:
(k+1)AA0)
TnT (1))) ((e) dl
(K+I)A
(A2h(e) (10) (()) + B (2) x(2) (10) (10)) a)
(29)
k(((1)
Similarly, for the second equation we have:
((+1)A Ø 11) (1(), (1U), d)
01
(1+1)))))
(Agh ((00,,10) + B¹ x(1) (100) (21) 2)
(30)
and so:
(1+1)A)))
Ōt(2) 1122 (1 ,(21) dt(2)
(((())
((+1)A
(A1h( (1()) (100) (()) + B²² )(2)x" x()) (600 ((0)) at
(31)
Next, the integrals can be simplified as:
h0 ((k + 1)A("), (10))
(1+1)AR))
(() n11) (15001),,12)) + KAA
-Asd))))s (())x(1)) x(1 (100) atl) (32)
and
(2) ((k + 1)A
(++1)A
= -c (k:S
Aald)) ((2) x2)) (10), (00) di (33)
AAr())
and similarly for the third and fourth equations we have:
1,02) (10) (E (10,(e+
(++1)A
= =CAGSAH) (((1) ((t2)) +
(A(()))))()))) (1) (10), (())) del)) (34)
19

---

and
(22 (1 (E+ + 1)
((+1)An1))
= - 1100) (At))) +
A B
2 x(2) (fe)) (10) de
(35)
Using ZOH assumption, we have:
(1(1)
Ais ds = A (cALAI")-) -1) -T)
(r(1)
A28 ds (2) ( -T)
(36)
(x(2)
Ass ds = A E Asdr -T)
(37)
Ar
Ass ds A4) ( ( -T)
(38)
Accordingly, the discretized form as follows:
RR41E = A1A (1)
(1)
+ A1))
-1 B 1
(1) (1)
(39)
(2)
Rk41, = AZAda
(2)
+ AA()) (2)
(AAZEE))
B (2) (2)
(40)
RRe+1 = AAAAE
(1)
(1)
R
+ A (3)-1
(AASA1
-1) B X1+H1
(41)
R,8+1 = CALANIO)
(2)
(2) 11 R
+ A (4)
(AAL
- 1) B
x k
(42)
which means that:
A exp (Ap(A
(43)
A exp (A2A1).
(44)
A3 exp (AsA2),
(45)
A. exp (A442),
(46)
(47)
and
A (cA,A, 1) B 11)
B
(48)
A (EAsi) - 1) B 2)
(3) (cAsdz 1) B
B₂
(49)
(1) (cAis: - 1) 1)
D Details of the Structure of Transition Matrices
Definition D.1 (Companion Matrix), A matrix A E NRN has companion form if it can be written
as:
0
0
02
0
1
0
03
A
(50)
ON;
0
1
AN
20

---

These matrices can be decomposed into a shift and low-rank matrix. That is:
0
0
0
Q1
0
0
0
Q1
02
1 0
02
O
0
(A3
0 1
0
0
43
A
+
(51)
"t.
:
ON
ONN
0 1
ON
0
1
O.N
Shift Marrix
Low-rank Matrix
This formulation can help us to compute the power of A faster in the convolutional form, as discussed by Zhang al. [28].
E
Bidirectionality Across Variates
The causal nature the 2D SSM result in limited information flow along the variate dimension as variate not ordered. To overcome this challenge, inspired by the bi-directional ID SSMs [47, 48, 1], we use two different modules for forward and backward pass along the variate dimension:
= Afh
(1) + A!n
Bfx
e,141
0,141 A4h000 + ADA B
(1)
(52)
(2)) = Ag4 + A/h
+
B5xx+1.41
1-11 = AS 0)
(2)
+ +A
B
(53)
YEr c + +c
(54)
YV, = Ch (1)
+ Ch
(55)
Yo,t = y',, yb..
(56)
where AT.AJ A5, AT RNNN BJ, B3 E RNx1, and Cj, C% E IIN parameters of the model, 1.1) .222) Nxd are hidden states, x,, 1xd the input, and E (S.b). Figure
illustrates the bi-directional recurrence process in Chimera. For the sake of simplicity, we continue with unidirectional pass, but adapting them for bi-directional setting is simple as we use two separate blocks, each of which for direction.
F
Time Complexity of 2D-SSM
Let be the length of the time series and be the number of variates. Since for each variate we have O(T") recurrence and we have V variates, the total number of recurrence is O(TV). In each step, have the matrix multiplication of transition matrices A, with the hidden states and also B,s with the input. Accordingly, if we use d as the state dimension, the complexity is OdFTV), Given the fact that d usually a small number, the overall complexity linear with respect to each V and T.
G Theoretical Results
G.1 Proof of Theorem 3.2
In this part, we want to prove that * is associative. This operator is defined as:
P1
q1
O
2 P3
+
2 P6 + 93
p n* 9 =
*
P4
Ps
p6
94
Q5
96
q4
p4
Q5
P5
q4
2 pa
45
P6
+ q6
Accordingly, we have:
(p x 4) xr=
91 C P1
42 ④ P2
91
X
P3
92
P6
43
*
T2
T3
(57)
④ P4
45
P5
44
P3
+ 8
P6
+ 46
F4
rs
T6
21

---

Table Dataset descriptions. The dataset size is organized in (Train, Validation, Test).
Tasks
Dataset
Dim
Series Length
Dataset Size
Information (Frequeney)
ETTml, ETTm2
7
(96. 192. 336, 720)
(34465, 11221, 11521)
Electricit (15 mins)
ETThl, ETTh2
7
(96. 192, 336, 720)
(8545, 2881, 2881)
Electricity (15 mins)
Forecasting
Electricity
321
(96, 192, 336, 720)
(18317, 2633, 5261)
Electricity (Hourly)
(Long-term)
Traffic
862
(96, 336, 720)
(12185, 1757, 3509)
Transporiation (Hourly)
Weather
21
[96. 192, 336, 720)
(36792, 5271, 10540)
Weather (10 mins)
Exchange
8
[96, 192, 336, 720)
(5120, 665, 1422)
Exchange rate (Daily)
M4-Yearly
1
6
(23000, 23000)
Demographic
M4-Quarterly
(24000. 24000)
Finance
Forecasting
M4-Monthly
1
18
(48000, 48000)
Industry
(short-term)
M4-Weakly
1
13
(359, 0, 359)
Macro
M4-Daily
1
14
(4227, 0, 4227)
Micro
M4-Hourly
48
(414, 414)
Other
EdhanslConcentraation
3
1751
(261, 263)
Alcohol Industry
FaceDetection
144
62
(5890, O, 3524)
Face (250H>)
Handwriting
3
152
(150, O, 850)
Handwriting
Heartbeat
405
(204, 205)
Heart Beat
Classification
SipaneseVowelk
12
29
(270, 370)
(UEA)
PEMS-SF
963
144
(267, 173)
Transportation (Daily)
ScltRogulationSCPI
6
896
(268, O, 293)
Health (256Hz)
SclfRegulationSCC2
7
1152
(200, 180)
Health (256Hz)
SpokecurrabieDigits
13
93
(6599, 2199)
Voice (11025Hz)
UWaveGesturLLibrayy
3
315
(120, 320)
Gesture
SMD
38
100
(566724, 141681, 708420)
Server Machine
Anomaly
MSL
55
100
(44653, 1664, 73729)
Spacecraft
Detection
SMAP
25
100
(108146, 27037, 427617)
Spacocraft
SWaT
51
100
(396000, 99000, 449919)
Infrastructre
PSM
25
100
(105984, 26497, 87841)
Server Machine
re-using the definition of *, we have:
(p 9) *
q1 O P1 ①
Q2 1 P2
91 C
Ps
+
92 2 P6 + 93
F1
12
T3
(58)
O ps
+
F4
Y5
T6
(g) O P1)
T2
(4
12)
8
71
(11 8 P3
42
P6
43
+ r2
(44
Pa
+
45
P6
96
r3
(44 ④ P4)
T5
(45
8
91
8 P3
P6
+ P4
8
(94
2 Ps
45
P6
96
r6
(59)
22

---

Table Long-lerm forecasting task with different horizons H.
Chimera tours)
TSME
Sinbb
TCN
Teaasfonee:
KLineu (2023)
PRcCEST
Chossiemmer
TIDE [2025]
TenesSee
DLinex
SCESet [2022]
FEDioener
Statotary
Aunefoener
[3124]
[2024]
[2004]
[2024]
[222]]
(2023)
[2023]
[2025]
[2022]
[2022]
(2021)
MSE MAEJMSE MAE
MSE MAE
MSE MAEIMSE MAE [MSE MAEIMSE MAEJMSE MAE
MSE MAE MSE MAEJMSE MAE MSE MAE MSE MAEINSE MAE MSE MAE
56
10.318
122
100 1.290 0.146..314 1:16
jo o soils 100 1671. 2.44 0446
166 0 187 stx 6.355 a 12 n.41 0.4% T7V aroju 19 10 905
192
0:531
1w 5.149
16.67
0.1k3 0.312 6.766 0.9
0.301 a sonl 167 0.385 2.50 2.30
0451
1.198
404 10504 6.167 5.100 o se n--9 2.43 a dan 0.449 0.444 (0.55 495
0.353
0.389
0.595
0.465
0.
0.365
0.391 9425
8430
K.424 0415 (1551.900 0.41
0.15
14425
425 0.410
0411
5415 4443 0.000 0.465 2.445 a 4.99999 0.464-06221 0.577 (6621
720
0.409
0413
[1451 0.477
0.416
6417 p481 0459
(.AST 6.00 0.45 0.419
0.589
(.467
4646 0.475 0450 p411 453 0.50s 0,550 2.545 o 003/ 0.516 0:671 0,561
JAsg|0.355 0.381
0.356 (.333 0.381
1.414 (.407)
357 0.400/15133
0,0N
[0.419
10,00 0.400|.415
0.415
407 0.485 0.451 (0/48 452j0.88 0,456 0:517
1.17
6.265
0.377
(0224 0.319
192
0.298
0,200
0.246
0.14
0:302
(1200
0.445
0.259
324
536
0.34403333
0.307
542
1.305
0.597
0.52)
0.325
365
(0.339 0.372
720
0,350
0.400
0.400
10.05
0.735 0:421
0:435 0.432
|og]o 0.317/2266
0.337
0.314
11266
307
0.336/0.77
2577
0.500
(0.75%
0.35
0.517
Seiju 0.347
In [0.35s
b.378
1.379
0:365
0.390
0.366
0,448
[04499
0.344
(u400) 0.3t6
0,500 TLT o 413/0.513
(0:440
192 0.402
0.41
0.422
0.24
0:44
0.437
0.42
0.445
0-47)
6474
0.525
0.-35
0:429
0.719
(U631 00 o
0.504 0.500
(.400 0.41
0.43 0.291
0:47
0458
0.479
0.440
(0501
0-46
6,54
0.565
015
0..03 0,400
0.778
0.659 0.459 463 00598 0.535 (.52) 0.495
720 02050 0.47 0.477.4222 0.422
[1483 0:459 0.400
0.503
0:451 0.470
0.053
0.621
(.50
0.558
0.521 0.500 0.515 01836 0.509 0.506 D 507 (1.43 0.536 (35514 0.512
|Ass|0.405 0.42pp.408
[0.44] 0.42/ 00.04 04200.454
0.42/00.04
J0446
0.434j0.40 0.454 4.522 0.522
[0.541 0.507 0.455 0.50024556 0.452 0.747 0.647 (0.400 o 4.000.5500 (.437)04496 0.487
86 [0:265
0.327 1.5
10:000 0.119 261 0.310
9.797
7A8 o to un o 318) 225 0
4000000000
sao e 0.35
0.207 0.621 o 197 476 1.85/ 346 20s
0:570
5.100 0400
so (a 1--13-11000
02%
1-578 0
41 cutta 0-477
0.600 014 a d 2.44%/o-455
0.316.36.000 0:381
0.306 8:40e
01:17
0:578
0-488
1:415 n 125
10-83 cue0
1.000
0.94 bo la
(0-422 d85
0.300.30 0:40004440 0-401
[.407 0:41 (0
0,438
0437
0.465
1:420
0.40 (1831 0.440| 1004
0:670
(1462 0.400
0.31
1.240
0.838/044 o 424 0.550/0.515 0,11
[os]]0.32] 0,377p3335
[0.301 0,391 00322 0.30jp335 0.40
J.3T4 a so0 287 0.007 0.504
[0.61 0 0.5500114444 0.27
0.515 10054 0.725 90433 a 4esju.500 0.536
0.45
in 10.135 0.25 2.142
161 0.25
0.120
0.146
1:300
10.01 0:281 In 181 o 0.000 200 1s 11110
(1335 0 12000 16k 0.270
0 253 1.24 0.345/0 on a sasln 1eo 0.21% 201 0
192 0.146
0.223
2.55
1.773 0:262
0.147
0:25
wass
0.201
0.287
In 188 0.22a 22221 18 0322
1:336
0 330 10154
0.280
205 0.289 0.355 002001 i 5 182 0.22 01594
Eas 0.155 0.259
2.77
ass 0.277
0.161
0:25
0.178
02209
0.215
0.200
(1.21 0:205 200
0355
1:200
3544 11:95 0.300
5,209
0.501 0.209 0.360 0.224 0 520 1.000 0.300 (:233 0235
020/0.154 0.250 0220
[.214 0,305
0.191
0:20 9.225
0.317
6.257
0.30: (1:20 0.224 if 2:200
0200
a 0.303 10200 0.320
0:245 0.3 0.00 0.300/0.240 o 1 0.321 (0.254 0361
|A0g 0.200p...0
ju 0.274
00.155
pu. 0.200
102.19
0 2000 (o 2n5 o 0.200p 2244 03554
11:51 0.344 10.92 202000022222200
0 300 0.205 0.365 10.244 0 52 193 0.200 0,335
Exchangg
26 10,37 0.166
00100 0.150 bo0t
0.300
k.09s 0.21 11 c60 0.205 i 255 0307
100000 o 2.2 a 0.254
5x05S 0.215 0.207 0.306 (0.14 0 0.277111111 0,237 0.170.19 0323
1og [0.159
0,20 p.229
0.160 0.25 2,177
0,209
5.154 0
1766 2000 Luto apei
111000 o 3377 11226 0.340
1. 0.315 11551 0.459 (2271 i 0.3151120 0.219 0.335 0 0:300 (S99
Eie 0.311
0,340
5.355
n 13377 0.356 0.331
b.117
0.351 i tut 1007 nos 1 0500
10000 oo 0311 13007 rueto
5.315 477 1.304 cus5 00-00 o 477 171 0.4T6 / 0-000 (S74
Fno 00000.00
2.99
nos 0.533 p.447
0.091
j.856 i 71 oor 7.71 ( no0 1000
a ove 1100 71700 p.119 i oor 1IS 0.797 (1 on o I 1.00 0.760 F44 (194)
Jaug/0,311 0.35ap.44
(0,302 0.30p.350 0.000
J0.77 2412700.07 0.40 00. 00000 10 013 1.416 4.444.5555.000 0.414 0.250 0.526 1.5 o 425/04444 0.454 (0.63 0,539
10.356
0.39
11400 0.266 0 305 0,251 0,395 0.200
10640 0 11 062
(1
1.517 0.331
3M5 0.559 5NT 300 in 612 373 01133 345
Traffle
F9r 0.334
0.207
1411 0.317
( 1.579 0:761 0.417
10617 0.356 5.5
0.50 (OADE o 573 1161 C. 340 0.16 303
Fi 0.450
0.311
5.17
135 0.344
0 90 0.31
1:000 n we 120 500 4 109
029 1088 0.116 DAA
0.506 (1621 a 10 11518 o. 128 6.612 517
0.445 0.390
0.394..499
[.561 0.297 00.400 0,200 0-457
0.102
587 (1 114 112 21 0:00 03535
00540 c 150
594
0.54 0,525 (063 o 182 ( 681 0.355 (0660 408
|Asg]0.403 0.250|0.42
0.291
0.270/3422
Ju666 0.378)0.441 1.30.....0 0.30
10:700
4.33 10.20 0.330 0.303
o 3.55//00044 0.324 0,379
Weltber
6:319
o 1or to 1o
oonti o
173 to
100 717
7TH 106 0.54
2232//0 766
0.207
580 (-50) 0.318
0.314
p.358
p.354
0,80 (.54 0.3466.994 0000000000000 0.419
0.359
0-477
0.400 0.14 0.41000-110 0.426
josss 0,250/0.20 0:260
10.012
0.200 (1.39 $0250 01315 (0:271 0:500 290
poses 0:517 j 0. NOO0 So00 0.314/0.336
Using the fact that @ and @ are associative, we have:
(p x q) * r-=
91 O P1
Q2 ② P2
91 C Ps
+
42
2
P6
+
43
*
[1
T2
T3
(60)
91 P1
95
1 Ps
91
P3
96
8
P6
+
96
F4
T5
T6
F1
(41
O P1)
12
P2)
71
8
(q
8
P3
+
42
8
P6
+ 43)
+ 12 2
(44
3
P3
+
4s
8
P6
+ 96
+ r33
ra
(qa
P4)
P5)
74
(9
P3
+
92
P6
43)
14
(94
Ps
45
P6
96
T6
(61)
P1
P2
P3
*
r 01
r 42
1
0
43
+
12 09
96 +
(62)
P4
P5
P6
F4 1 94
0
14
0
43
+ 13
96
+ T6
p
(q * r);
(63)
which proves the theorem.
G.2 Proof of Theorem 3.3
For each v t, we can pre-compute Byx., and B2xe, Accordingly, all the following parameters are pre-computed:
(L.j.k.)
Ai A2 Bix
(64)
Ag A4 Bexe+katt)
23

---

Table 9: Full results for the short-term forecasting task in the M4 dataset. in the Transformers indicates the name of =former. Stationary means the Non-stationary Transformer.
Models
Chimera Co PaschTST
NimesNett-HHiT N-BEATS
ETS+ LighTTS DLinear FED+ Statonayy Antos Pym
LSTM
(ours)
[2024]
[2023]
[2023)
[2022]
[2019)
[2022)
(2022)
[2023]
[2022]
[2022] (2021](202) (2021] [2020] [1997]
Yearly
SMAPE
13.107
13.226
13.258
13.387
13:418
13,436
18,009
14,247
13.728
13,717
13,974
14.727
16,169
176.040
MASE
2902
2.957
2985
2.996
3.045
3.043
4.487
3,109
4.283
3.048
3,078
3.134
3.711 3,418
3,800
31,033
OWA
0.767
0.722
0.781
0.786
0.793
0.794
1,115
0.827
1.058
0.503
0.807
0.822
0,942
0.881
0,973
9.290
Quarterly
SMAPE
9.921
10.179
10,100
10.302
10.124
13.376
11.364
12.145
10,792
10,958
11,338
15
1.366113,113
172,80
MASE
L105
1.167
0
1.182
1.194
1.169
1.906
1.328
1.520
1.283
1.325
1,365
2,350
1.401
1.775
19.753
OWA
0.53
0.878
0.803
0,890
0.899
0,886
1,302
1,000
1.106
0.958
0.981
1,012
1,558
1.027
1,252
15.019
Monthly
SMAPE
12.549
12,556
12.641
12,670
12.791
12,677
14,588
14,014
13.514
14,360
13,917
13,958
17
14.022
20.128
143.233
MASE
0.914
0.917
0.930
0,933
0.969
0.937
1,368
1.053
1.037
1.102
1,097
1,103
19113
1.343
2,614
16,551
OWA
0.864
0.866
0.876
0.878
0.899
0,880
1.149
0.981
0.956
1.012
0.998
1.002
1.511
1.024
1.977
12.747
Others
SMAPE
4.688
4.715
4.946
4,891
5.061
4.925
7.267
15,880
6,709
4.954
6.302
5.485
24.786
24.
32.42. 33.491
186.282
MASP
3,007
3,107
2,985
3,302
3,216
3.391
5,240
11.434
4.953
3,264
4,064
3.865
18.581
20.960
33.355 119.294
0.986
1.044
1.035
1.053
1.591
3.474
1.487
1.036
1.304
1.187
5,538
8.679
38.411
Weighted
Average
SMAPE
11.618
11,608
11.807
11,829
11.927
11,851
14.718
13.525
13.639
12.840
12,780
12,909
16.987
14.086
18,200
160.031
MASE
1,556
1,590
1,585
1.613
1.599
2,408
2,111
1.701
1.756
1,771
3.265
2.718
25.788
0.838
0.855
1.172
1,051
1.051
0.918
1,400
1.775
12,642
(1
1 0)
for all inputs Xx,t and k, E {0, 1}.
Now, starting from
we have:
S
S
I
0
A1
A2
B B2X1,0)
S
I
1
0
*
As
A1
(65)
A1
A2
B
(66)
Ag
A1
B2X1,0/
Re-using operator *, we have:
S1.1
S
A1
A2
B
$1,1
1
*
A3
A4
B2X1,1
(67)
Pre Pompulee
A
A
A, + A₂B2X1, + B1X1,1) AgBixo,1 + AaB2x1, + B2x1,1)
(68)
A3
A
Fixing the variate by putting 0 as the initial points of the operator, and then looking at the third element of each row, these elements are calculating the hidden states of the recurrent (it can be shown by straightforward induction). Accordingly. using this operation, we can recursively calculate the outputs of 2D SSM
However, using Theorem 3.2, we know that this is an associative operation, so instead of calculating in the recurrent form, sequential operations required to calculate the hidden states. Note that since our above operation can model the problem as an parallel prefix, all the algorithms for this problem can be used to enhance the efficiency.
use parallel pre-fix sum make this computation parallel, decreasing the
G.3 Proof of Theorem 3.4
To prove this theorem, we need to (1) show that Chimera can recover SpaceTime. Given this, since SpaceTime is capable of recovering ARIMA [14], exponential smoothing [16], and controllable linear lime-invariant systems [87], we can conclude that Chimera can also recover these methods. Then, (2) we need to prove that Chimera recover SARIMA. This is the model that Sime is not capable of recovering due to the additional seasonal terms.
Note that using A₂ A3 A4 = 0, results in a 1D SSM, with companion matrix as the structure of A1, which is SpaceTimee Accordingly, Space is a special case of Chimera when the recurrence only happen along the time direction.
24

---

Note that as discussed in Proposition 3.1, multiplying the discretization parameter A results in multiplying the steps. Accordingly, using as the Á in our seasonal module and also letting 2 A3 A4 = 0 for the seasonal module, we can model the seasonal terms in the formulation of (p q, 8), meaning that Chimera can also recover SARIMA which ARIMA with seasonal terms. Note that the reason that Chimera is capable of such modeling is that it uses two heads separately for trend and seasonal terms. Therefore, using different discretization parameters, each can model their own corresponding terms in SAR(p, q, g)
G.4 Proof of Theorem 3.5
Similar to the above, using A2 A3 = O. our formulation is equivalent to S4D, while we use diagonal matrices as the structure of A1. Similarly, as discussed by Behrouz al. [31], MambaMixer equivalent to S4ND but on patched data. Using our Theorem 5, we can recover linear layers, resulting in recovering TSMixer by setting A2 A, =.
G.5 Proof of Theorem 3.6
We fact will show that restricting Chimera results in recovering 2DSSM [49]. As discussed earlier, this method do not use discretization and initially starts from a discrete system. Also. it uses input- independent parameters. Therefore, we use Linear (.) = Linear (-) as broadcast function, and restrict Chimera have input-independent parameters, then Chimera recover 2DSSM [49].
H
Experimental Settings
We provide the description of datasets in Table 7.
H.1 Baselines
In our experiments, we use the following baselines:
Table 8: TSM2 [31], Simba [57]. TCN [9], iTransformer [27]. RLinear [58], PatchTST [29].
Crossformer [26], TiDE [55], TimesNet [8), DLinear [59], SCINet [56], FEDformer [24],
Stationary [86], Autoformer [21].
Table 9: ModernTCN [9]. PatchTST [29], TimesNet [8], N-HiTS [65], N-BEATS [66],
ETSformer [20], LightTS [67]. DLinear [11], FEDformer [24]. Stationary [68], Auto-
former [21], Pyraformer [23], Informer [6], Reformer [25], LSTM [69]]
Table 10: LSTM [69], LSTNet [88], LSSL [34], Trame.former [17], Reformer [25], In-
former [6]. Pyraformer [23], Autoformer [21], Station. [68], FEDformer [24], ETS-
former [20], Flowformer [22], DLinear [11], LighTTS. [67], TimesNet [8], PatchTST [29],
MTCN [9].
For the results of the baselines, re-use the results reported by Wu al. [8], or from the original cited papers.
I Additional Experimental Results
I.1 Long Term Forecasting Full Results
The complete results of long term forecasting are reported in Table 8.
1.2
Short-Term Forecasting
The complete results short term forecasting are reported in Table 9.
L.3
Classification
The complete results of time series classification reported in Table 10.
25

---

Table 10: Full results for the classification task (accuracy %). We omit "former" from the names of Transformer-basee methods. For all methods, the standard deviation is less than 0.1%.
LSTM LSTNer LSSL. Trans, Re. In. Pym. Auto. Station. FED. /ETS. /Flow./Dl.mm wtw.://.iinearl.aa. LighTTS /
Datasets/ / Models
[1997]1200512202220001 2110022000020012200000000000
01200/00000100000000001222111110 [2022]
(2022]2022][202] [2023]
[2022]
[2023]
[2023]
[2024] (ours)
RchanolComentrraionn
32.3
39.9
31.1
31.9
31.6
30.8
32.7
28.1
33.8
32.6
29.7
35.7
32.8
36.3
39.8
FacdDetcction
57.7
65.7
66.7
67.3
68.6
67.0
65.7
68.4
68.0
66.0
66.3
67.6
68.0
67.5
68.6
68.3
70,8
70.4
Handwriting
15.2
25.8
24.6
32.0
27.4
328
29.4
36.7
31.6
28.0
32.5
33.8
27.0
26.1
32.1
29.6
30.6
32.9
Heartbeat
72.2
77.1
72.7
76.1
77.1
80.5
75.6
74.6
73.7
73.7
71.2
77.6
75.1
75.1
78.0
74.9
77.2
81.3
VenvesseVonvels
79.7
98.1
98.4
98.7
97.8
98.9
98.4
96.2
99.2
98.4
95.9
98.9
96.2
96.2
98.4
97.5
98.8
99.1
PEMS-SF
39,9
86.7
86.1
82.1
82.7
81.5
83.2
82.7
87.3
80.9
86.0
83.8
75.1
88.4
89.6
89.3
89.1
89.5
SelfRegshatioSCPP
68.9
84.0
90.8
92.2
90.4
90.1
88.1
84.0
89.4
88.7
89.6
92.5
87.3
89.8
91.8
90.7
93.4
93.7
SelfRegalationSCC2
46,6
52.8
52.2
53.9
56.7
53.3
53.3
50.6
57.2
54.4
55.0
56,1
50.5
51.1
57.2
57.8
60.3
59.9
SpolenAraahiDDigiss
31.9
100.0
100.0
97.0
100.0
99.6
100.0
100.0
98.8
100.0
98.7
100.0
UWwoGGstucLilbbayy
41.2 87.8
85.9
85.6
85.6
85.6
83.4
85.9
87.5
85.3
85.0
86.6
82.1
80.3
85.3
85.8
86.7
86.7
Average Accuracy 48.6 71.8 70.9 71.9 71.5 72.1 70.8
71.1 72.7
70.7
71.0
73.0
67.5
70.4
73.6
72.5
74.2
75.3
Table 11: Full results for the anomaly detection task. The P, R and Fl represent the precision, recall and F1 Fc-score (%) respectively. higher value of P. R and F1 indicates better performance.
Datasets
SMD
MSL
SMAP
SWaT
PSM
Avg
Metries
R
F1
P
FI
R
F1
P
FI
F1
(%)
LSTM
[1997]
78.52
65.47
71.41
78.04
86.22
81.93
91.06
57.49
70.48
78.06
91.72
84,34
69.24
99.53
81.67
77.97
Transformer
[2017]
83.58
76.13
79.56
71.57
87.37
78.68
89.37
57.12 69.70
68.84
96.53
80.37
62.75
96.56
76.07
76.88
LogTrans
[2019]
83.46 70.13 76.21
73.05
87.37
79.57
89.15
57.59 69.97
68.67
97.32 80.52
63.06
98.00 76.74
76.60)
TCN
[2019] 84.06 79.07 81.49
75.11
82.44
78.60
86.90
59.23 70.45
76.59
95.71 85.09
54.59
99.77 70.57
77.24
Reformer
[2020] 82.58 69.24 75.32
85.51
83.31 84.40
90,91
57.44 70.40
72.50 96.53 82.80
59.93
95.38 73.61
77.31
Informer
[2021]
86.60 77.23 81.65
81.77
86.48
84.06
90.11
57.13 69.92
70.29
96.75 81.43
64.27
96.33 77.10
78.83
Anomaly"
[2021]
88.91 82.23 85.49)
79.61
87.37 83.31
91.85
58.11 71.18
72.51 97.32 80 65.35 94.72 79.40
80.50
Pyraformer
[2021] 85.61 80.61 83.04
83.81
85.93 84.86
92.54
57.71 71.09
87.92
96.00 91.78 71.67 96.02 82.08
82.57
Autoformer
[2021]
88.06 82.35 85.1
77.27
80.92 79.05
90.40 58.62 71.12
89.85
95.81 92,74 99.08 88. 15 93.29
84.26
LSSL
[2022]
78.51 65.32 71.31
77.55
88. 82.53
89.43 53.43 66.90
79.05
93.72 85.76 66.02 92.93 77.20
76.74
Stationary
[2022]
88.33 81.21 84.62
68.55
89.14 77.50
89.37 59.02 71.09
68.03
96.75 79.88 97.82 96.76 97.29
82.08
DLincar
[2023]
83.62 71.52 77.10
88.34
85.42 84.88
92.32
55.41 69.26
80.91
95 87.52 98.28 89.26 93.55
82.46
ETSformer
[2022]
87.44 79.23 83.13
85,11 84.93 85.03
92.25
55.75 69.50
90.02
80.36 84.91 99.31 85.28 91.76
82.87
LighTTS
[2022]
87.10 78.42 82.53
82.40 75.78 78.95
92.58
55.27 69.21
91.98
94.72 93.33 98.37 95.97 97.15
84.23
PEDfermer
[2022]
87.95 82.39 85.08
77.14 80.07 78.57
90.47
58.10 70.76
90.17
96.42 93,19 97.31 97.16 97.23
84.97
TimesNer
[2023]
87.76 82.63 85.12
82.97
85.42 84.18
91.50
57.80 70.85
88.31
96.24 92. 98.22 92.21 95.21
85,49
TimesNet (R) [2023]
83.14 85.81
83.92
86.42 85.15
92.52
58.29 71.52
86.76
97.32 91.74 98.19 96.76 97.47
86.34
CrossFormer
[2023]
83.6 76.61 79.70
84.68
83.71 84.19
92.04
55.37 69.14
88.49 93.48 90.92 97. 16 89.73 93.30
83.45
PatchTST
[2023]
87.42
81.65 84.44
84.07
86.23 85.14
92,43
57.51 70.91
80.70
94.93 87.24 98.87 93.99 96.37
84,82
ModemTCN
[2024]
87.86
83.85 85.81
83,94
85,93
84.92
93.17
57.69
71.26
91.83
95.98 93.86 98.09 96.38 97.23
86.62
Chimera
(ours) 87.74 83.29 85.46
84.01
86.83
85.39
93.05
58.12
71.55
92.18
95.93
97.30 99.19 96.74
86.69
1.4
Anomaly Detection
The complete results of anomaly detection tasks are reported in Table 11.
J Additional Ablation Study
K Comparison with Koopman-based Deep Models
We also compare the results of Chimera with Koopman- based deep models [84]. In ECL dataset, Chimera achieves 0.132, 141, 0.144 for (h 96, 144, 192), while Koopa [84] achieves 0. 0.136, 0.149, 0.156. Similarly, in ETTh2, Chimera achieves 0.262, 0.309, 0.320 for (h- = 96, 144, 192), while Koopa achieves 0.297, 0.333, 0.356. Similarly, in Exchange, Chimera achieves 0.077, 0. 126, 0.159 for (h- = 96, 144, 192), while Koopa achieves 0.083, 0. 130, 0.184 Since Liu et al. [84] has focused on these datasets and values for the horizon, we use the same setting for Chimera, The results for Koopa are from its original paper.
26

---

Table 12: Ablation Study on additional datasets.
BVFC
Exchange
Traffic
Method
Acc. %h
MSF
MAE
MSE
MAE
Chimera
58.99
0.077
0.198
0.366
0.248
-ireecirtti
$7.29
0.203
0.369
0.255
w/o Gating
56.18
0.094
0.210
0.373
0.259
Input-independen
55.31
0.113
0.228
0.402
0.271
wño seasonal
58.12
0,083
0.202
0.372
0.258
Diagonal
57.98
0.095
0.209
0.370
0.257
Companion
0.226
0.375
0.261
L
Limitations
The main goal this study is to enhance the time series modeling with a wide range of impact on society, from improving the healthcare system using developing deep learning models for analysing medical health records to forecasting stock. We, however, emphasis that our work is a proof-of- concept. meaning that has error modes.
27

---

1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix L.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
See Appendix G.
Guidelines:
28

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: See Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: It is available in this link.
29

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //niss. wwtppbccc/ppppcccccccccccccccccc gewiiiiggi/////eeeeeeeeiiiiiesee geoidese/ooeuummiiiiiccc Sttp//ooooSusssssaeeoeoee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See above question.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
30

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See question 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tteee//neuri...eeee http://urrpp.cpppulliiieeepeeeee httppp/cleticccauueeelelee..eeee cswuidelslllllleecccccccccc ines?
Answer: [Yes]
Justification:
We have reviewed NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix
L.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
31

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:[NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, saperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
32

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "fjzXpKznQN", "reviewer": "Reviewer_v8PB", "summary": "The paper introduces Chimera, a novel 2-dimensional State Space Model (SSM) for multivariate time series modeling, addressing key challenges such as capturing complex temporal and inter-variate dependencies, and efficient training. Chimera uses two SSM heads with different discretization processes and time-variant parameters to learn long-term progression, seasonal patterns, and dynamic autoregressive processes. A new 2-dimensional parallel selective scan improves training efficiency. Experimental results show Chimera's superior performance in various benchmarks, including ECG and speech classification, time series forecasting, and anomaly detection, marking a significant advancement in the field.", "questions": "What could be the interpretations for Structure of Transition Matrices? For example,  it is observed that a simpler structure of diagonal matrices is effective fuse information along the variate dimension.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper is well written. The notations are clear and the literature review is sufficient.\n2. By using SSMs, the proposed Chimera Neural Architecture can achieve faster training and less memory consumption.\n3. Experimental results cover a wide range of applications, including Long-Term Forecasting, Anomaly Detection, Short-Term Forecasting, etc. The results are very solid and demonstrate the advantages of adopting the proposed method well.", "weaknesses": "1. the writing and expression may need correction: Line 308, it is hard to understand the message:\n'they are also data-dependent and so shows the second best results in second and third datasets.'\n2. It would be great t know what kind of market dynamics are captured by with visual impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your time and constructive review. We really appreciate it. Please see below for our response to your comments:\n\n> *the writing and expression may need correction …*\n\n**Response:** Thank you for bringing this to our attention. Following your suggestion, we will make sure to further simplify expressions and make our messages clearer in the final version. Regarding this specific sentence, the message is: iTransformer and Transformer both are also data-dependent. As a result of their data-dependency, they are capable of achieving good results in this experiment, where iTransformer achieves the second best result (after Chimera) and Transformer achieves the third best result (after Chimera and iTransformer). Note that we have three datasets in this experiment, and both Transformer and iTransformer are out of memory for the first dataset. Therefore, these good results have been achieved in the second and third datasets. \n\n> *It would be great t know what kind of market dynamics are captured by with visual impact.*\n\n**Response:** Following your suggestion, we have added the visualization of the trend and seasonal patterns to the attached pdf to the general response. These results visually show how Chimera learns different types of patterns using its different modules, verifying our claim about its ability to capture both seasonal and trend patterns.\n\n> *What could be the interpretations for Structure of Transition Matrices?*\n\n**Response:** The transition matrix can be interpreted as how we want to mix the features of the hidden states. That is, assuming a dense structure for the transition matrix A, then A_{i, j} means the corresponding transition weight of feature j for feature i. When we have a diagonal matrix, it means that we treat each feature separately and so given feature i, other features j \\neq i do not affect the transition of feature i. While dense transition matrices are more expressive, unfortunately, it has been proven that the SSMs themselves are not capable of learning dense transition matrices and SSMs need a structured transition matrix. To this end, several studies theoretically and empirically explored the diagonal transition matrices and showed that they are effective and provably powerful. \nIn general, the structure of the transition matrix tells us how we want the features of hidden states to be mixed in each step. In other words, this structure shows the underlying dependencies of features in the hidden states. \n\n**Contributions:**\nWe are so grateful for recognizing the fact that Chimera achieves state-of-the-art performance in various datasets and tasks. We further wanted to kindly bring some of the main contributions of the paper to your consideration. \n\n(1) New mathematical framework: (a) We present a novel 2-dimensional SSM with theoretical guarantees for its expressiveness that uses data-dependent parameters, enabling the model to capture both the variation-specific and inter-variate dynamics over time.  (b) Using a new discretization for our 2D SSM, we present a novel approach to capture seasonal and trend patterns. We provide theoretical motivation for how the interpretation of our discretization can be seen as learning seasonal and trend patterns.\n\nOne of the controversial debates in recent years has been whether mixing variates in multivariate time series is needed or not. Our data-dependent 2D-SSM naturally addresses this issue as it is able to learn variate dependencies when it is informative. To the best of our knowledge, our work is the first study that proposes a 2D-SSM for time series data.\n\n(2) New simple alternative, yet effective architecture for time series: A considerable amount of research effort in recent years has focused on Transformers and/or MLP-based architecture. This study challenges the necessity of Transformers and MLPs by proposing an alternative method based on a simple linear recurrence. Not only does this simplicity come with significant efficiency with respect to memory and time, but it also results in a model that performs better or on par with state-of-the-art Transformer- or MLP-based models. \n\n(3) New Training: Recurrent neural networks have been popular paradigms to model time series data. They, however, have slow training due to their recurrent nature. Simple time-invariant (data-independent) SSMs have addressed this limitation by training in the convolutional form. They, however, have limited expressivity, due to their time-invariant (data-independent) parameters. Our study presents a new training for 2D recurrence that enables parallelization. This new training is $\\times 3$ faster than recurrence, making the model feasible for large datasets.   \n\n(4) Empirical Results: In forecasting tasks, Chimera improves the best Transformer-based, convolution-based, recurrent, and MLP-based models by 13%, 8 %, 11 %, 15 % improvement. Its fast training is $\\times 7$ faster than Transformers, $\\times 3$ faster than convolutions, and $\\times 2.5$ faster than LSTM.\n\n\nWe want to kindly bring to your consideration that coming up with one architecture like Chimera that provides an alternative to existing backbones (e.g., Transformers or MLPs), has theoretical motivations, improves training, is simple and fast, yet very effective and has generalizable performance to unseen variates, is highly challenging and usually has not been done in the existing studies. As some examples, [1] (ICML Oral) has focused on efficiency, [2] (ICML Oral) has focused on training improvement and theoretical motivations, [3] (ICLR Spotlight) has focused on generalizable performance to unseen variates, and [4] (ICLR Spotlight) has focused on alternative architecture.\n\nWe hope our responses have fully addressed your concerns/questions about the paper. We would be more than happy to answer any further questions or discuss any remaining concerns of the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BcCvfWI8C4", "reviewer": "Reviewer_hxax", "summary": "The paper addresses the challenge of multivariate time series modeling using a neural architecture based on a variation of two-dimensional state-space models (SSMs), referred to as Chimera. This approach features a stack of 2D SSMs combined with nonlinearities, a decomposition of time series into trend and seasonal components, various discretization processes, time-variant parameters, and a novel two-dimensional parallel selective scan for rapid training with linear complexity. The authors demonstrate that Chimera can encompass multiple existing time series models, including SARIMA, SpaceTime, and S4nd. Experiments across various tasks, such as time series forecasting, classification, and anomaly detection, validate the effectiveness of Chimera.", "questions": "See weaknesses.", "limitations": "The limitations have not been discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- State-space models (SSMs) have proven to be very effective and efficient for modeling temporal data compared to Transformers. Designing new efficient SSM architectures for multivariate time series is an important and challenging problem.\n\n- Chimera allows the state space to vary in two dimensions (time and variate). It also combines multiple components to build a specific architecture for multivariate time series.\n\n- Chimera is compared to multiple baselines across various datasets for tasks such as time series forecasting, classification, and anomaly detection.", "weaknesses": "- **Motivation**\n  - Two-dimensional state-space models (2D-SSMs) have been considered for multivariate time series with specific structures, such as S4ND and 2DSSM. The authors discuss the discretization process and data-independent parameters as new challenges. However, I found lines 113-128 unconvincing regarding the need for a new 2D-SSM for multivariate time series.\n\n  I believe this paper proposes yet another deep model for time series, while it is known that after removing strong seasonality, multivariate time series from real-world phenomena often exhibit weak time and variate dependencies/signals, which do not require such complex models. This is very different from text and image data.\n\n- **Contributions**\n  - The paper essentially combines multiple existing ideas: 2D-SSM, bi-directional models [46], selection mechanisms [32], companion structures [27], convolutional forms [33], etc. There are no substantial or specific contributions. The authors claim that one of their main technical contributions is having input-dependent parameters, which is a relatively weak contribution.\n\n- **Experiments**\n  - Even considering the experimental results, many (simpler) methods achieve similar results to the proposed approach (see Tables 8-11).\n\n  - I also found that the paper tends to overstate its claims, using phrases like \"outstanding performance\" and \"significantly outperforms recurrent models, including very recent Mamba-based architectures.\"\n\n  - I question whether a fair comparison was made between the methods. For example, Chimera involves a time series decomposition (lines 218-219, 236-237), whereas other baselines do not. Can the authors comment on this?\n\n  Additionally, there is a lack of naive baselines, such as methods that treat the multivariate series as multiple univariate series.\n\n  It would also be useful to visualize some time series decompositions of Chimera to validate its accuracy.\n\n  - The authors write, \"We attribute the outstanding performance of Chimera, specifically compared to SpaceTime [27], to its ability to capture seasonal patterns and its input-dependent parameters, resulting in dynamically learned dependencies.\" Why didn't you confirm this by analyzing the results?\n\n  - Given that the authors combine multiple existing ideas/components, I found the ablation study relatively weak.\n    - \"The results show that all the components of Chimera contribute to its performance.\" I do not agree with this statement. Table 6 only shows that input-independence reduces accuracy (significantly).\n    - Why did you only choose ETT datasets?\n\n  - For the forecasting problems, it would be useful to provide the results per horizon or group multiple horizons together to analyze performance over the forecast horizon. Does the improvement come from the first few horizons?\n\n\n\n- **Data**\n  - I question the validity of the benchmark datasets used.\n    - For time series forecasting, there is the Monash Time Series Forecasting Repository (https://forecastingdata.org/). I think the multivariate time series datasets considered in the paper have been overly used and overfitted by the community.\n    - For anomaly detection, I encourage the authors to check the following paper: Multivariate Time Series Anomaly Detection: Fancy Algorithms and Flawed Evaluation Methodology (https://arxiv.org/abs/2308.13068)\n\n- **Other comments**\n  - I think Theorems 1-5 should be labeled as Propositions.\n  - Seasonal autoregressive process: The authors introduce the AR(p) model with a multivariate time series (x_k in R^d, not R^1). This is confusing as there is also the VAR(p) model with a different formulation. The AR(p) presented uses the same coefficients for each dimension.\n  - Table 7: What do you mean by \"series length\"? M4 series do not have six observations. Also, it is not long-term vs short-term but rather multivariate vs univariate time series. This should also be changed in the main text. Does this mean that Chimera has been used with both univariate and multivariate time series? What are the implications? Do you still need the second dimension in that case?\n  - The paper uses different terminologies: \"intervariate information flow,\" \"inter-variate dependencies,\" \"complicated dependencies,\" \"dynamically model the dependencies of variate and time dimensions,\" \"cannot capture the dynamics of dependencies.\" Please be consistent and define these terms clearly. There are a lot of \"buzz\" words without clear definitions.\n\n  - The authors write: \"Note that anomaly detection can be seen as a binary classification task.\" This is supervised anomaly detection, which is rare as labeled anomalies are hard to find. How did you handle the imbalance during training?\n\n- **Typos**\n  - Line 665: \"These matrices can be decompose\"\n  - Line 695: \"recursively calculate the the outputs\"\n\n- Please provide standard errors in your tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your time and constructive review. We really appreciate it. Please see below for our response to your comments:\n\n**Motivation:** \n> *2D-SSMs have been considered for multivariate time series ...*\n\n**Response:** To the best of our knowledge, neither S4ND nor 2DSSM has been used or designed for multivariate time series. Both of these methods are designed for vision tasks and also, due to the fact that they assume continuous data for their input, their adoption to multivariate time series is non-trivial. Even in the case of adoption, both of these models have a global convolutional form that is causal and so sensitive to the order of variates, which is undesirable. \n\nOn the other hand, please note that data-dependent and data-independent SSMs are two different classes of models with different theoretical expressiveness [1]. Our 2D SSM is theoretically more expressive than both S4ND and 2DSSM, and these two can be seen as the special cases of our 2D SSM. For example, making parameters data-independent and assuming $A_2 = A_3 = 0$, then our 2D SSM is equivalent to S4ND. To the best of our knowledge, Chimera is the first work that introduces deep 2D SSMs for multivariate time series. \n\nTo the best of our knowledge, there are still debates about this in the community and it is still in exploration. Also, one important counterexample is the brain activity, where variates are brain voxels. In this case, even after removing (potential) seasonal patterns, the dependency of variates and their co-activation is the key to decode neural activity (we have reported the performance of Chimera in brain decoding in Table 5). \n\nThis raised concern by the reviewer indeed is one of the main motivations of Chimera (discussed in lines 51-62). That is, Chimera has transition matrices $A_2$ and $A_3$ as well as two different hidden states, which determines cross variate and cross time information flow. When there are no variate dependencies, Chimera learns to consider $A_2$ and $C_2$ as zero matrices. Similarly, when there are no time dependencies, Chimera learns to consider $A_3$ and $C_1$ as zero matrices. This adaptability and flexibility of Chimera allows us to overcome these controversial debates in the community and let the model learn dependencies from the data. \n\nWe realized that the reviewer has recognized our work as a complex model. We want to kindly bring to your consideration that our model is a simple recurrence without any complex attention, Transformer block, or even MLP layers, which have been inseparable part of recent deep models. Indeed, one of the main motivations of Chimera is to present a simple alternative to these complex modules. Please note that, all the discussions about selection mechanism, seasonality, convolutional form, and parallel scan, are not additional elements but are embedded in the simple 2D data-dependent recurrence and provide theoretical interpretations/motivations for the good performance of Chimera. Please see Figure 1 for the architecture of Chimera, which has only three distinct modules. \n\n**Contributions:**\n\n> *The paper essentially combines multiple existing ideas …*\n\n**Response:** We want to kindly bring some of our main contributions to your attention: \n1. Extension to 2D for Multivariate Time Series: To the best of our knowledge, Chimera is the first deep model that suggests using 2D SSM for multivariate time series. The formulation of the 2D SSM as well as its discretization is new and it is specifically designed to effectively model multivariate time series (Theorems 3, 4, 5). \n2. Data Dependency: Chimera in nature is a simple model that uses linear layers across (i) time, (ii) variates, (iii) time-to-variate, and (iv) variate-to-time. This linear model, however, by itself (with data-independent parameters) does not perform competitively to more complex architectures like Transformer-based (PatchTST, iTransformer), MLP-based (TSMixer), and convolution-based (ModernTCN) models. We believe making the parameters data-dependent in this context is a non-trivial contribution and is significant as with a small modification (3 lines of code), this simple linear model becomes a competitive alternative for these complex models. Also, please note that Chimera is one of the first studies that discusses the importance of data-dependency in time series models (and supports it with a case study on brain activity). \n3. Fast Training: The main drawback of making parameters data-dependent is to lose the convolutional form and efficient training. We present a non-trivial training procedure for the recurrence of Chimera, which allows parallelization. \n4. Time Series Decomposition with Discretization.\n5. Multivariate Closed-loop: We further present multivariate closed-loop block, which improves the performance of Chimera for very long horizons (please see the attached PDF). \n\nMoreover, we have made some additional smaller contributions, but important and novel: (1) **Bidirectionality**: Please note that existing bidirectional SSMs use two black-box blocks of SSMs in the forward and backward direction. Our version of bidirectionality, does not use two different blocks, but only makes the recurrence across variates bidirectional. This helps us to improve the efficiency and use less parameters. (2) **Transition Matrix Structure**: Please note that even in the context of vision-tasks, existing 2D SSMs (i.e., 2DSSM and S4ND) use the same transition matrix structure for both dimensions. To the best of our knowledge, our work is the first work that suggests using different structures for transition matrices to further improve the expressive power. We showed this by our theoretical results. To further support this claim, we have provided additional ablation studies in the attached PDF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fA0irJL5y6", "reviewer": "Reviewer_gRR3", "summary": "This paper proposed to use a state space model for time series modelling. Instead of using the SSM along the time dimension, the authors also have the space updated along the variables dimension, which makes the established both inter- and intra- variable dependencies.", "questions": "- The evaluation is conducted on the forecasting sequence instead of only on the targeting time steps. How is it valid to tell whether the model performs better in the targeting time step or is it only better because it has superior performance in the more recent ones? It is necessary to also add the performance of different methods only on the targeting time step.\n- What is the advantage of using a state space model to establish intervariable dependencies in other neural network structures (such as CNNs or feedforward networks), given that there are no state variations between variables?\n\n\nAdditional\n- The proposed model looks very similar to WITRAN, which conducted 2D RNN. It would be good to have Chimera distinguished from WITRAN.", "limitations": "The limitation of the paper is mainly about how the method may not apply to real-world applications. The technical limitations are not mentioned.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clear and the focus is valid for time series forecasting.\n- The results show that the model holds value in providing better results over different time series tasks with competitive memory consumption.", "weaknesses": "- For forecasting tasks, the way the authors conducted evaluation metrics does not reflect the model performance for the target horizon. See detail in Questions.\n- The method does not present a consistent improvement over prior work.\n- Typos:\n    - The notation does not match the content. For example in Table 8, the first, second and third best results are all not highlighted with the provided notation.\n    - [minor] Checklist missing guidelines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your time and constructive review. We really appreciate it. Please see below for our response to your comments:\n\n> *For forecasting tasks, the way the authors conducted evaluation metrics does not reflect the model performance for the target horizon …*\n\n**Response:**\nIn our experiments on forecasting tasks, we have used the commonly used pipeline in the literature [1-6] and conducted experiments on different targeted horizons (i.e., 96, 192, 336, 720). The results for each target horizon are reported in Table 8. If Chimera’s better performance would be related to its superior performance in the more recent time steps, then increasing the horizon would diminish this effect. In contrast,  comparing different horizons, Chimera even achieves relatively better results on longer horizons. We have provided additional experiments on the performance of Chimera in even longer horizons in the attached PDF. \n\n> *The method does not present a consistent improvement over prior work.*\n\n**Response:**\nWe want to kindly bring to your consideration that coming up with an architecture that outperforms all the baselines consistently is an extremely challenging task. That is, due to the variety of datasets and some of their specific properties, a method might not achieve SOTA performance on all datasets. This pattern is seen in the original paper of most general time series models with diverse evaluations, including known methods like iTransformer (ICLR Spotlight), PatchTST, ModernTCN (ICLR Spotlight), SAMformer (ICML Oral). Chimera outperforms baselines in most datasets (27/32), indicating its generalizability and good performance. Also, please note that as is shown in Figure 4, this good performance is achieved while Chimera is more than $\\times 5$ faster than Transformers and $\\times 3$ faster than RNNs. \n\n> *Typos*\n\n**Response:**\nThank you for bringing this to our attention, we will make sure to fix this and highlight second and third best results in Table 8. We will further proofread the paper to make sure to address any inconsistency. We also will modify the checklist following the NeurIPS guideline in the final version of the paper.\n\n> *What is the advantage of using a state space model to establish intervariable dependencies in other neural network structures ...*\n\n**Response:**\nThank you. This is a very interesting question and indeed is a promising research direction. Due to the limited resources and also page limit, and for the sake of consistency, in this paper, we have focused on our new proposed 2D SSM and how simple linear models across (i) time, (ii) variates, (iii) time-to-variate, and (iv) variate-to-time can result in a powerful and fast architecture with theoretical justification. However, considering hybrid models of CNNs, MLPs, Transformers with SSMs to capture inter-variable dependencies is indeed an important question, which we will explore in future studies. As you mentioned, for some of these structures like CNNs, there are no states. However, one idea can be to use the output of CNN layers after each layer of the neural network and treat them as the state variation. Another interesting approach is to use the idea of recurrent convolutional neural networks [7].\n\n> *It would be good to have Chimera distinguished from WITRAN.*\n\n**Response:**\nThank you for bringing this paper to our attention. Indeed, this is a relevant paper and we will discuss it in our final version. Despite the 2D recurrence of both models, there are fundamental differences: \n1. Goal: The main goal of WITRAN is to use 2D RNN to model univariate time series data. That is, the first dimension corresponds to short-term, and the second dimension corresponds to long-term patterns. On the other hand, our 2D SSM aims to model multivariate time series where the first dimension corresponds to time, and the second dimension corresponds to variates. \n2. Backbone: WITRAN is based on GSC (LSTM-like cell), which uses non-linear transition. Our 2D SSM is based on state space models, is simpler, and uses linear transitions. \n3. Data-dependency: WITRAN is based on data-independent parameters, meaning that it uses the same transition process for all time steps. 2D SSM, however, is based on data-dependent parameters, which allows it to filter irrelevant/noisy time steps. \n4. Recurrence, Hidden States, and Training: Although both models have recurrent form, WITRAN’s recurrence is over extracted short-term and long-term patterns. Our 2D SSM’s recurrence, however, is over time and variate dimensions. WITRAN uses only one hidden state for each state of the system, while 2D SSM uses 2 different hidden states for each state of the system, allowing more flexibility to capture cross-variate and cross-time information. \n5. Seasonality: The architecture and the discretization process in Chimera is designed to capture both seasonal and trend patterns, which is different from WITRAN.\nFollowing your suggestion, we will discuss and compare these two models in more detail in the next version of our paper. \n\nWe hope our above responses have fully addressed your concerns/questions about the paper. We would be more than happy to answer any further questions or discuss any remaining concerns of the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SB0dLuJeL7", "reviewer": "Reviewer_yYuG", "summary": "The paper presents Chimera, a novel two-dimensional State Space Model (SSM) that can effectively model multivariate time series data. The model is designed to address several key challenges in multivariate time series modeling, including nonlinear dynamics along the temporal dimension, inter-variate dependencies, and efficient training and inference. The proposed discretization is original and suitable for the time series modality. Comprehensive experiments are provided to validate the effectiveness of multiple time series tasks.", "questions": "What is the lookback length in long-term forecasting (Table 1)?", "limitations": "The paper does not explicitly discuss limitations. Discussing the limitations of the model would be useful to improve the paper. I also suggest the authors provide the code implementations for a better illustration of the method.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The motivations are well-summarized and the paper's writing is easy to follow.\n* The method exhibits promising results in handling multiple time series tasks.\n* The proposed 2D SSM is novel and the bi-directional modification for variate relations makes sense.", "weaknesses": "* About several obscure writing, which is detailed as follows:\n\n    (1) The second paragraph of Section 2 introduces Multi-Dimensional State Space Models, which are stated to be different from typical multi-dimensional SSMs like Mamba. For better clarification, the authors could elaborate more on the 2D grid spanning by the (seemingly) temporal variables $t^{(1)}$ and $t^{(2)}$. I suggest avoiding the plausible notation since \"each state is a function of both time and variates\" (Line 129)\", not both temporal variables.\n\n    (2) Would the author explain the meaning of $N_i$, $\\tau_i$ in Line 93?\n\n    (3) Some overloaded notations: (1) $x$ in Line 89 (complex number) and Line 217 (input time series, real number vector); $\\mathbf{x}$  in Line 85 (real number vector) and Line 95 (complex number).\n\n    (4) Unify the usage of \"variable\" and \"variate\".\n\n    (5) Equation 8: $x \\to \\mathbf{x}$.\n\n* About the soundness of 2D SSM. The author uses 2-dimensional SSMs, where \"the first axis corresponds to the time dimension and the second axis is the variates\" (Line 128) and can be \"viewed as the modification of the discrete Roesser’s SSM\" (Line 134). I am uncertain about the soundness, which further incorporates the variable dimension. It is well-acknowledged to depict the variable relationship by correlations instead of transitions (Equation 6, Left). Even though the authors make sensible bi-directional modifications, the non-causal transitions can not directly reveal the relationship between any two variables. It is also influenced by the variable permutation. However, I think it can be addressed by providing experiments on shuffled variates.\n\n* As the author stated the Classical 2D SSMs are only able to capture linear dependencies, It could be better to compare with Koopman-based deep models [1]\\[2], which are similar to SSMs and excel at tackling nonlinear dynamics. I also encourage the author to discuss this branch in related works.\n\n* Concern about the experimens: (1) Whether the classification (anomaly detection) is performed at the series level or the point level? (2) It can be observed from Table 6 that input-independent has a great influence on the performance, could the authors provide more explanation here? (3) About Figure 6, can the author provide a comparison with PatchTST? (4) Could the author provide theoretical complexity analysis on Chimera and other SSMs/Transformers?\n\n[1] Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors.\n\n[2] Koopman Neural Operator Forecaster for Time-series with Temporal Distributional Shifts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your time and constructive review. We really appreciate it. Please see below for our response to your comments:\n\n> *About several obscure writing …*\n\n**Response:**\nThank you for bringing this to our attention. We agree with the reviewer that using a better notation can improve the clarity. We will fix this in the final version of the paper. \n $\\tau_1$ and $\\tau_2$ are used to avoid repetition of the same term but with different indices. That is, we have all $A_1, A_2, A_3, A_4, B_1, B_2, C_1$, and $C_2$ matrices and to avoid repeating the same term for all, we use these indices. Also, $N_i$s are the dimension of inputs and also the state dimension of our state space models. These dimensions, indicating the number of variables we use for summarizing the previous information. \n\n> *Some overloaded notations*\n\n**Response:**\nThank you very much for bringing them to our attention. We will make sure to address all of them in the final version. More specifically, we will consistently use variate in the paper and will use different notations for complex numbers and inputs.  \n\n> * I am uncertain about the soundness …*\n\n**Response:**\nThank you for mentioning that. Bidirectional SSMs (in 1D case) have shown promising results in non-causal data, including graphs [1], images [2], and point clouds [3]. Accordingly, we expect our 2D SSM with bidirectionality across variates to be able to capture the dependencies of all variates. To support this claim, following your suggestion, we performed an experiment and shuffled the variates. The results are provided in the attached PDF. Chimera shows robust performance when shuffling the order of variates, which supports its ability to capture the correlation of each pair of variates.  \n\n    \n**Experiments:**\nIn our experiments, we have performed anomaly detection at the point level. However, following your comment, we have added additional experiments to show the performance of Chimera in series-level anomaly detection. In this experiment, we use the anomaly detection task of the BVFC dataset. The results are reported in the attached PDF. \nWhen making parameters data-dependent, the model can use different $\\Delta_i$ as well as $A_i, B_i,$ and $C_i$ matrices for each time step and each variate. Based on Proposition 1 (similar to the discussion in lines 160-168), this data dependency allows the model to adaptively choose $\\Delta_i$. Larger $Delta_1$ means less context window and so the model focuses more on the recent time steps. Similarly, smaller $Delta_1$ means larger context window. Similarly, $A_i$, $B_i$, and $C_i$s can control the information flow across time and variates. Therefore, when they are functions of the current input, the model can select what and how the information from the past should be incorporated. We will make sure to further explain this process in the paper. As a result of this selection mechanism, we expect Chimera to perform better (or at least not worse) when we increase the lookback length. The reason is, if increasing the lookback length is not informative and is noisy, our model should learn to ignore it by using larger $\\Delta_1$. Following your suggestion, we will further explain this experiment in the final version of the paper. \n\nFollowing your suggestion, we have conducted experiments on PatchTST and will add it to this figure in the final version of the paper. PatchTST due to the patching mechanism shows more stable performance than other baselines, but still increasing the lookback damages its performance and does not show a consistent improvement. \n\nYes, following your suggestion, we will add theoretical complexity analysis on Chimera. In summary, let $T$ be the length of the time series and $V$ be the number of variates. Since for each variate we have $T$ recurrence and we have $V$ variates, the total number of recurrence is $O(TV)$. In each step, we have the matrix multiplication of transition matrices $A_i$ with the hidden states and also $B_i$s with the input. Accordingly, if we use $d$ as the state dimension, the complexity is $O(d^2 TV)$. Given the fact that $d$ is usually a small number, the overall complexity is linear with respect to each of $V$ and $T$. On the other hand, a simple Transformer considers the pairwise correlation of all time steps, which results in $O(T^2)$. Using the same procedure for all variates the overall complexity is $O(VT^2)$.\n\nPlease note that in the above theoretical complexity analysis, we consider the naive implementation, and with our proposed parallel scan, in practice Chimera is even more efficient. We will discuss this in more detail in the final version of the paper. \n\n>*What is the lookback length in long-term forecasting (Table 1)?*\n\n**Response:** Thank you for mentioning that. The lookback length is 96 for all forecasting datasets. We will mention it in the final version.\n\n\nWe hope our above responses have fully addressed your concerns/questions about the paper. We would be more than happy to answer any further questions or discuss any remaining concerns of the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Once again, we thank all the reviewers for their time and constructive reviews, which have helped us to improve the paper. 

Following the reviewer suggestions, we have conducted additional experiments, and the results are attached to this comment.  
1. In Figure 1, we visualize the found patterns by Chimera. The results show that using different 2D SSM modules with different discretization results in learning both seasonal and trend patterns. 
2. In Table 2, we perform experiments on ultra-long-range forecasting to show the significance of our Multivariate closed-loop (MCL) module. The results show that not only Chimera outperforms baselines in very-long-range forecasting tasks, but also removing its MCL module can damage the performance.
3. We provide the results of our ablation study on additional datasets. We also added two variants of All Diagonal (resp. All Companion), where we use Diagonal (resp. Companion) matrices for both transition across time and variates. The results show the importance of our parametrization.
4. In Figure 2, we report the results of 10 runs when we permute the order to variates. The results show robust performance, which validate the effectiveness of bidirectionality across variates.
5. To further motivate the importance of data-dependency across both time and variates, we perform an experiment, where we add noisy variates to the dataset. We expect Chimera to be more robust to noisy variates when using data-dependent parameters. The results are reported in Figure 3 and validate our claim. The data-dependency allows Chimera to filter irrelevant variates.
6. We further report the performance of Chimera on additional datasets. The results are reported in Table 3 and validate our initial findings in the submission.

Meta-review: This paper introduces Chimera, a novel two-dimensional State Space Model (SSM) for multivariate time series modeling. The method aims to capture complex temporal and inter-variate dependencies while providing efficient training and inference.

The reviewers generally agree that the paper should be accepted. This paper presents a novel and promising approach to multivariate time series modeling, with strong empirical results and theoretical foundations. The authors have adequately addressed reviewer concerns, and the revised version should make a valuable contribution to the field.
Decision: Accept (poster)