Effectively Modeling Time Series with Simple Discrete State Spaces
Authors: Michael Zhang
Khaled Kamal Saab
Michael Poli
Tri Dao
Karan Goel
Christopher Re
Venue: ICLR 2023 poster (2023)
Keywords: time series
forecasting
state-space models
time series classification

Abstract: Time series modeling is a well-established problem, which often requires that methods (1) expressively represent complicated dependencies, (2) forecast long horizons, and (3) efficiently train over long sequences. State-space models (SSMs) are classical models for time series, and prior works combine SSMs with deep learning layers for efficient sequence modeling. However, we find fundamental limitations with these prior approaches, proving their SSM representations cannot express  autoregressive time series processes. We thus introduce SpaceTime, a new state-space time series architecture that improves all three criteria. For expressivity, we propose a new SSM parameterization based on the companion matrix---a canonical representation for discrete-time processes---which enables SpaceTime's SSM layers to learn desirable autoregressive processes. For long horizon forecasting, we introduce a "closed-loop" variation of the companion SSM, which enables SpaceTime to predict many future time-steps by generating its own layer-wise inputs. For efficient training and inference, we introduce an algorithm that reduces the memory and compute of a forward pass with the companion matrix. With sequence length $\ell$ and state-space size $d$, we go from $\tilde{O}(d \ell)$ naïvely to $\tilde{O}(d + \ell)$. In experiments, our contributions lead to state-of-the-art results on extensive and diverse benchmarks, with best or second-best AUROC on 6 / 7 ECG and speech time series classification, and best MSE on 14 / 16 Informer forecasting tasks. Furthermore, we find SpaceTime (1) fits AR($p$) processes that prior deep SSMs fail on, (2) forecasts notably more accurately on longer horizons than prior state-of-the-art, and (3) speeds up training on real-world ETTh1 data by 73% and 80% relative wall-clock time over Transformers and LSTMs.

Published as a conference paper at ICLR 2023
EFFECTIVELY MODELING TIME SERIES WITH SIMPLE DISCRETE STATE SPACES
Michael , Khaled Saab', Michael Poli, Tri Dao, Karan Goel & Christopher Ré Stanford University
mzhang@cs. stanford. edu, . ksaabestanford.een
ABSTRACT
Time series modeling is a well-established problem, which often requires that methods (I) expressively represent complicated dependencies, (2) forecast long horizons, and (3) efficiently train over long sequences. State-space models (SSMs) are classical models for time series, and prior works combine SSMs with deep learning layers for efficient sequence modeling. However, we find fundamental limitations with these prior approaches, proving their SSM representations can- not express autoregressive time series processes. We thus introduce SPACETIME, a new state-space time series architecture that improves three criteria. For expressivity, propose
SSM parameterization based on the companion matrix-a canonical representation for discrete- time trocesses-wiid enables SPACETIME's SSM layers to learn desirable autoregressive processes. For long horizon forecasting, we introduce a "closed-loop" variation of the companion SSM, which enables SPACETIME ing its own layer-wise inputs, For efficient training and inference, we introduce an algorithm that reduces the memory and compute of forward pass with the panion matrix. With sequence length and state-space size d, we go from (de) naively Öd f). In experiments, our contributions lead to state-of-the-- re- sults on extensive and diverse benchmarks, with best or second-best best AUROC on 6 7 ECG and speech time series classification, and best MSE on 14 / 16 Informer forecasting tasks. Furthermore, we find SPACET TIME (1) fits AR(p) processes that prior deep SSMs fail on, (2) forecasts notably more accurately on longer horizons than prior state-of-ther of-the-art and (3) speeds up training worldworld ETThl data by 73% and 80% relative wall-clock time over Transformers and LSTMs.
predict many future time-steps by generat-
INTRODUCTION
Time series modeling is well-established problem, with tasks such as forecasting and classification motivated by many domains such as healthcare, finance, and engineering (Shumway et al., 2000). However, effective time series modeling presents several challenges:
First, methods should be expressive enough to capture complex, long-range, and autoregressive dependencies. Time series data often reflects higher order dependencies, seasonality, and trends, which govern how past samples determine future samples (Chatfield, 2000). This motivates many classical approaches that model these properties (Box et al., 1970; Winters, 1960), alongside pressive deep learning mechanisms such as attention (Vaswani et al., 2017) and fully connected layers that model interactions between every sample in an input sequence (Zeng et al., 2022).
Second, methods should be able to forecast a wide range of long horizons over various data domains. Reflecting real world demands, popular forecasting benchmarks evaluate methods 34 different tasks (Godahewa et al., 2021) and 940960 time-step horizons Zhou et al. (2021). Furthermore, testament accurately learning time series processes, forecasting methods should ideally also be able to predict future time-steps on horizons they were not explicitly trained on.
Finally, methods should be efficient with training and inference. Many time series applications require processing very long sequences, e.g., classifying audio data with sampling rates up to 16,000 Hz (Warden, 2018). To handle such settings-where where we still need large enough models that

---

Published as a conference paper at ICLR 2023
Input
0
0
0
40
Mgy Up
M1-1
1
0
0
01
11,
A
Octput
0
1
0
02
WI 32 . H1.
0
0
1
03
Øk+1 = A + Bug
yk+1
C2E+1
State Space Time Series
Expressive Representation
Efficient Computation
Fast modeling ss convolution
Built-in data preprocessing as forward pass
Dynamic forecasting as a recurrence
Ut
CA----BB
ge c( A
BC) 37
Works
Weather Faresasting
Forecweling
Speech Audio Classittation
Medical ECG Clessiftaatio And morel
New Architecture Broadly Applicable for Time Series
We learn time series processes as state state-space models (SSMs) (top left). We represent SSMs with the companion matrix, which is highly expressive for discrete time series (top middle). and compute such SSMs efficiently as convolutions or recurrences via a shift + low-rank decomposition (top right). We use these SSMs to build SPACETIME, new time series architecture broadly effective across tasks and domains (bottom).
Figure 1:
can expressively model this Muta--raaning and inference should idcally scale subquadratically with sequence length and model size time and space complexity.
Unfortunately, existing time series methods struggle to achieve all three criteria. Classical meth- ods (c.f., ARIMA (Box et al., 1970), exponential smoothing (ETS) (Winters, 1960)) often require manual data preprocessing and model selection to identify expressive-enough models. Deep learn- ing methods commonly train to predict specific horizon lengths, ie., as direct multi-step forecast- ing (Chevillon, 2007), and we find this hurts their ability forecast longer horizons (Sec. 4.2.2). They also face limitations achieving high expressivity and efficiency. Fully connected networks (FCNs) in Zeng al. (2022) scale quadratically in ((h)) space complexity (with input length l and forecast length h)) Recent Transformer-busee models reduce this complexity to OE h), but do not always outperform the above FCNs on forecasting benchmarks (Liu et al., 2022: Zhou al., 2021).
We thus propose SPACETIME, a deep state-space architecture for effective time series modeling. To achieve this, we focus on improving each criteria via three core contributions:
1. For expressivity, our key idea and building block is a linear layer that models time series pro- cesses as state-space models (SSMs) via the companion matrix (Fig. 1). We start with SSMs due to their connections both classical time series analysis (Kalman, 1960; Hamilton, 1994) and recent deep learning advances (Gu et al., 2021a). Classically, many time series models such as ARIMA and exponential smoothing (ETS) can be expressed SSMs (Box et al., 1970; Win- ters, 1960). Meanwhile, recent state-of of-the-ar deep sequence models (Gu et al., 2021a) have used SSMs to outperform Transformers and LSTMs on challenging long-range benchmarks (Tay et al., 2020). Their primary innovations show how to formulate SSMs as neural network pa- rameters that are practical to train. However, we find limitations with these deep SSMs for time series data. While we build on their advances, we prove that these prior SSM representations (Gu et al., bb2,,;a; Gupta, 2022) cannot capture autoregressive processes fundamental for time se- rics. We thus specifically propose the companion matrix representation for its expressive and memory-efficient properties. We prove that the companion matrix SSM recovers fundamental autoregressive (AR) and smoothing processes modeled in classical techniques such ARIMA and ETS, while only requiring (dd) memory to represent an ((P) matrix. Thus, SPACETIME inherits the benefits of prior SSM-based sequence models, but introduces improved expressivity to recover fundamental time series processes simply through its layer weights.
2. For forecasting long horizons, we introduce a new "closed-loop" view of SSMs. Prior deep SSM architectures either apply the SSM as "open-loop" (Gu et al., 2021a), where fixed-length inputs necessarily generate sane-length outputs, or use closed-loop autoregression where final layer outputs are fed through the entire network next-time-step inputs (Goel et al., 2022). We
2

---

Published as a conference paper at ICLR 2023
describe issues with both approaches in Sec. 3.2, and instead achieve autogressive forecasting in a deep network with only a single SSM layer. We do so by explicitly training the SSM layer to predict its next time-step inputs, alongside its usual outputs. This allows the SSM to recurrently generate its own future inputs that lead to desired outputs-iee. those that match an observed time series-so we can forecast over many future time-steps without explicit data inputs.
For efficiency, we introduce an algorithm for efficient training and inference with the companion matrix SSM. We exploit the companion matrix's 's structure as a "shift plus low-rank" matrix, which allows us to reduce the time and space complexity for computing SSM hidden states and outputs from (dé) to (d + E) in SSM state size d and input sequence length l.
In experiments, we find SPACETIME consistently obtains state-of-the-art or near-state-of-fhe tfrthe results, achieving best or second-bes AUROC on 6 out of 7 ECG and audio speech time series classification tasks, and best mean mean-squares squared error (MSE) on 14 out of 16 Informer benchmark fore- casting tasks (Zhou et al., 2021). SPACETIME also sets new best average ranking across 34 tasks on the Monash benchmark (Godahewa et al., 2021). We connect these gains with improvements on our three effective time series modeling criteria. For expressivity, cesses SPACETIME learns AR processes that prior deep SSMs cannot. For long horizon forecasting, SPACETIME consistently outperforms prior state-of-the-art on the longest horizons by large mar- gins. SPACETIME also generalizes better to new horizons not used for training. For efficiency, on speed benchmarks SPACETIME obtains 73% and 80% relative wall-clock speedups over parameter- matched Transformers and LSTMs respectively, when training on real-world ETTh data.
synthetic ARIMA pro-
2 PRELIMINARIES
Problem setting, We evaluate effective time series modeling with classification and forecasting tasks. For both tasks, we are given input sequences of "look-back" or "lag" time series samples U4-8tt1 (ue-é...., (4-1) E R exm for sample feature size m. For classification, we aim to classify the sequence as the true class out possible classes y. For forecasting, we aim to
correctly predict H future time-steps over a "horizon" V1,++-11 (u,, U++H-1) E R#
State-space models for time series. We build on the discrete-time state-space model (SSM), which maps observed inputs ug hidden states zk, before projecting back to observed outputs yk via
Fk+1 = Cra + Dug
Ar + Buk
(2)
where A E Raxd, B E Raxm C E Rmm xd and D E Rmm xm For now, we stick to single-input single-output conventions where m, m' = and let D 0. To model time series in the single SSM setting, we treat u and y as copies of the same process, such that
yk++ uk+1 = C(Arx + Buz)
We can thus Iearn time series SSM by treating A, B, C' as black-box parameters in neural net layer, i.e., by updating A, B, via gradient descent s.t. with input uy and state 23 at time- step k following (3) predicts U that matches the next time-step sample yk+1 Us+1. This SSM framework and modeling setup is similar to prior works (Gu et al., 2021 which adopt a similar interpretation of inputs and outputs being derived from the "same" process, e.g., for language modeling. Here we study and improve this framework for time series modeling. As extensions, in Sec. 3.1.1 show how (1) and (2) express univariate time series with the right A representation. In Sec. 3.1.2 we discuss the muulti-layer setting, where layer-specific u and y now differ, and we only model first layer inputs and last layer outputs copies of the same time series process.
3 METHOD: SPACETIME
We now present SPACETIME, a deep architecture that uses structured state-spaces for more effective time-series modeling. SPACETIME is standard multi-layer encoder-decoder sequence model, built as a stack of repeated layers that each parametrize multiple SSMs. We designate the last layer as the "decoder", and prior layers as "encoder" layers. Each encoder layer processes input time series sample as a -oouenece----eeeeeee map. The decoder layer then takes the encoded sequence repre- sentation as input and outputs prediction (for classification) or output sequence (for forecasting).
3

---

Published as a conference paper at ICLR 2023
Multi-SSM SpaceTime Layer
SpaceTime Network
Output Projection
*Closed-lopp Recurrent View (Desoding Foreccsting
Layer Architeciture
MUItI-SM Views
Stut
Maas
(1)
Outputs mixod
by FRN
Speasiine
2b+1
Companion State Spares Modil
A.
A.
A.
A,
Linetr
Layor
Lincader
piliple
(2)
C
A
B
Suese lone
Layar
Convolutiona View (Ensoding Preprozesotngl
Companion SBMs
Rpasniiva
A.B
1
Input Projection
Figure 2: SPACETIME architecture and components. (Left): Each SPACETIME layer carries weights that model multiple companion SSMs, followed optionally by nonlinear FFN. The SSMs learned in parallel (1) and computed as single matrix multiplication (2). (Right): We stack these layers into a SPACETIME network, where earlier layers compute SSMs as convolutions for fast sequence-to to to-sequence modeling and data preprocessing, while a decoder layer computes SSMs as recurrences for dynamic forecasting.
Below we expand on our contributions that allow SPACETIME to improve expressivity, long-horizon forecasting, and efficiency of time series modeling. In Sec. 3.1, we present our key building block, layer that parametrizes the companion matrix SSM (companion SSM) for expressive and autoregres- sive modeling. In Sec. 3.2, we introduce a specific instantiation of the companion SSM flexibly forecast over long horizons. In Sec. 3.3, we provide an efficient inference algorithm that allows SPACETIME to train and predict over long sequences in sub-quadratic time and space complexity.
3.1 THE MULTI-SSM SPACETIME LAYER
We discuss our first core contribution and key building block of our model, the SPACETIME layer, which captures the companion SSM's s expressive properties, and prove that the SSM represents mul- tiple fundamental processes. To scale up this expressiveness in neural architecture, we then go over how we represent and compute multiple SSMs each SPACETIME layer. We finally show how the companion SSM's expressiveness allows us to build in various time series data preprocessing oper- ations in a SPACETIME layer via different weight initializations of the same layer architecture
3.1.1 EXPRESSIVE STATE-SPACE MODELS WITH THE COMPANION MATRIX
For expressive time series modeling, our SSM parametrization represents the state matrix A as a companion matrix. Our key motivation is that A should allow to capture autoregressive relation- ships between a sample na and various past samples 1,-1, Wk 0k-2,--.3 Uk-n- Such dependencies are basic yet essential premise for time series modeling: they underlie many fundamental time series processes, e.g., those captured by standard ARIMA models. For example, consider the simplest version of this, where Ug a linear combination of p prior samples (with coefficients di...., hp)
Up = Φ1 11-1 + 42 + ... Up
(4)
i.e., noiseless, unbiased AR(p) process in standard ARIMA time series analysis (Box al., 1970).
To allow (3) to express (4), we need the hidden state Ty to carry information about past samples. However, while setting the state-space matrices as trainable neural net weights may suggest we can
leam arbitrary dask-desirable A and B via supervised learning, prior work showed this could not be done without restricting to specific classes of matrices (Gu al., 2021b; Gupta, 2022).
Fortunately, we find that class of relatively simple A matrices suffices. We propose to set A E Rdxd as the d companion matrix, a square matrix of the form:
0 0
0
ag
1
0
Q1
for i - 1=
(Companion Matrix) A
0
1
0
02
iee.
Aij
a for j d (5) otherwise
0
ad-1J
Then simply letting state dimension dl = p, assuming initial hidden state 2o 0. and setting
= |00 a1
a2-1)" 0, B
0
0"
C (0b
6r)
4

---

Published as a conference paper at ICLR 2023
allows the discrete SSM in (1, 2) to recover the AR(p) process in (4). We next extend this result in Proposition 1. proving in App. that setting A as the companion matrix allows the SSM to recover wide range of fundamental time series and dynamical system processes beyond the AR(p) process.
Proposition 1. A companion state matrix SSM can represent ARIMA (Box et al., 1970), exponential smoothing (Winters, 1960; Holt, 2004), controllable linear time-invariant systems (Chen, 1984).
As a result, by training neural network layers that parameterize the companion SSM, we provably enable these layers to learn the ground-truth parameters for multiple time series processes. In addi- tion, as we only update a E IRd (5), we can efficiently scale the hidden-state size to capture more expressive processes with only ((d) parameters. Finally, by learning multiple such SSMs in single layer, and stacking multiple such layers, we can further scale up expressivity in a deep architecture.
Prior SSMs are insufficient. We further support the companion SSM by proving that existing related SSM representations used in Gu et al. (2021a); Gupta (2022); Smith et al. (2022); Alcaraz and Strodthoff (2022) cannot capture the simple yet fundamental AR(p) process. Such works, including S4 and S4D, build on the Linear State-Space Layer (LSSL) (Gu et al., 2021b), and cannot represent AR processes due to their continuous-tin or diagonal parametrizations of A.
Proposition 2. No class of continuous-time LSSL SSMs can represent the noiseless AR(p) process. We defer the proof to App. B. Sec. 4.2.1, we support this analysis with empirical results, showing that these prior SSMs fit synthetic AR processes less accurately than the companion SSM. This suggests the companion matrix resolves a fundamental limitation in related work for time series.
3.1.2 LAYER ARCHITECTURE AND MULTI-SSM COMPUTATION
Architecture. To capture and scale up the companion SSM's expressive and autoregressive model- ing capabilities, we model multiple companion SSMs in each SPACETIME layer's weights. SPACE- TIME layers are similar to prior work such as LSSLs, with A, B, C' as trainable weights, and D added back as a skip connection. To model multiple SSMs, we add a dimension to each matrix. For SSMs per SPACETING slice in the s dimension represents an individual SSM. We thus compute outputs and hidden states in parallel by following (1) and (2) via simple matrix multiplications on standard GPUs.
layer, we specify weights A E Rxxdxd B E Rd and C' E Raxd Each
To model dependencies across individual SSM outputs, we optionally follow each SPACETIME layer with one-layer nonlinear feedforward network (FFN). The FFN thus mixes the m outputs across a SPACETIME layer's SSMs, allowing subsequent layers to model dependencies across SSMs.
Computation. To compute the companion SSM, we could use the recurrence in (1), However, this sequential operation slow on modern GPUs, which parallelize matrix multiplications. Luckily, as described in Gu et al. (2021a) we can also compute the SSM as a 1-D convolution. This enables parallelizable inference and training. To see how, note that given a sequence with at least & inputs and hidden state zg 0, the hidden state and output at time-step & by induction are:
k-1
3% Σ
A
Buy and Vk > CA BBj
(6)
We can thus compute hidden state og and output yv as 1-D convolutions with "filters" as F = =B, AB, B A 4 B)
(Hidden State Filter)
(7)
FD (CB.CBB, CA
CA B)
(Output Filter)
(8)
Ig = (F" [kk] and yk = =FF= [k(k]
(9)
So when we have inputs available for each output (i.e., equal-sized input and output sequences) we can obtain outputs by first computing output filters F9 (8), and then computing outputs efficiently with the Fast Fourier Transform (FFT). We thus compute each encoder SSM as a convolution.
For now we note two caveats. Having inputs for each output is not always true, e.g., with long hori- zon forecasting. Efficient inference also importantly requires that F can be computed efficiently, but this is not necessarily trivial for time series: may have long input sequences with large k.
Fortunately we later provide solutions for both. In Sec. 3.2, we show how to predict output samples many time-steps ahead of our last input sample via "closed-loop" forecasting SSM. In Sec. 3.3 we show how to compute both hidden state and output filters efficiently over long sequences via an efficient inference algorithm that handles the repeated powering of A
5

---

Published as a conference paper at ICLR 2023
3.1.3 BUILT-IN DATA PREPROCESSING WITH COMPANION SSMS
We now show how beyond autoregressive modeling, the companion SSM also enables SPACETIME layers to do standard data preprocessing techniques used to handle nonstationarities. Consider dif- ferencing and smoothing, two classical techniques to handle nonstationarity and noise:
4/4 = ug - u (Ist-order differencing) u/4
Uk- (n-order moving average smoothing)
We explicitly build these preprocessing operations into a SPACETIME layer by simply initializing companion SSM weights. Furthermore, by specifying weights for multiple SSMs, we simultane- ously perform preprocessing with various orders in one forward pass. We do by setting a = 0 and B = [1 0, 01.0...., oj* such that SSM outputs via the convolution view (6) are simple sliding windows / 1-D convolutions with filter determined by C. We can then recover arbitrary n-order differencing or average smoothing via C' weight initializations, egg., (see App. D.7.1 for more examples),
1 1/n
2
0
0
0
(2nd-order differencing)
C=
(10)
1/n
0
(n-order moving average smoothing)
3.2
LONG HORIZON FORECASTING WITH CLOSED-LOOP SSMs
We now discuss our second core contribution, which enables long horizon forcasting, Using a slight variation of the companion SSM, we allow the same constant size SPACETIME model to forecast over many horizons. This forecasting SSM recovers the flexible and stateful inference of RNNs, while retaining the faster parallelizable training of computing SSMs as convolutions.
Challenges and limitations. For forecasting, model must process an input lag sequence of length l and output a forecast sequence of length h, where * necessarily. Many state-of-the-arr neural nets thus train by specifically predicting h-long targets given ê-long inputs. However, in Sec. 4.2.2 we find this hurts transfer to new horizons in other models, as they only train to predict specific horizons. Alternatively, we could output horizons autoregressively through the network similar to stacked RNNs as in SASHIMI (Goel et al., 2022) or DeepAR (Salinas et al., 2020). However, we find this still be relatively inefficient, requires passing states to each layer of a deep network.
Closed-loop SSM solution. Our approach is similar to autoregression, but only applied at a single SPACETIME layer. We treat the inputs and outputs distinct processes in a multi-layer network, and add another matrix K to each decoder SSM to model future imput time-steps explicitly. û (No,...,, me-1) be the input sequence to a decoder SSM and input sequence, we jointly train A, B, C, K such that Øb+1 = A + Büs, and
(uo...., ue-1) be the original
ÿk+11 C2E+1
(where Úk+1 yk+1 uk+1) (where u ux+1)
(11)
KEst1
(12)
We thus train the decoder SPACETIME layer to explicitly model its own next time-step inputs with A, B, K, and model its next time-step outputs (i.e., future time series samples) with A, B, C. For forecasting, we first process the lag terms via (11) and (12) as convolutions
3 = E -- -uj and ân = K Σ A 1-1j Bū,
(13)
for ls E E --11 1]. To forecast h future time-steps, with last hidden state xe we first predict future input de via (12). Plugging this back into the SSM and iterating for h future time-steps leads to
Jets = (A BK)'x, for =
(14)
(ye, W14h-1) (C(A+ BK) ( tE(k-1]
(15)
We can thus use Eq. 15 to get future outputs without sequential recurrence, using the same FFT operation as for Eq. 8, This flexibly recovers O + h) time complexity for forecasting h future time-steps, assuming that powers (A + BK)" are taken care of. Next, we derive efficient matrix powering algorithm to take care of this powering and enable fast training and inference practice.
3.3 EFFICIENT INFERENCE WITH THE COMPANION SSM
We finally discuss our third contribution, where we derive an algorithm for efficient training and inference with the companion SSM. To motivate this section, we note that prior efficient algorithms

---

Published as a conference paper at ICLR 2023
Algorithm 1 Efficient Output Filter F" Computation
Require: A is companion matrix parameterized by the last column a E R. B B C(I A R", sequence length é.
I: Define quad(u, v) E R'' for vectors u, E Re: compute q u*u (linear convolution). zero-pad to length
f {[d/E]. split into [d/e] chunks of size l of the form "d",....g(((())) and return the length- Fourier
transform of the sum Fr(g(t) + ........ + (fd//(D)
2: Compute the roots of unity= z = [ ...... w] where w = exp(-2211)
3: Compute F" quad(C, B) + quad a) quad(es. B)/(2 quad(ea, a)) E R', where ea
[0.... 1] is the d-th basis vector.
4: Return the inverse Fourier transform FV F. (P)).
to compute powers of the state matrix A were only proposed to handle specific classes of A, and do not apply to the companion matrix (Gu et al., 202 la; Goel et al., 2022; Gu et al., 2022).
Recall from Sec. 3.1.2 that for sequence of length f, we want to construct the output filter F = (CBB...,, ..a...,,A--11 CA where A is a d x d companion matrix and B, C. are d x and 1 x d matrices. Naively, we could use sparse matrix multiplications compute powers CA B for j = 0, ...... 6-1 sequentially. As has (dd) nonzeros, this would take (ld) time. We instead derive an algorithm that constructs this filter in Ofllog& E + d dolog d) time. The main idea is that rather than computing the filter directly, we can compute its spectrum (its discrete Fourier transform) more easily, i.e.,
FM [m]] == F(F))= = E5=o wm B = C(I - A²)(I )
m = 0, 1, 1,...,-11
where a' exp(-2xi/) is the ê-th root of unity. This reduces to computing the quadratic form of the resolvent (I Aw on the roots of unity (the powers of w). Since A is a companion matrix, we can write A as shift matrix plus a rank-l matrix, A = S + ae Thus Woodbury's formula reduces this computation to the resolvent of shift matrix (I - Sw ) with rank-1 correction. This resolvent can be shown analytically to be a lower-triangular matrix consisting of roots of unity, and its quadratic form can be computed by the Fourier transform of linear convolution of size d. Thus one can construct Fo by linear convolution and the FFT, resulting in O log l + d log d) time.
We validate in Sec. 4.2.3 that Algorithm 1 leads to a wall-clock time speedup of 2 compared to computing the output filter naively by powering A. In App. B.2, we prove the time complexity (l log & + d log d) and correctness of Algorithm 1. We also provide extension to the closed- SSM, which can also be computed in subquadratic time as A + B K is a shift plus rank-2 matrix.
EXPERIMENTS
We test SPACETIME on a broad range of time series forecasting and classification tasks. In Sec. 4.1, evaluate whether SPACETIME' contributions lead state-of-the-art results standard bench- marks. To help explain SPAcETIMe'' performance and validate our contributions, in Sec. 4.2 we then evaluate whether these gains coincide with empirical improvements in expressiveness (Sec. 4.2. 1), forecasting flexibility (Sec. 4.2.2), and training efficiency (Sec. 4.2.3).
4.1 MAIN RESULTS:
TIME SERIES FORECASTING AND CLASSIFICATION
For forecasting, we evaluate SPACETIME on 40 forecasting tasks from the popular Informer (Zhou et al., 2021) and Monash (Godahewa et al., 2021) benchmarks, testing on horizons steps long. For classification, we evaluate SPACETIME on seven medical ECG or speech audio classification tasks, which test on sequences up to 16,000 time-steps long. For all results, we report mean evaluation metrics over three seeds. X denotes the method was computationally infeasible on allocated GPUs, e.g., due to memory constraints (same resources for all methods; see App. for details). App. C also contains additional dataset, implementation, and hyperparameter details.
960 time-
Informer (forecasting). We report univariate time series forecasting results in Table 1, comparing against recent state-of-the-an methods (Zeng et al., 2022; Zhou et al., 2022a), related state-space models (Gu et al., 202 1a), and other competitive deep architectures. We include extended results on additional horizons and multivariate forecasting in App. D.2. We find SPACETIME obtains lowest MSE and MAE on 14 and 11 forecasting settings respectively, 3x more than prior -tate-of-thh-are SPACETIME also outperforms S4 on 15 / 16 settings, supporting the companion SSM representation.
Monash (forecasting). We also evaluate on 32 datasets in the Monash forecasting benchmark (Go- dahewa et al., 2021), spanning domains including finance, weather, and traffic. For space, we report
7

---

Published as a conference paper at ICLR 2023
Table 1: Univariate forecasting results on Informer ETT datasets, Best results in bold. SPACETIME results reported as means over three seeds. Additional datasets, horizons, and method comparisons in App. D.2
Methods
SpaceTime
NLinear
FILM
S4
FedFormer
Autoformer
Informer
ARIMA
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
96
0.084
0.181
0.053
0.177
0.055
0.178
0.316
0.490
0.079
0.215
0.071
0.200
0.193
0.377
0.058
0.184
B7712 BTTbl
0.066
0.207
0.069
0.204
0.345
0.245
0.262
0.073
336
0.069
0.212
0.081
0.226
0.083
0.229
0.825
0.846
0.119
0.270
0.107
0.258
0.202
0.381
0.086
0.231
720
0.076
0.222
0.080
0.226
0.090
0.240
0.190
0.355
0.142
0.299
0.126
0.283
0.183
0.355
0.103
0.253
0.119
0.268
0.278
0.127
0.272
0.381
0.128
0.271
0.153
0.306
0.373
0:273
192
0.151
0.306
0.169
0.324
0.182
0.335
0.352
0.458
0.185
0,330
0.204
0.351
0.227
0.387
0.315
0.446
3356
0.169
0.332
0.194
0.355
0.204
0.367
0.655
0.670
0.231
0.378
244
0.389
0:242
0,401
0.367
0.488
720
0.188
0.352
0.225
0.381
0.241
0.396
0.630
0.662
0.278
0.420
0.268
0.409
0.291
0.439
0.413
0.519
96
0.026
0.121
0.026
0.122
0.029
0.127
0.651
0.733
0.033
0,140
0.056
0,183
0.109
0.277
0.033
0.136
BTTa22 ETTml
0.039
0.152
0.039
0.149
0.041
0.190
0.058
0.186
0.216
0.151
0.310
0.049
0,169
336
0.051
0.173
0.052
0.172
0.053
0.175
0.428
0.581
0.084
0.231
0.076
0,218
0.427
0.591
0.065
0.196
720
0.074
0.213
0.073
0.207
0.071
0.205
0.254
0.433
0.102
0,250
0.110
0.267
0.438
0,586
0.089
0,231
0.060
0.179
0.182
0.065
0:153
0.318
0.067
0.198
0.189
0.211
192
0.090
0.222
0.090
0.223
0.094
0.233
0.183
0.350
0.102
0.245
0.118
0.256
0.132
0.283
0.237
0,371
336
0.113
0.255
0.117
0.259
0.124
0,274
0.204
0.367
0.130
0.279
0.1
0.305
0.180
0.336
0.264
0.396
720
0.166
0.318
0.170
0.318
0.173
0.323
0.482
0.567
0.178
0.325
0.
0.335
0.300
0.435
0:310
0.441
Count
14
11
4
0
0
0
0
Table 2: ECG statement classification on PTB-XL (100 Hz version). Base-
Table 3: Speech audio classification
line AUROC from Strodthoff et al. (2021) (error bars in App. D.4).
Task AUROC
All
Diag
Sub-diag
Super-diag
Form
Rhythm
Method
Acc. (%)
SPACETIME
0.936
0.941
0.933
0,929
0.883
0.967
SPACETIME
97.29
S4
0.938
0.939
0,929
0,931
0.895
0.977
S4
98.32
Inception-1D ID
0.925
0.931
0,930
0,921
0.899
0,953
LSSL
X
XRN-101
0.925
0.937
0,929
0.928
0.896
0.957
WaveGan-D
96,25
LSTM
0.907
0,927
0.928
0,927
0,851
0.953
Transformer
X
Transformer
0.857
0.876
0.882
0.887
0.771
0.831
Performer 30.77
Wavelet + NN
0.849
0,855
0.859
0.874
0.757
0.890
CKConv 71.66
results in Table 20 (App. D.3). We compare against 13 classical and deep learning baselines. SPACE-
TIME achieves best RMSE on 7 tasks and sets new state-of-the-art average performance across all 32 datasets. SPACETIME's relative improvements also notably grow on long horizon tasks (Fig. 6).
ECG (multi-label classification). Beyond forecasting, we show that SPACETIME can also perform thete-r time series classification. To classify sequences, we use the same sequence model architecture in Sec. 3.1. Like prior work (Gu et al., 2021a), we simply use the last-layer FFN to project from number of SSMs to number of classes, and mean pooling over length before softmax to output class logits. In Table 2, we find that SPACETIME obtains best or second-best AUROC on five out of six tasks, outperforming both general sequence models and specialized architectures.
Speech Audio (single-label classification). We further test SPACETIME on long-range audio clas- sification on the Speech Commands dataset (Warden, 2018). The task is classifying raw audio sequences of length 16,000 into 10 word classes. We use the same pooling operation for classifica- tion as in ECG. SPACETIME outperforms domain-specific architectures, e.g., WaveGan-D (Donahue et al., 2018) and efficient Transformers, e.g., Performer (Choromanski et al., 2020) (Table 3).
4.2 IMPROVEMENT ON CRITERIA FOR EFFECTIVE TIME SERIES MODDELIGG
For further insight into SPACETIME's performance, we now validate that our contributions improve expressivity (4.2.1), forecasting ability (4.2.2), and efficiency (4.2.3) over existing approaches.
4.2.1 EXPRESSIVITY
To first study SPACETIME's expressivity, we test how well SPACETIME can fit controlled autoregres- sive processes. To validate our theory on SPACETIME's expressivity gains in Sec. 3.1, we compare against recent related SSM architectures such as S4 (Gu et al., 2021a) and S4D (Gu et al., 2022).
For evaluation, we generate noiseless synthetic AR(p) sequences. We test if models learn the true process by inspecting whether the trained model weights recover transfer functions specified by the AR coefficients (Oppenheim, 1999). We use simple I-layer 1-SSM models, with state-space size equal to AR p. and predict one time-step given p lagged inputs (the smallest sufficient setting).
In Fig. 3 we compare the trained forecasts and transfer functions (as frequency response plots) of SPACETIME, $4, and S4D models relatively smooth AR(4) process and sharp AR(6) process.
8

---

synthetic Published as a conference paper at ICLR 2023
Seaating
Grourd-trath SpaceTime
6-T
S4D
54D
Frequency (w)
Frequency (w)
(a) AR(4) Forecast
(b) AR(4) Transfer Func.
(c) AR(6) Forecast
(d) AR(6) Transfer Func.
Figure 3: AR(p) expressiveness benchmarks. SPACETIME captures AR processes more preciscly than simi-
lar deep SSM models, forecasting future samples and learning ground-truth transfer functions accurately.
Table 4:
Longer horizon forecasting on Informer ETTh datasets. Mean standardized MSE reported. SPACETIME ob- tains lower MSE when trained forecast longer horizons.
MJnoar
Nireea
Soace
SpaceTirus
0.3
Dataset Horizon
720
1080 1440
1920
ETTh1 NLinear
0.080
0.089
0.085
0.094
0.102 0.104
SPACETIME 0.075
0.074
0.072
0.080
0.081
0.088
Taxt Horican
Tesk Ro: kpo
Figure 4: Forecasting transfer. Mean MSE (11 SD). SPACETIME transfers more accurately and consistently new horizons than NLincar.
NLincar
0,224 0.273 0.290
0.329 0,450 0.493
ETTh2
SPACETIME 0.188 0.225 0.265 0.299 0.438 0.459
Our results support the relative expressivity of SPACETIME's companion matrix SSM. While all models accurately forecast the AR(4) time series, only SPACETIME recovers the ground-truth trans- fer functions for both, and notably forecasts the AR(6) process more accurately (Fig. 3c, d).
4.2.2 LONG-HORIZON FORECASTING
To next study SPACETIME's improved long horizon forecasting capabilities, we consider addi- tional long horizon tasks. First, we test on much longer horizons than prior settings (c.f., Table I). Second, we test a new forecasting ability: how well methods trained forecast one horizon transfer to longer horizons at test-time. For both, we use the popular Informer ETTh datasets. We compare SPACETIME with Nhinear-the prior state-of-the-art on longer-horizon ETTh datasets-an FCN that learns dense linear mapping between every lag input and horizon output (Zeng et al., 2022).
We find SPACETIME outperforms NLinear on both long horizon tasks. training to predict long horizons, SPACETIME consistently obtains lower MSE than NLinear all settings (Table 4). On transferring to new horizons, SPACETIME models trained to forecast 192 time-step horizons transfer more accurately and consistently to forecasting longer horizons up to 576 time-steps (Fig. 4). This suggests SPaceTime's autoregressive forecasting more convincingly learns the time series process; rather than only fitting the specified horizon, the same model can generalize new horizons.
4.2.3 EFFICIENCY
To finally study if our companion matrix algorithm enables efficient training on long sequences, we conduct two speed benchmarks. We (1) compare the wall-clock time per training epoch SPACETIME to standard sequence models, e.g., LSTMs and Transformers, with similar pararemeter counts, and (2) empirically test our theory Sec. 3.3, which suggests SPACETIME trains near-linearly with se- quence length and state dimension. For (1), we use ETTh data with lag and horizon 720 time-steps long. For (2),
Table 5: Train wall-clock time.
ETTh1 seconds per epoch.
Method
# params sec/epoch
SPACETIME
148k
66
Algorithm
148k
132
S4
151k
49
Transformer
155k
LSTM
145k
336
data, scaling sequences from 100-2000 time-steps long.
Epoch wall cark timee
Spac
On (1) we find SPACETIME reduces clock time ETThl by 73% and 80% compared to Transformers and LSTMs (Table 5). Our efficient algorithm (See. 3.3) is also important; it speeds up train- ing by 2x, and makes SPACETIME's training time competitive with efficient models such S4. On (2), we find SPACETIME also scales near- linearly with input sequence length, achieving 91% faster training time versus similarly recurrent LSTMs (Fig. 5).
LSTM
Trnnsormer
20 520 d0 1000 sequenre length
1350 1000 Li0 2000
Figure 5: Wall-clock scaling. SPACETIME scales near-linearly.
CONCLUSION
We introduce SPACETIME, a state-space time series model. We achieve high expressivity by mod- eling SSMs with the companion matrix, long-horizon forecasting with closed-loop SSM variant, and efficiency with a new algorithm to compute the companion SSM. We validate SPACETIME's proposed components extensive time series forecasting and classification tasks.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
main objective our work is to improve the ability to classify and forecast time series, which has real-world applications in many fields. These applications may have high stakes, such classifying abnormalities in medical time series. In these situations, incorrect predictions may lead to harmful patient outcomes. It is thus critical to understand that while we aim improve time series modeling towards these applications, we do not solve these problems. Further analysis and development into where models fail in time series modeling is necessary, including potentials intersections with research directions such as robustness and model biases when aiming to deploy machine learning models in real world applications.
REPRODUCIBILITY
We include code reproduce main results in Table 1 in the supplementary material. We pro- vide training hyperparameters and dataset details for each benchmark in Appendix C, discussing the Informer forecasting benchmark in Appendix C.I, the Monash forecasting benchmark in Ap- pendix C.2, and the ECG and speech audio classification benchmarks in Appendix C.3. We provide proofs for all propositions and algorithm complexities in Appendix B.
ACKNOWLEDGEMENT
We thank Albert Gu, Yining Chen, Ke Alexander Wang, and Rose Wang for helpful discussions and feedback. We also gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume Velocity), and 1937301 (RTML); US DEVCOM ARL under No. W911NF-21-2-0251 (Interactive Human-AI Teaming); ONR under No. N000141712266 (Unifying Weak Supervision); ONR N00014-20-1-24800 Un- derstanding and Applying Non-Euclidean Geometry in Machine Learning: N000142012275 (NEP- TUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC. Toshiba, TSMC, ARM, Hitachi, BASE, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI- GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Facebook, Google, and VMWare. The U.S. Government is autho- rized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those the authors and do not necessarily reflect the views, policies, or endorsements, either expressed implied, of NIH, ONR, the U.S. Government.
REFERENCES
J. M. I.. Alcaraz and N. Strodthoff. Diffusion-based time series imputation and forecasting with structured state space models. arXiv preprint arXiv:2208.09399 2022.
M. Amini, F. Zayeri, and M. Salehi. Trend analysis of cardiovascular disease mortality, incidence, and mortality-to-incidene ratio: results from global burden of disease study 2017. BMC Public Health, 21 2021.
V. Assimakopoulos and K. Nikolopoulos. The theta model: decomposition approach to forecast- ing. International journal of forecasting. 16(4):521-530, 2000.
Z. I. Attia, S. Kapa, F. Lopez-Jimenez, P. M. McKie, D. J. Ladewig, G. Satam, P. A. Pellikka, M. Enriquez- Enriquez-Sarano, Sarano, P. A. Noseworthy, T. M. Munger, et al. Screening for cardiac contractile dysfunction using artificial intelligence-enate enabled electrocardiogram. Nature medicine, 25(1): 70-74, 2019a.
Z. I. Attia, P. A. Noseworthy, F. LLopz-Jimenezz S. Asirvatham, A. J. Deshmukh, B. Gersh, R. E. Carter, X. Yao, A. A. Rabinstein, J. Erickson, al. An artificial intelligence-enabled eog algo- rithm for the identification of patients with atrial fibrillation during sinus rhythm: retrospective analysis of outcome prediction. The Lancet, 394(10201) (0100)):6118888 2019b.
I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.00 05000 2020.
10

---

Published as a conference paper at ICLR 2023
G. E. Box and G. M. Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Sociery. Series C (Applied Statistics), 17(2):91-109, 1968.
G. E. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 1970.
R. G. Brown. Statistical forecasting for inventory control. 1959.
C. Chatfield. Time-series forecasting. Chapman and Hall/CRC, 2000.
C.-T. Chen, Linear system theory and design. Saunders college publishing, 1984.
R. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equa- tions. Advances in neural information processing systems, 31, 2018.
G. Chevillon. Direct multi-step estimation and forecasting. Journal of Economic Surveys, 21 (4):746-785, 2007. doi: https//bbi.gg//0.111144446---1
URL https://
onlinelibrare wiley. com/ doi abaaaasss10000 wbtpa/...s./.................l.e 00144666661120777111111
K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, et al. Rethinking attention with performers. arXiv preprini arXiv:2009. 14794, 2020.
Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint 1rXiv:1412.3553 2014.
D. A. Cook, S.-Y. Oh, and M. V. Pusic. Accuracy of physicians' electrocardiogram interpretations: systematic review and meta-analysis. JAMA internal medicine, (80(11):1461-1 2020.
W. Culver. On the existence and uniqueness the real logarithm matrix. Proceedings of the American Mathematical Society, 17(5): 146-111461111,,,, 1966.
A. M. De Livera, R. J. Hyndman, and R. D. Snyder. Forecasting time series with complex seasonal patterns using exponential smoothing. Journal of the American statistical association, 106(496): 1513-1527, 2011.
C. Donahue, J. McAuley, and M. Puckette. 18Xiv:1802.042288, 2018.
Adversarial audio synthesis. arXiv preprint
A. V. Dorogush, V. Ershov, and A. Gulin. Catboost: gradient boosting with categorical features support. arXiv preprint arXiv: 13363, 2018.
K.-i. Funahashi and Y. Nakamura. Approximation of dynamical systems by continuous time recur- rent neural networks. Neural networks, 6(6):801-806, 1993.
E. S. Gardner Jr. Exponential smoothing: 1985.
The state of the art Journal of forecasting, 4(1):1-28,
R. Godahewa, C. Bergmeir, G. I. Webb, J. Hyndman, and P. Montero-Manso. Monash time series forecasting archive. arXiv preprint arXiv:2105.06644 2021.
K. Goel, A. Gu, C. Donahue, and C. Ré. It's raw! audio generation with state-space models. arXiv preprint arXiv:2202.097733 2022.
A. L. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. Moody, C.-K. Peng, and H. F. Stanley. PhysioBank, PhysioToolkit, and PhysioNet. Circulation, 101(23):e11--e220, 2000. doi: 10. (661001..II. 101. e21...
S. Goto, K. Mahara, L. Beussink-Nelson H. Ikura, Y. Katsumata, J. Endo, H. K. Gaggin, S. J. Shah, Y. Itabashi, C. A. MacRae, et al. Artificial intelligence-enabled fully automated detection of cardiac amyloidosis using electrocardiograms and echocardiograms. Nature communications, 12(1):1-12, 2021.
A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2 1.00966 2021a.
11

---

Published as a conference paper at ICLR 2023
A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. Ré. Combining recurrent, convolu- tional, and continuous-tin models with linear state space layers. Advances in neural information processing systems, 34:572-588 2021b.
A. Gu, A. Gupta, Goel, and C. Ré. the parameterization and initialization of diagonal state space models. arXiv preprint arXiv:2206. 11893, 2022.
A. Gulati, Qin, C.-C. Chiu, N. Parmar, Y. Zhang. J. Yu, Han, S. Wang. Z. Zhang, Wu, et al. Conformer: Convolution-angnennt transformer for speech recognition. arXiv preprint arXiv:2005.00 10000 2020.
A. Gupta. Diagonal state spaces are as effective as structured state spaces. arXiv:2203. 14343, 2022.
arXiv preprint
J. D. Hamilton. State-space space models. Handbook of econometrics, 4:3039-300 -000,, 1994.
A. Y. Hannun, P. Rajpurkar, M. Haghpanahi, G. H. Tison, C. Bourn, M. P. Turakhia, and A. Y. Ng. Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using deep neural network. Nature medicine, 25(1):65-68 2019.
C. Hawthorne, A. Jaegle, C. Cangea, Borgeaud, C. Nash, M. Malinowski, S. Dieleman, O. Vinyals, M. Botvinick, I. Simon, al. Gereral-purpose, long-context autoregressive mod- eling with perceiver ar. arXiv preprint arXiv:2202.07765. 2022.
H. P. Hirst and W. T. Macey. Bounding the roots of polynomials. The College Mathematics Journal, 28(4):292-29) 295, 1997.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 93)51733- 1997.
C. C. Holt. Forecasting seasonals and trends by exponentially weighted moving averages. Interna- tional journal of forecasting, 20(1):5-10, 2004.
R. Hyndman, A. B. Koehler, J. K. Ord, and R. D. Snyder. Forecasting with exponential smoothing: the state space approach. Springer Science Business Media, 2008
R. J. Hyndman and G. Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
R. S. Jablonover, E. Lundberg, Y. Zhang, and A. Stagnaro- -Green Competency in electrocardiogram interpretation among graduating medical students. Teaching and Learning in Medicine, 26(3): 279-284, 2014. doi: 10. 000///000133.201100 10401334 2014 4. 918882.
URL https: http://ccooorroooo 1080/
PMID: 25010240.
R. E. Kalman. A approach to linear filtering and prediction problems. 1960.
Kidger. On neural differential equations. arXiv preprint arXiv:2202.02453: 2022.
P. Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-6707, 2020.
S. Kim, W. Ji. S. Deng, Y. Ma, and C. Rackauckas. Stiff neural ordinary differential equations. Chaos: An Interdisciplinary Journal of Nonlinear Science, (1(9):093122, 2021.
N. Kitaev, t. Kaiser, and A. Levskaya. Reformer: The efficient transformer. arXiv preprint
MXXi:2000100451,, 2020.
M. Liu, A. Zeng, Z. Xu, Q. Lai, and Q. Xu. Time series is a special sequence: Forecasting with sample convolution and interaction. arXiv preprint (xx::2106600900) 2021.
Liu, H. Yu, C. Liao, Li, W. Lin, A. X. Liu, and S. Dustdar. Pyraformer: Low-complexity pyramidal attention for long- rong-range time series modeling and forecasting. In International Confer- ence on Learning Representations, 2022. URL https:/ /ttp::////////eeeee/ net forum?ie id 5Un5..
12

---

Published as a conference paper at ICLR 2023
1. Loshchilov and F. Hutter. arXiv:1711.05101, 2017.
Decoupled weight decay regularization. arXiv preprint
K. Madhusudhanan, J. Burchert, N. Duong-Trung, S. Born, and L. Schmidt-Thiems Yformer: U-net inspired transformer architecture for far horizon time series forecasting. arXiv preprint arXiv:21 2021.
S. Massaroli, M. Poli, S. Sonoda, T. Suzuki, J. Park, A. Yamashita, and H. Asama. Differentiable multiple shooting layers. Advances in Neural Information Processing Systems, 34:16532-16544, 2021.
J. Morrill, C. Salvi, P. Kidger, and J. Foster. Neural rough differential equations for long time series. In International Conference on Machine Learning, pages 7829-7838. PMLR, 2021.
A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: wtt.:/.09909999 2016.
A. V. Oppenheim. Discrete-time signal processing. Pearson Education India, 1999.
B. N. Oreshkin, D. Carpov, N. Chapados, and Y. Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint urXiv: /0437, 2019.
R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318. PMLR, 2013.
M. Poli, S. Massaroli, J. Park, A. Yamashita, H. Asama, and Park. Graph neural ordinary differ- ential equations. arXiv preprint 19X 2019.
A. F. Queiruga, N. B. Erichson, D. Taylor, and M. W. Mahoney. Continous-in-depth neural net- works. arXiv preprint arXiv:2008.023:: 2020.
Rangapuram, M. Seeger, Gasthaus, L. Stella, Y. Wang. and Januschowski. Deep state space models for time series forecasting. In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS' 18, page 7796-7805, Red Hook, NY, USA, 2018. Curran Associates Inc.
A. H. Ribeiro, M. H. Ribeiro, G. M. Paixão, D. M. Oliveira, P. R. Gomes, A. Canazart, M. P. Ferreira, C. R. Andersson, P. W. Macfarlane, W. Meira Jr, et al. Automatic diagnosis of the 12-lead ecg using a deep neural network. Nature communications, 11(1):1-9, 2020.
Y. Rubanova. T. Chen, and D. K. Duvenaud. Latent ordinary differential equations for irregularly- sampled time series. Advances in neural information processing systems, 32, 2019.
D. Salinas, V. Flunkert, J. Gasthaus, and T. Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1 181-1191, 2020.
C. Secretary. Health informatics standard communication protocol - part 91064: Computer- assisted electrocardiograply. 2009.
R. H. Shumway, S. Stoffer, and D. S. Stoffer. Time series analysis and its applications, volume 3.
Springer, 2000.
K. C. Siontis, A. Noseworthy, I. Attia, and P. A. Friedman. Artificial intelligence-enhaneed electrocardiography in cardiovascular disease management. Nature Reviews Cardiology, 18(7): 465-478, 2021.
Smith, A. Warrington, and W. Linderman. Simplified state space layers for sequence model- ing. arXiv preprint ar0iv:2208. htt.:2208.09333... 2022.
G. Strang and T. Nguyen. Wavelets and filter banks. SIAM, 1996.
N. Strodthoff, P. Wagner, T. Schaeffter, and W. Samek. Deep learning for eeg analysis: Benchmarks and insights from ptb-xl. IEEE Journal of Biomedical and Health Informatics, 25(5):1519-1522 2021. doi: 10. H..0//BH..200000229999
13

---

Published as a conference paper at ICLR 2023
Y. Tay, M. Dehghami, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang. S. Ruder, and D. Met- zler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.040000 2020.
R. Trapero, N. Kourentzes, and R. Fildes. On the identification of sales forecasting models in the presence of promotions. Journal of the operational Research Society, 66(2):299-307, 2015.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Wagner, N. Strodthoff, R.-D. Bousseljot, D. Kreiseler, I. Lunze, W. Samek, and Scha- effter. PTB-XL, large publicly available electrocardiography dataset. Scientific Data, (1):154, 2020a. doi: 1010088441197-000-449--666 URL https: htapps///doo /ow.ooog/...1000000 441597-020-04955
Wagner, N. Strodthoff, R.-D. Bousseljot, W. Samek, and T. Schaeffter. PTB-XL, a large publicly available clectrocardiography dataset, 2020b.
P. Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint 2018.
E. Weinan, A proposal on machine learning via dynamical systems. Communications in Mathemat- ics and Statistics, 1(5):111,, 2017.
R. Winters. Forecasting sales by exponentially weighted moving averages. Management science, 6(3):324-342, 1960.
G. Woo, C. Liu, D. Sahoo, A. Kumar, and S. Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381, 2022.
H. Wu, J. Xu, J. Wang, and M. Long. Autoformer: Decomposition transformers with auto- correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-2244 2021.
A. Zeng, M. Chen, L. Zhang, and Q Xu. Are transformers effective for time series forecasting? arXiv preprint 2rX1v:2205.33504 2022.
H. Zhang, Z. Wang, and D. Liu. comprehensive review of stability analysis of continuous-time recurrent neural networks. IEEE Transactions Neural Networks and Learning Systems, 25(7): 1229-1262, 2014.
H. Zhou, S. Zhang, Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 1 11106-11115, 2021.
T. Zhou, Z. Ma, Q. Wen, L. Sun, T. Yao, R. Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. arXiv preprint 2xXi:22005.08897, 2022a.
Zhou, Z. Ma, Q Wen, X. Wang, L. Sun, and R. Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting, arXiv preprint arXiv:2201.127444 2022b.
14

---

Published as a conference paper at ICLR 2023
Effectively Modeling Time Series with Simple Discrete State Spaces
Table of Contents
Related Work
Classical Approaches
A.2 Deep Learning Approaches
15
B Proofs and Theoretical Discussion
17
B.1 Expressivity Results
17
B.2 Efficiency Results
19
B.3 Companion Matrix Stability
22
Experiment Details
22
Informer Forecasting
22
C.2 Monash Forecasting
23
C33 Time Series Classification
23
D Extended experimental results
24
D.1 Expressivity on digital filters
24
D.2 Informer Forecasting
24
D.3 Monash Forecasting
25
D.4 ECG Classification
26
D.5 Efficiency Results
26
D.6 SPaceTime Ablations
27
D.7 SPACETIME Architectures
28
A RELATED WORK
A.1 CLASSICAL APPROACHES
Classical approaches in time series modeling include the Box-Jenkins method (Box and Jenkins, 1968), exponential smoothing (Hyndman al., 2008; Winters, 1960), autoregressive integrated moving average (ARIMA) (Box et al., 1970), and state-space models (Hamilton, 1994). In such approaches, the model usually manually selected based analyzing time series features (e.g., sea- sonality and order of non-stationarity), where the selected model is then fitted for each individ- ual time series. While classical approaches may be more interpretable than recent deep learning techniques, the domain expertise and manual labor needed succesfully apply them renders them infeasible to the common setting of modeling thousands, or millions, of time series.
A.2 DEEP LEARNING APPROACHES
Recurrent models. Common deep learning architectures for modeling sequence data are the family of recurrent neural networks, which include GRUs (Chung et al., 2014), LSTMs (Hochreiter and Schmidhuber, 1997), and DeepAR (Salinas et al., 2020). However, due to the recurrent nature of RNNs. they slow to train and may suffer from vanishing/exploding gradients, making them difficult to train (Pascanu et al., 2013).
15

---

Published as a conference paper at ICLR 2023
Deep State Space models. Recent work has investigated combining the expressive strengths of SSMs with the scalable strengths of deep neural networks (Rangapuram et al., 2018; Gu et al,, 2021a). Rangapuram et al. (2018) propose to train global RNN that transforms imput covariates to sequence-speific SSM parameters; however, one downside of this approach is that they inherit the drawbacks of RNNs. More recent approaches, such as LSSLL (Gu et al., 2021b), S4 (Gu et al., 2021a), S4D (Gu et al., 2022), and S5 (Smith et al., 2022), directly parameterize the layers of a neural network with multiple linear SSMs, and overcome common recurrent training drawbacks by leveraging the convolutional view of SSMs. While deep SSM models have been shown great promise in time series modeling, we show in our work which builds off deep SSMs - that current deep SSM approaches are not able to capture autoregressive processes due to their continuous nature.
Neural differential equations as nonlinear state spaces. (Chen et al., 2018) parametrizes the vector field of continuous-timg autonomous systems. These models, termed Neural Differential Equations (NDEs) have seen extensive application to time series and sequences, first by Rubanova et al. (2019) and then by Kidger et al. (2020); Morrill et al. (2021); Massaroli et al. (2021) with the notable extension to Neural Controlled Differential Equations (Neural CDEs). Neural CDEs can be considered the continuous timee, nonlinear version of state space models and RNNs (Kidger, 2022). Rather than introducing nonlinearity between linear state space layers, Neural CDEs model nonlinear systems driven by a control input.
The NDE framework has been further applied by Poli et al. (2019) model graph time series via Neural Graph Differential Equations. In Queiruga et al. (2020), continuous-depth ResNet generalization based on ODEs is proposed, and in Kim et al. (2021) numerical techniques to enable learning of stiff dynamical systems with Neural ODEs are investigated. The idea of parameterizing the vector field of differential equation with a neural network, popularized by NDEs, can be traced back to earlier works (Funahashi and Nakamura, 1993; Zhang et al., 2014; Weinan, 2017).
Transformers. While RNNs and its variants have shown some success at time series modeling, a major limitation is their applicability to long input sequences. Since RNNs are recurrent by nature, they require long traversal paths to access past inputs, which leads to vanishinglexploding gradients and as a result struggle with capturing long-range dependencies.
To counteract the long-range dependency problem with RNNs, recent line of work considers Trans- formers for time series modeling. The motivation is that due to the attention mechanism, Trans- former can directly model dependencies between any two points in the input sequence, indepen- dently of how far apart the points are. However, the high expressivity of the attention mechanism comes at the cost of the time and space complexity being quadratic in sequence length, making Transformers infeasible for very long sequences. As result, many works consider specialized Transformer architectures with sparse attention mechanisms to bring down the quadratic complex- ity. For example, Beltagy et al. (2020) propose LogSparse self-attention, where a cell attends to a subset of past cells (as opposed to all cells), where closer cells are attended to more frequently, proportional to the log of their distance, which brings down complexity from ((e2) to OEEllog 2) Zhou et al. (2021) propose ProbSparse self-attention, which achieves O log E) time and memory complexity, where they propose a generative style decoder to speed inference. Liu et al. (2022) pro- pose pyramidal attention mechanism which shows linear time and space complexity with sequence length. Autoformer (Wu et al., 2021) suggests more specialization is needed in time series with a decomposition forecasting architecture, which extracts long-term stationary trend from the seasonal series and utilizes an auto-correlation mechanism, which discovers the period-based dependencies. Zhou et al. (2022b) believes previous attempts of Transformer- baased architectures do not capture global statistical properties, and to do so requires an attention mechanism in the frequency domain. Conformer (Gulati et al., 2020) stacks convolutional and self-attention modules into shared layer to combine the strengths of local interactions from convolutional modules and global interactions from self-attention modules. Perceiver AR (Hawthorne et al., 2022) builds on the Perceiver architecture, which reduces the computational complexity of transformers by performing self- self-attention in a latent space, and extends Perceiver's applicability causal autoregressive generation.
While these works have shown exciting progress on time series forecasting, their proposed architec- tures are specialized to handle specific time series settings (e.g.. long input sequences, or seasonal sequences), and are commonly trained to output fixed target horizon length (Zhou et al., 2021), ie., as direct multi-step forecasting (DMS) Chevillon (2007). Thus, while effective at specific fore-
16

---

Published as a conference paper at ICLR 2023
casting tasks, their setups not obviously applicable broad range of time series settings (such as forecasting arbitrary horizon lengths, or generalizing to classification or regression tasks).
Moreover, Zeng et al. (2022) showed that simpler alternatives to Transformers, such as data normalization plus a single linear layer (NLinear), can outperform these specialized Transformer architectures when similarly trained to predict the entire fixed forecasting horizons. Their results suggest that neither the attention mechanism nor the proposed modifications of these time series Transformers may be best suited for time series modeling. Instead, the success of these prior works may just be from learning to forecast the entire horizon with fully connected dependencies between prior time-step inputs and future time-step outputs, where fully connected linear layer is sufficient.
Other deep learning methods. Other works also investigate pure deep learning architectures with no explicit temporal components, and show these models can also perform well on time series fore- casting. Oreshkin et al. (2019) propose N-BEATS, a deep architecture based on backward and forward residual links. Even simpler, Zeng et al. (2022) investigate single linear layer models for time series forecasting. Both works show that simple architectures are capable of achieving high performance for time series forecasting. In particular, with just data normalization, the NLinear model in Zeng et al. (2022) obtained state-of-the-an performance on the popular Informer bench- mark Zhou et al. (2021). Given imput sequence of past lag terms and target output sequence of future horizon terms, for every horizon output their model simply learns the fully connected depen- dencies between that output and every input lag sample. However, FCNs such NLinear also carry inefficient downsides. Unlike Transformers and SSM-based models, the number of parameters for FCNs scales directly with input and output sequence length, i.e., O(hh) for inputs and h outputs. Meanwhile, SPACETIME shows that the SSM can improve the modeling quality of deep architec- tures, while maintaining constant parameter count regardless of input or output length. Especially when forecasting long horizons, we achieve higher forecasting accuracy with smaller models.
B PROOFS AND THEORETICAL DISCUSSION
B. EXPRESSIVITY RESULTS
Proposition 1. An SSM with companion state malrix can represent
i. O ARIMA (Box et al., 1970)
i. Exponential smoothing
iii. Controllable LTI systems (Chen, 1984)
We show each case separately. We either provide a set of algebraic manipu- lations to obtain the desired model from companion SSM, or alternatively invoke standard results from signal processing and system theory.
Proof of Proposition 1.
i. O
We start with standard ARMA(p. q) model
P Σ
Σ 0:up-1
yk 2k +
We consider two cases:
Case (1): Outputs y shifted (lag-1) version of the inputs u
1k-1 uk +
0 0.0n-1 +
-+1k
(16)
= (1+ + ) +
Σ (0, + Hi 1)
0
im9+1
where, without loss of generality, we have assumed that p > q for notational convenience. The autoregressive system (16) is equivalent to
17

---

Published as a conference paper at ICLR 2023
0
0
0
A
B D
C
(1 Φ1) (01 22)
P
Od
0
in state-space form, with E Rd and d max g). Note that the state-space formulation is not unique.
Case (2): Outputs are "shaped noise". The ARMA(p.q) formulation (classically) defines in- puts u as white noise samples', Vk p(ux) is a normal distribution with mean zero and some variance. In this case, we can decompose the output follows:
Vk
dcyk-iPi
yk the
0
such that yk 2/ + mm The resulting state-space models are:
A
Bar D
Car
Φp
and
O
0
1
0
A ma mma
m mma
01 O2
0--1
0g
Note that A E RRXP, A E R9xq, More generally, our method can represent any ARMA process the sum of two SPACETIME heads: one taking as input the time series itself, and one the driving signal u.
ARIMA ARIMA processes are ARMA(P. q) applied differenced time series. For example, first-order differencing yk = Uk-1- Differencing corresponds to high-pass filtering of the signal y, and can thus realized via convolution (Strang and Nguyen, 1996).
Any digital filter that can be expressed as difference equation admits a state space representation in companion form (Oppenheim, 1999), and hence can be learned by SPACETIME.
ii. O Simple exponential smoothing (SES) (Brown, 1959)
yv 0://1-1 a(1 a)yx-2 + ... a(1 a)p-1 yk-p
(17)
is AR process with parametrization involving single scalar 0 a and thus be represented in companion form as shown above.
iii. O Let (A, B, B,C) be any controllable linear system. Controllability corresponds to invertibility of the Krylov matrix (Chen, 1984, Thm 6.1, p145)
K(A, B) [B. AB. ..... d B),
K(A,B) Rdxd
From rank(KC) = follows that there exists a a E Rd
ag + a AB
a. A B Ad 0.
Other formulations with forecast residuals also common.
18

---

Published as a conference paper at ICLR 2023
Thus
AC = [AB, A ...,,ABBB B
AB, A Ad B
(a B a1 Ab +... a A B)
column left shift of K.
lincar combination.
= K((S) a1)
where G = -S --) is a companion matrix.
AK = KG = G= AK.
Therefore is similar to A. We can then construct companion form state space (G,B,C, from A using the relation above.
Proposition No class of continuous-time LSSL SSMs can represent noiseless AR(p) process.
Proof of Proposition 2. Recall from Sec. 3.1.1 that noiseless AR(p) process defined by
VI din- 0181-1 + Pp31-p
(18)
with coefficients $1..... Op. This is represented by the SSM
T++1 Szt + Bu:
(19)
yt Cr Due
(20)
when E RRXP is the shift matrix, B E TRxx1 is the first basis vector 61, C E R coefficients .1,-..3 Op. and D Le.,
TRIXP a vector of
0
S
0
0
B
0
C=
Apl
(21)
We prove by contradiction that continuous time LSSL SSM cannot represent such a process. Con- sider the following solutions to continuous-time system and system (18), both in autonomous form
diac Sxi.
I f++
It follows
disc rt+1
S
T
A log (S).
have reached contradiction by (Culver, 1966, Theorem 1), as S is singular by definition and thus its matrix logarithm does not exist.
B.2 EFFICIENCY RESULTS
We first prove that Algorithm yields the correct output filter FV, We then analyze its time com- plexity, showing that it takes time O log é + d d) for sequence length & and state dimension d.
Theorem 1. Algorithm returns the filter F
CA--BB.
19

---

Published as a conference paper at ICLR 2023
Proof. We follow the outline of the proof in Section 3.3. Instead of computing F directly,
compute its spectrum (its discrete Fourier transform):
[m[m] = F(F)) = Σ B-I BB -((I-Au")-1B m= = 0, 0.1,...,,-1. f-1.
==
where w exp(-2ri/() is the E-th root of unity.
This reduces to computing the quadratic form of the resolvent (I- D- on the roots unity (the powers of w). Since A is companion matrix, we can write A shift matrix plus rank-1
matrix, A = S + ae where ed is the d-th basis vector [0, (0....,0, 0, 1]
and the shift matrix S is:
O
0
0
07
0
0
S
0
1
0
1
Thus Woodbury's matrix identity (i.e., Sherman -Morrison formula) yields:
(I - -w = (I S - wn )
(I w S) + (I w" -----)"""")) ae T S) we (1 S)-
S)
This is the resolvent of the shift matrix (I - tm S)- with a rank-l correction. Hence
Ć - -. ac (I tm S) -)-BBB w ed (I w 5)-1a la
F" = C(I S) B
(22)
We now need to derive how compute the quadratic form of resolvent of the shift matrix effi- ciently. Fortunately the resolvent the shift matrix has a very special structure that closely relates to the Fourier transform. We show analytically that:
m 2m
(I (I-W"S) S1-1-
mm
Lu 1)-1m
2)m 2)m
m
20

---

Published as a conference paper at ICLR 2023
It is easy to verify by multiplying this matrix with I - w S if we obtain the identity matrix. Recall that multiplying with S on the left just shifts all the columns down by one index. Therefore:
1
0
2m
w
0
" S)
Ledld (w) 1)m
wid w(d-2)m 2)m
wm
wm
0
w
0
0
wam
m
w s
2m
w
0
wuld 1)m
(ldd2)m
wm
w wld-1)m 1)m
wd 2)m
wm
0
0
wm
1
0
1
0
wmm
m
0
wm
0
(uld-1)) (mm
w(d-2)m 2)m
wm
Lala 2)m
(d-33)m
1
0
0
0
0
0
wm
0
0
wmm
mm
2m
w
wuld 1mm
2ld-2)m
w
wld-1)m 1)m
(d-
=I.
Thus the resolvent of the shift matrix indeed has the form of lower-triangular matrix containing the roots of unity.
Now that we have the analytic formula of the resolvent, can derive its quadratic form, given some vectors us U E Rdd Substituting in, we have
u²"(I - w S) U1 + 42th + U2 + W3t1 2m + U3U2w + U3v1 +
Grouping terms by powers of w, we see that we want to compute 1 + W242 +... Ud then 42th + 12302 - 11 and so on. The term corresponding to km exactly the k-th element of the linear convolution of u * v. To deal with the case where just need to extend the output of then sum them up and take the length- Fourier transform. This is exactly the procedure quad(u, v) defined in Algorithm 1.
Define q u= then u u''(I (I-w""S) -" is just the Fourier transform we note that the powers of roots of unity will repeat, so be multiples of E, then split them into chunk of size l,
Once we have derived the quadratic form of the resolvent (T-w"S)- simply plugging it into the Woodbury's matrix identity (Equation (22)) yields Algorithm
We analyze the algorithm's complexity.
Theorem 2. Algorithm 1 has time complexity log + dllog d) for sequence length & and state dimension d.
Proof. We see that computing the quadratic form of the resolvent (I - -. S) S) involves linear convolution of size d and a Fourier transform of size The linear convolution can be done by performing FFT of size 2d on both inputs, multiply them pointwise, then take the inverse FFT of size 2d. This has time complexity Od d). The Fourier transform of size has time complexity Olllog E).
The whole algorithm needs to compute four such quadratic form, hence it takes time O log + d flogd). d).
21

---

Remark. We see that the algorithm easily extends the case where the matrix A is companion matrix plus low ow-rank matrix (of some rank k). We can write as the sum of the shift matrix and a rank-(k + 1) matrix (since A itself is the sum of shift matrix and rank-1 I matrix). Using the same Published as a conference paper at ICLR 2023
strategy, we can use the Woodbury's matrix identity for the rank- + +1 case. The running time will then scale O(k(Elogg & d log d)).
B.3 COMPANION MATRIX STABILITY
Normalizing companion parameters for bounded gradients Proposition 3 (Bounded SPACETIME Gradients). Given the norm of the gradient of SPACE-
TIME layer is bounded for all < s if
a =1
Proof. Without loss of generality, assume To 0. Since the solution time is
C A
Σ
Bu,
1-1
compute the gradient w.r. ug
dys CA
(23)
duk
The largest eigenvalue A mix(eig(AA)
d-1
S max 1.
Σ a.1}}
Corollary of Gershgorin (Hirst and Macey, 1997, Theorem 1)
using
Σ la
1, which implies convergence of the operator CA*-k-1 B. Thus, the gradients are bounded.
We use the proposition above to ensure gradient boundedness in SPACETIME layers by normalizing every forward pass.
C EXPERIMENT DETAILS
C.1 INFORMER FORECASTING
Dataset details. In Table I, we evaluate all methods with datasets and horizon tasks from the In- former benchmark (Zhou et al., 2021). We use the datasets and horizons evaluated on in recent works (Wu al., 2021; Zhou al., 2022b:a; Zeng al., 2022), which evaluate on electricity trans- former temperature time series (ETTh1, ETTh2, ETTm1, ETTm2) with forecasting horizons (96, 192, 336, 720). We extend this comparison in Appendix D.2 to datasets and forecasting horizons in Zhou al. (2021), which also consider weather and electricity (ECL) time series data.
Training details. We train SPACETIME on all datasets for 50 epochs using Adam opti- mizer (Loshchilov and Hutter, 2017), cosine scheduling, and early stopping based best validation standardized MSE. We performed a grid search over number of SSMs (64, 128) and weight decay (0. 0.0001). Like prior forecasting works, treat the input lag sequence hyperparameter, and train to predict each forecasting horizon with either 336 or 720 time-step-long input sequences for 3-layer SPACETIME network with 128 SSMs per layer. We train with learning rate 0.01, weight decay 0.0001, batch size 32, and dropout 0.25.
all datasets and horizons. For all datasets,
Hardware details. All experiments
single NVIDIA Tesla P100 GPU.
22

---

Published as a conference paper at ICLR 2023
C.2 MONASH FORECASTING
The Monash Time Series Forecasting Repository (Godahewa et al., 2021) provides an extensive benchmark suite for time series forecasting models, with over 30 datasets (including various config- urations) spanning finance, traffic, weather and medical domains. We compare SPACETIME against 13 baselines provided by the Monash benchmark: simple exponential smoothing (SES) (Gard- ner Jr, 1985). Theta (Assimakopoulos and Nikolopoulos, 2000), TBATS (De Livera et al., 2011), I), ETS (Winters, 1960), DHR-ARIMA (Hyndman and Athanasopoulos, 2018), Pooled Regression (PR) (Trapero et al., 2015), CatBoost (Dorogush et al., 2018), FFNN, DeepAR (Salinas al., 2020), N-BEATS Oreshkin et al. (2019), WaveNet (Oord et al., 2016), vanilla Transformer (Vaswani et al., 2017). A complete list of the datasets considered and baselines, including test results (average RMSE across 3 seeded runs) is available in Table 20.
Training details. We optimize SPACETIME all datasets using Adam optimizer for 40 epochs with a linear learning rate warmup phase of 20 epochs and cosine decay. We initialize learning rate at 0.001, reach 0.004 after warmup, and decay 0.0001. We do not use weight decay or dropout.
We perform a grid search over number of layers {3. 4, 5, 6), number of SSMs per layer (8, 16, 32, 64, 128) and number of channels (width of the model) {I, 4, 8, 16). Hyperparameter tuning is performed for each dataset. We pick the model based on best validation RMSE performance.
Hardware details. All experiments were run on single NVIDIA GeForce RTX 3090 GPU.
C.3 TIME SERIES CLASSTFICATION
ECG classification (motivation and dataset description). Electrocardiograns (ECG) are com- monly used as one of the first examination tools for assessing and diagnosing cardiovascular dis- eases, which are major cause of mortality around the world (Amini et al., 2021). However, ECG in- terpretation remains a challenging task for cardiologists and general practitioners (Jablonover al., 2014; Cook et al., 2020). Incorrect interpretation of ECG can result in misdiagnosis and delayed treatment, which can be potentially life-threatening in critical situations such as emergency rooms, where an accurate interpretation is needed quickly.
To mitigate these challenges, deep learning approaches are increasingly being applied to interpret ECGs. These approaches have been used for predicting the ECG rhythm class (Hannun et al., 2019), detecting atrial fibrillation (Attia et al., 2019h), rare cardiac diseases like cardiac amyloidosis (Goto et al.. 2021), and variety of other abnormalities (Attia et al., 2019a; Siontis et al., 2021). Deep learning approaches have shown preliminary promise in matching the performance of cardiologists and emergency residents in triaging ECGs, which would permit accurate interpretations in settings where specialists may not be present (Ribeiro et al., 2020; Hannun et al., 2019).
We use the publicly available PTB-XL dataset (Wagner et al., 2020a;b; Goldberger et al., 2000), which contains 21, 31,837 12-lead ECG recordings of 10 seconds each obtained from 18, 8888 patients. Each ECG recording is annotated by up to two cardiologists with one or more of the 71 ECG state- ments (labels). These ECG statements conform to the SCP-ECG standard (Secretary, 2009). Each statement belongs to one or more of the following three categories diagnostic, form, and rhythm statements. The diagnostic statements are further organised in a hierarchy containing 5 superclasses and 24 subclasses.
This provides six sets of annotations for the ECG statements based on the different categories and granularities: all (all ECG statements), diagnostic (only diagnostic statements including both sub- class and superclass statements), diagnostic subclass (only diagnostic subclass statements), diag- nostic superelass (only diagnostic superclass statements), form (only form statements), and rhythm (only rhythm statements). These six sets of annotations form different prediction tasks which are ferred to as all, diag, sub-diag, super-diag, form, and rhythm respectively. The diagnostic superclass task is multi-class classification, and the other tasks multi-label classification.
ECG classification training details. To tune SPACETIME and S4, we performed a grid search over the learning rate {0.01. 0.001), ), model dropout (0.1,0.2), number of SSMs per layer {128, 256), and number of layers {4. 6), and chose the parameters that resulted in highest validation AUROC. The SSM state dimension was fixed to 64, with gated linear units as the non-linyayity between
23

---

Published as a conference paper at ICLR 2023
stacked layers. We additionally apply layer normalization. We use cosine learning rate scheduler, with a warmup period of 5 epochs. We train all models for 100 epochs.
Speech Commands training details. To train SPACETIME, by S4: learning rate of 0.01 with a plateau scheduler with patience 20, dropout of 0.1, 128 SSMs per layer, 6 layers, batch normalization, trained for 200 epochs.
the same hyperparameters used
Hardware details. For both ECG and Speech Commands, experiments were run on a single NVIDIA Tesla A100 Ampere 40 GB GPU.
D EXTENDED EXPERIMENTAL RESULTS
D.1 EXPRESSIVITY ON DIGITAL FILTERS
We experimentally verify whether SPACETIME can approximate the input- map of digital filter admitting a state-space representation, with improved generalization over baseline models given test imputs of unseen frequencies.
We generate a dataset of 1028 sinusoidal signals of length 200
x(t) sin (2twt)
where w 40) U [50, 100) in the training set and w E (40, 50) in the test set. The outputs are obtained by filtering æ, i... y = F(x) where F is the family of digital filters.
layers models baselines: the original S4 diagonal plus low-rank (Gu al., 2021a), single-layer LSTM, a single 1d convolution (Conv Id), dense linear layer (NLinear), single self aff-attention layer. All models trained for 800 epochs with batch size 256, learning rate 10 3 and Adam. We repeat this experiment for digital filters of different orders (Oppenheim, 1999). The results shown Figure 8. SPACE IIME learns match the frequency response of the target filter, producing the correct output for inputs at test frequencies.
We introduce common various sequence-to-sequeete
Table Comparing sequence models on the task of approximating the input- input-outpur map defined by digital filters of different orders. Test RMSE on held-out inputs unseen frequencies.
Filter
Order
SPaCETIMe
$4
CD
LSTM
NLinear
Transformer
Butterworth
2
0.0055
0.0118
0.0112
0.0115
1.8420
0.5535
3
0.0057
0.3499
0.0449
0.0231
1.7085
0.6639
10
0.0039
0.8077
0.4747
0.2753
1.5162
0.7191
Chebyshev
0.0187
0.0480
0.0558
0.0285
1.9313
0.2452
3
0.0055
0.0467
0.0615
0.0178
1.8077
0.4028
10
0.0620
0.6670
0.1961
0.1463
1.5069
0.7925
Chebyshev 2
2
0.0112
0.0121
0.0067
0.0019
0.4101
0,0030
3
0.0201
0.0110
0.0771
0.0102
0.4261
0.0088
10
0.0063
0.6209
0.3361
0.1911
1.5584
0.7936
Elliptic
2
0.0001
0.0300
0.0565
0.0236
1.9150
0.2445
3
0.0671
0.0868
0.0551
0.0171
1.8782
0.4198
10
0.0622
0.0909
0.1352
0.1344
1.4901
0.7368
D.2 INFORMER FORECASTING
Univariate long horizon forecasts with Informer splits. Beyond the ETT datasets and horizons evaluated on in Table 7, we also compare SPACETIME to alternative time series methods on the complete datasets and horizons used in the original Informer paper (Zhou et al., 2021). We compare against recent architectures which similarly evaluate on these settings, including ETSFormer (Woo al., 2022), SCINet (Liu al., 2021), and Yformer (Madhusudhanan et al., 2021), 1) and other com- parison methods found in the Informer paper, such as Reformer (Kitaev et al., 2020) and ARIMA. SPACETIMES obtains best results 20 out of 25 settings, the most of any method.
24

---

Published as a conference paper at ICLR 2023
Table 7: Univariate forecasting results on Informer datasets. Best results in bold. SPACETIME obtains best MSE on 19 out of 25 and best MAE on 20 out of 25 dataset and horizon tasks.
Mitiool
SenseCTnss
S4
Vivemer
Informe:
Recorner
24 N-BRACT
Diep-lt
ARIMA
Prophis
MS5
MAF
MSR
MAN
MA-F
MAA.
MSE
M44
MSS:
MAAF
M.A..
MSS
MAR.
M45F
MAT
18.33
D/9-1
IL158
11551
01.333
9.221
0.338
0.358
0.310
8.667
1:228
13384
03445
3166
0.301
0.162
11322
08:15
0.324
100
0.056
0.300
8JKG
IL313
0088
0.222
0.104
0,558
0.255
11133
0.546
8.30
00375
1.322
1
3.108
0258
0.299
0.422
9:504
1.2%
0.665
723
03.15
8.356
0.227
00.76
0.348
0.116
0.211
0:226
0.934
0.289
0.435
82233
0.485
2112
1435
0.300
B422
06588
0.537
03.53
0.785
2.755
3.25
48
0.895
8:316
0111
0.363
0.343
0.565
0.773
0.334
1155
0.114
0.38
0.458
0.545
011 113
0.71
0165
0.34
196
0:414
0.36
0.802
5.000
11328
01311
06327
0331
0380
0336
0442
1022
0.44
334
1,310
1.22
009
0,557
3.53
0.738
3106
2.50
720
1.503
0.226
0.385
10.55
0.403
0.887
0.588
0.211
0.352
0.237
0451
533
0.25
2030
1.211
0:35
0632
0429
0.500
2575
1844
3.505
4.054
13.33
0.874
10554
10029
0.588
0.044
0417
01118
10.000
0.131
0121
0233
0.225
3101
0117
0091
0.243
18.00
0.315
0.12)
0.2M)
0.811
8.101
01007
09815
0176
01773
0.501
000
0770
03500
3106
016
0719
0.30
0155
750
0.851
8.11
0.179
0.1
09.66
160
0359
01100
0.36
0.40
0.56
5411
0.57
1108
1335
0141
0511
0998
0.785
0.163
0.58
0.452
0.5%
03.78
8077
01314
01.90
0.325
01476
1.70
1.228
0.201
0370
2447
1.352
0.539
0.601
2.297
1.14
24
18.05
8.16
125
0.551
01:13
0.551
5.16
0.59
10311
0.10
01155
0.33
0.314
0.555
0.36
5.36
0.30
0.335
166
13:33
0.110
0.768
03554
0.501
01451
0.501
0.44
236
0.263
8.500
2.300
0447
0.229
0416
3350
0446
1292
1.09
0.503
0.544
0.113
0.30
1967
2445
48
18.14
0.322
0.500
01151
0.323
03333
0.550
0.200
1.22
01971
0.584
0.366
0.357
03:75
0.06
0.95
176
0.383
C.363
0.708
0.30
11375
0.885
0.38
2411
11563
3.538
2.186
1:138
0576
3.366
3077
4.911
11383
06645
5:105
Table 8: Multivariate forecasting results on Informer datascts. Best results in bold. SPACETIME obtains MSE and MAE competitive with NLincar, the prior state-of-the-art.
Methods
SPACETIME
NLinear
FiLM
S4
FEDFormer
Autoformer
Informer
Metric
MSF
MAE
MSF
MAF
MSE
MAE
MSE
MAF
MSF
MAF
MSE
MAE
MSE
MAR
ETThl
720
0.499
0.453
0.472
1.074
0.814
0.506
0.507
0.514
0.512
1.811
0.865
ETTh2
720
0.402
0.434
0.394
0.436
0.439
0.456
2.973
1.333
0.463
0.474
0.515
0.511
3.647
1.625
ETTml
720
0.408
0.415
0.433
0.422
0.420
0.420
0.738
0.655
0.543
0.49
0.671
0.561
1.166
0.823
ETTm2
720
0.358
0.378
0.368
0.384
0.393
0.422
2.074
1.074
0.421
0.415
0.433
0.432
3,379
1,338
Multivariate signals. We additionally compare the performance of SPACETIME to -tate-of-the-art
comparison methods on ETT multivariate settings. We focus on horizon length 720, the longest evaluated in prior works. In Table 8, we find SPACETIME is competitive with NLinear, which achieves best performance among compparison methods. SPACETIME also notably outperforming S4 by large margins, supporting the companion matrix representation once more.
D.3 MONASH FORECASTING
We report the results across all datasets in Table 20. We also investigate the performance of models by aggregating datasets based on common characteristics. Concretely, we generate sets of tasks? based on the following properties:
*
Large dataset: the dataset contains more than 2000 effective training samples.
*
Long context: the models are provided context of length greater than 20 as input.
*
Longg horizon: and the models are asked to forecast longer than 20 steps in the future.
Figure 6 shows the average x/ 13 model ranking in terms of test RMSE across splits. We contex- tualize SPACETIME results with best classical and deep learning methods (TBATS and DeepAR). SPACETIME relative performance is noticeably higher when context and forecasting horizons are longer, and when a larger number of samples is provided during training.
A task can belong to multiple splits, resulting in overlapping splits. For example, a task can involve both long context as well long forecasting horizon.
25

---

Published as a conference paper at ICLR 2023
Monash Model Rankings
Long
Horizon
3.08
4.25
Short
Short
5.14
5.67
5.76
5.15
5.10
5,67
6,50
6.77
Context
4.01
5.42
2.30
5.63
3.03 Long
Large
5.37
4.25
Dataset
Small
Space Time
DeepAR
TBATS
Figure 6: Relative test RMSE rankings (+/13 models) across different slices of the 33 datasets in the Monash repository (Godahewa al., 2021). SPACETIME sets best overall ranking across all tasks and is significantly more accurate on tasks involving long forecast horizon and larger number of training samples.
D.4 ECG CLASSIFICATION
In addition our results table in the main paper, we also provide the mean and standard deviations of the two models we ran in house (SPACETIME and $4) Table
Table 9: ECG statement classification on PTB-XL (100 Hz version). We report the mean and standard deviation three random seeds for the three methods in house.
Task AUROC
All
Diag
Sub-diag
Super-diag
Form
Rhythm
SPACETIME
93.6(0.13)
94.1(0.12)
93.3(0.34)
92.9(0.09)
88.3(0.63)
96.7(0.05)
S4
93.8(0.38)
93.9(0.15)
92.9(0.11)
98.1(0.07)
89.3(0.66)
97.7(0.04)
Transformer
85.7(0.30)
87.6(0.41)
88.2(0.20)
88.7(0.28)
77.1(0.45)
83.1(0.72)
D.5 EFFICIENCY RESULTS
We additionally empirically validate that SPACETIME trains in near near-linear time with horizon se-
quence length. We also synthetic data, scaling horizon from 1 1000.
SpaceTime
Epoch wall-clock time
400
LSTM
Transformer
300
200
100
200
400
600
800
1000
Horizon sequence length
Figure 7: Train wall-clock time as we scale horizon length for SPACETIME, S4, LSTM, and Trans- former.
26

---

Published as a conference paper at ICLR 2023
D.6
SPACETIME ABLATIONS
To better understand how the proposed SPACETIME SSMs lead to the improved empirical perfor- mance, include ablations on the individual closed-loop forecasting SSM (Section 3.2) and pre- processing SSMs (Section 3.1.3).
D.6.1 CLOSED-LOOP FORECASTING SSM
To study how the closed-loop SSM improves long horizon forecasting accuracy, we remove the closed-loop SSM component in default SPACETIME forecasting architecture (c.f., Appendix D.7, and compare the default SPACETIME with one without any closed-loop SSMs on Informer forecast- ing tasks. For models without closed-loop SSMs, replace the last layer with the standard "open- loop" SSM framework in Section 3.1.2), and keep all other layers the same. Finally, for baseline comparison against another SSM without the closed-loop component, we compare against S4.
In Table 10, we report standardized MSE on Informer ETT datasets. Adding the closed-loop SSM consistently improves forecasting accuracy, on average lowering relative MSE by 33.2%. Mean- while, even without the closed-loop SSM, SPACETIME outperforms S4, again suggesting that the companion matrix parameterization is beneficial for autoregressive time series forecasting.
Table 10: Closed-loop SSM Ablation ablate closed-loop SSM component in SPACETIME, comparing against the prior S4 SSM on four Informer time series forecasting tasks. Removing the closed-loop SSM consistently hurts forecasting accuracy for SPACETIME.
ETThi (720)
ETTh2 (720)
ETTml (720)
ETTm2 (720)
Method / Ablation
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
SPACETIME
0.076
0.222
0.188
0.352
0.074
0.213
0.166
0.318
SPaceTime No Closed-loop
0.114
0.271
0.278
0.431
0.156
0.310
0.213
0.365
S4 (No Closed-loop)
0.190
0.355
0.630
0.662
0.254
0.433
0.482
0.567
D.6.2 PREPROCESSING SSM
To study how the preprocessing SSM improves long horizon forecasting accuracy, next compare how SPACETIME performs with and without the weight-initializing preprocessing SSMs introduced in Section 1.3. We compare the default SPACETIME architecture (Table 12 with (1) replacing the preprocessing SSMs with randomly initialized default companion SSMs, and (2) removing the preprocessing SSMs altogether. For the former, we preserve the number of layers, but now train the first-layer SSM weights. For the latter, there is one-less layer, but the same number of trainable parameters (as we fix and freeze the weights for each preprocessing SSM).
In Table 11, we report standardized MSE on Informer ETT datasets. We find fixing the first layer SSMs SPACETIME network to preprocessing SSMs consistently improves forecasting perfor- mance, achieving 4.55% lower MSE on average than the ablation with just trainable companion matrices. Including the preprocessing layer also improves MSE by 9.26% on average compared to removing the layer altogether. These results suggest that preprocessing SSMs are beneficial for time series forecasting, e.g., by performing classic time series modeling techniques on input data. Unlike other approaches. SPACETIME is able flexibly and naturally incorporate these operations into its network layers via simple weight initializations of the same general companion SSM structure.
Preprocessing SSM Ablation We ablate the preprocessing SSM layer in SPACETIME, comparing against either replacing the SSMs with companion SSMs (Companion) removing the layer (Removed), Including preprocessing SSMs consistently improves forecasting accuracy.
Table 11:
ETThl (720)
ETTh2 (720)
ETTml (720)
ETTm2 (720)
Method / Ablation
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
SpaceTime
0.076
0.222
0.188
0.352
0.074
0.213
0.166
0.318
SpaceTime No Preprocessing (Companion)
0.076
0.224
0.194
0.358
0.079
0.218
0.182
0.336
SpaceTime No Preprocessing (Removed)
0.078
0.227
0.204
0.367
0.087
0.232
0.188
0.326
27

---

Published as a conference paper at ICLR 2023
D.7 SPACETIME ARCHITECTURES
We provide the specific SPACETIME architecture configurations used for forecasting and classifi- cation tasks. Each configuration follows the general architecture presented in Section 3.1 and Fig- ure 2, and consists of repeated Multi-SSM SPACETIME layers. We first provide additional details on specific instantiations of the companion SSMs we use in our models, egg. how we instantiate pre- processing SSMs to recover specific techniques (Section 3.1.3). We then include the layer-specific details of the number and type of SSM used in each network.
D.7.1 SPECIFIC SSM PARAMETERIZATIONS
In Section 3.1.1, we described the general form of the companion SSM used in this work. By default, for any individual SSM we learn the column in A and the vectors B, C' as trainable parameters in neural net module. We refer to these SSMs specifically
companion SSMs.
In addition, as discussed in Sections 3.1.1 and 3.1.3, we can also fix a, B, or C' to specific values recover useful operations when computing the SSM outputs. We describe specific instantiations of the companion SSM used in our models below (with dimensionality referring to one SSM).
Shift SSM. We fix the vector in the companion state matrix A E [dddd to the O vector Rdd such that A is the shift matrix (see Eq. 21 for an example). This is generalization of a 1-D "sliding window" convolution with fixed kernel size equal to SSM state dimension To see how, note that if B is also fixed to the first basis vector e₁ E Rd then this exactly recovers 1-D convolution with kernel determined by C.
Differencing SSM. As a specific version of the preprocessing SSM discussed in Section 3.1.3, we fix = 0, B C1, and set C' recover various order differencing when computing the SSM, i.e.,
C= [1
0
0
0 (0-order differencing, i.e., an identity function)
(24)
-= [1 -1
0
0
0
(Ist-order differencing)
(25)
== [1 -2
0)
(2nd-order differencing)
(26)
C= -3 3
(3rd-order differencing)
(27)
In this work, we only use the above O, 1st, 2nd, or 3rd-order differencing instantiations. With multiple differencing SSMs a multi-SSM SPACETIME layer, we initialize differencing SSMs by rumning through the orders repeatedly in sequence. For example, given five differencing SSMs, the first four SSMs perform 0, 1st, 2nd, and 3rd-order differencing respectively, while the fifth performs O-order differencing again.
Moving Average Residual (MA residual) SSM. As another version of the preprocessing SSM, we can fix O, B e1, and set C such that the SSM outputs sample residuals from moving average applied over the input sequence. For an n-order moving average, we compute outputs with C' specified as
C C= - 11/n,,
-1nn,
11/n, 0 0] (n-order moving average residual) (28)
For each MA residual SSM, we randomly initialize the order by uniform-randomll sampling an integer in the range [4, d), where is again the state-space dimension size (recall C E IR (xd). We pick as a heuristic which was finetuned; we leave additional optimization here for further work.
D.7.2 TASK-SPECIFI SPACETIME ARCHITECTURES
Here, we provide layer-level details on the SPACETIME networks used in this work. For each task, we describe number of layers, number of SSMs per layer, state-space dimension (fixed for all SSMs in network), and which SSMs are used in each layer.
Expanding on this last detail, as previously discussed in Section 3.1.2, in each SPACETIME layer we specify multiple SSMs in each layer, computing their outputs in parallel to produce multidi- mensional output that is fed as the input to the next SPACETIME layer. The "types" of SSMs do not all have be the same per layer, and we list the type (companion, shift, differencing, MA residual) and closed-loop designation (standard, closed-loop) of the SSMs in each layer below.
For additional visual overview of SPACETIME network, please refer back to Figure 2.
28

---

Published as a conference paper at ICLR 2023
Table 12: SPACETIME forecasting architecture. For all SSMs, we keep state-space dimension d' 128. Repeated Identity denotes repeating the input to match the number of SSMs in the next layer, ie.. 128 SSMs in this case. For each forecasting task, d' denotes time series samples' number of features, l denotes the lag size (number of past samples given as input), and / denotes the horizon size (number future samples to be predicted).
Layer
Details
Input Size
Output Size
Decoder
Linear
128 l
d' x h
Companion (closed-loop)
SSM Layer 3
128
128 l
128
Companion (standard)
SSM Layer
128
128 l
128 l
Differencing (standard)
64
SSM Layer
128 x 8
128 l
MA Residual (standard)
x 64
Encoder
Repeated Identity
x l
128
Forecasting: Informer and Monash. We describe the architecture in Table 12. We treat the first SPACETIME layer as "preprocessing" layer, which performs differencing and moving average resid- ual operations on the input sequence. We treat the last SPACETiMe layer as "forecasting" layer, which autoregressively outputs future horizon predictions given the second-to-las layer's 's outputs as an input sequence.
Classification: ECG. We describe the architectures for each ECG classification task in Tables 13- 18. For all models, we use state-space dimension classification we compute logits with mean pooling over the output sequence, where pooling is computed over the sequence length.
As described the experiments, for
We describe the architecture for the Speech Audio task in Table 19. We use state-space dimension d = 1024. As described in the experiments, for classification we compute logits with mean pooling over the output sequence, where pooling is computed over the sequence length.
Classification:
Speech Audio.
29

---

Published as a conference paper at ICLR 2023
Table 13: SPACETIME architecture for ECG SuperDiagnostic classification. For all SSMs, keep state-space dimension = 64. Input samples have d' 12 features and are length f 1000 time- steps long. The number of classes 5.
Layer
Details
Input Size
Output Size
Classifier
Mean Pooling
x
x 1
Decoder
Linear
256
x l
Companion (standard)
SSM Layer
256
256
256
Companion (standard)
SSM Layer
256
256 f
256 x
Companion (standard)
SSM Layer
256
256 E
256 x
Companion (standard)
SSM Layer
256
256 x
256 x
Differencing (standard)
SSM Layer
256
256 x
256 x
Encoder
Linear
d'
256 x
30

---

Published as a conference paper at ICLR 2023
Table 14: SPACETIME architecture for ECG SubDiagnostic classification. For all SSMs, keep state-space dimension = 64. Input samples have d' 12 features and are length f 1000 time- steps long. The number of classes 23.
Layer
Details
Input Size
Output Size
Classifier
Mean Pooling
x
x 1
Decoder
Linear
256
x l
Shift (standard)
SSM Layer
256
256
256
Shift
SSM Layer
256
256 f
256 x
(standard)
Shift (standard)
SSM Layer
256
256 E
256 x
Shift
SSM Layer
x 256
256 x f
256 x
(standard)
Differencing (standard)
SSM Layer
256
256 x
256 x
Encoder
Linear
d'
256 x
31

---

Published as a conference paper at ICLR 2023
Table 15: SPACETIME architecture for ECG Diagnostic classification. For all SSMs, we keep state- space dimension = 64. Input samples have 12 features and are length 1000 time-steps long. The number of classes 44.
Layer
Details
Input Size
Output Size
Classifier
Mean Pooling
x
x 1
Decoder
Linear
256
l
Shift (standard)
SSM Layer
256
256
256
Shift
SSM Layer
256
256 f
256 x
(standard)
Shift (standard)
SSM Layer
256
256 E
256 x
Shift
SSM Layer
x 256
256 x f
256 x
(standard)
Differencing (standard)
SSM Layer
256
256 x
256 x
Encoder
Linear
d'
256 x
32

---

Published as a conference paper at ICLR 2023
Table 16: SPACETIME architecture for ECG Form classification. For all SSMs, we keep space dimension d = 64. Input samples have 12 features and are length & 1000 time-steps long. The number classes 19.
Layer
Details
Input Size
Output Size
Classifier
Mean Pooling
x
Decoder
Linear
256
x l
Companion (standard)
SSM Layer
256
256
256 x
Companion (standard)
SSM Layer
256
256
256
Companion (standard)
SSM Layer
256
256 8
256
Companion (standard)
SSM Layer 2
256
256 {
256 x
Differencing (standard)
x 192
SSM Layer
256 x
256 x
MA Residual (standard)
64
Encoder
Linear
d' x
256
33

---

Published as a conference paper at ICLR 2023
Table 17: SPACETIME architecture for ECG Rhythm classification. For all SSMs, keep state- space dimension = 64. Input samples have 12 features and are length 1000 time-steps long. The number of classes 12.
Layer
Details
Input Size
Output Size
Classifier
Mean Pooling
x
X
Decoder
Linear
256 e
x l
Companion
128
(standard)
SSM Layer
256 L
256 x &
Shift (standard)
x 128
Companion (standard)
x 128
SSM Layer
256
256
Shift (standard)
x 128
Companion
128
(standard)
SSM Layer
256 x
256 x
Shift (standard))
x 128
Companion
128
(standard)
SSM Layer
256 f
256
Shift
x 128
(standard)
Differencing (standard)
SSM Layer
x 256
256 x
256 x
Encoder
Linear
d' E
256 x
34

---

Published as a conference paper at ICLR 2023
Table 18: SPACETIME architecture for ECG All classification. For all SSMs, we keep space dimension d = 64. Input samples have 12 features and are length & 1000 time-steps long. The number classes c 71.
Layer
Details
Input Size
Output Size
Classifier
Mean Pooling
x
Decoder
Linear
256
x l
Shift 256
SSM Layer
256
256 x
(standard)
Shift
SSM Layer
256
256
256
(standard)
Shift 256
SSM Layer
256 8
256
(standard)
Shift (standard)
SSM Layer
256
256 f
256 x
Differencing (standard)
x 192
SSM Layer
256 x
256 x
MA Residual (standard)
64
Encoder
Linear
d' x
256
35

---

Published as a conference paper at ICLR 2023
Table 19: SPACETIME architecture for Speech Audio classification. For all SSMs, we keep state- space dimension 1024. Input samples have d' features and length 16000 time-steps long. The number of classes 10.
Layer
Details
Input Size
Output Size
Classifier
Mean Pooling
x
x
Decoder
Linear
256 x &
x l
Companion (standard)
SSM Layer
256
256 t
256 x
Companion (standard)
SSM Layer
256
256 x
256 x
Companion (standard)
SSM Layer
256 256
256 x
Companion (standard)
SSM Layer
256
256
256
Companion (standard)
SSM Layer
256
256 É
256
Companion (standard)
SSM Layer
256
256 É
256
Encoder
Linear
d' e
256 x
36

---

Published as a conference paper at ICLR 2023
SpaceTime, filtering on test signals
Input
Output
150
150
150
200
200
$4, filtering on test signals
Input
Output
200
50
Convid, filtering test signals
Output
100
150
200
100
150
200
50
100
150
200
50
100
150
200
LSTM, filtering on best signals
Input
Output
150
200
50
150
200
150
200
150
200
NLinear, filtering test signals
Inyut
Output
0.0
2.5
50
100
150
200
50
100
150
200
50
100
150
200
50
100
150
200
Transformer filtering on test signals
Input
Output
160
150
200
50
100
150
200
200
150
Figure 8: Testing the capability of different sequence-to-seeueeeeeeeeeeee---e models to approximate the input- output map of digital filters. blue, we show the output signal filtered by each model. The ground- truth digital filter is Butterworth of order 10.
37

---

Published as a conference paper at ICLR 2023
Trans former
182850.6
2231.5
3129.8
1084.8
819.2
948.4
271.0
739.1
902.4
456.9
233.6
391.2
80089.3
11724.1
2660.1
70.2
2.8
24.0
7.2
693.8
514.7
78894.7 4.
5098.9 9
0.02
1.9
40.5
480.0
49.
686.5
0.5
0.5
33.0
Table 20: Monash forecasting. We report test RMSH of SPACE SPACETIMEfOr I'MEfor each dataser (best result selected via validation RMSE, average of 133 runs).
WaveNet
312821.
2271.7
2578.9
1147.6
606.8
845.3
277.0
697.0
787.9
437.3
220.5
468.
77581.3
11546.6
2694.2
68.0
3.0
24.2
8.0
2569.3
489.9
63916.9
2779.5
0.03
1.6
23.4
1135.4
43,0
768.8 8
0.7
0.58
29.0
N-BEATS
192489.8
2267.3
2183.4
1117.4
582.8
796.9
248.5
684.7
705.2
330.8
221.7
501.2
78241.7
11306.0
2596.2
99.3
3.1
17.4
6.6
1307.6
$10.9
35576.8
2813.0
1.4
24.2
186.5
48.9
627.7
14.5
1.0
33.6
DeepAR
173075.1
2313.3
2202.2
1157.9
606.6
873.7
277.7
700.3
740.3
422.2
343.5
1095.
78470.7
11762.0
2359.9 ,9
65.8
2.7
18.5
7.2
873.6
478.0
53100.3
4638.7
0.02
1.5
22.0
230.5
45.3
684.0
1.1
0.50
26.5
CalBoost
237644.5
2161.0
2461.7
1341.7
698. 0
874.2
349.9
714.2
734.8
420.8
263.1
344.6
87489.0
12788.0
3 8
60.8
3.1
18.7
8.7
1754.2
582.7
37289,7
2679.4
23.5
607.9 9
39.3
618.4
2.4
0.71
27.3
1520338.7
1909.3
2478.8
11818
605.5
830.0
262.3
711.9
720.5
350.3
213.0
313.0
89645.6
11746.9
2739.4
61.8
9.1
18.6
7.2
1168.2
689.9
47802.
9736.9
0.03
1.5
23.5
394.1
47.7
732.1
4.0
0.58
31.7
(DHR- AAIMAA
175343.8
2538.5
2450.6
1662.2
650.8
790. 8
220
710.0
702.1
386.3
212.6
1563.1
106082.6
12564.8
3132.4
820.3
3.1
18.6
5.6
967.5 .9
1082.4
32594.8
3312.5
0.04
1.5
23.7
100.5
43.2
705.5
3.0
0,71
35.0
ETS
167739.0
2408
2264.0
1189 .2
598.7
755.3
224.1
674.3
705.7
408.5
230.0
3830.4
104700.5
10812.3
2543.0
278.3
3.0
18.8
7.2
1264.4
1524.9
70369.0
2341.7
0.04
1.5
22.0
102.
50.4
607.2
5.0
0.71
37.6
TBATS
116850.9
2673.9
2594.5
1386.3
653.6
765.2
217.0
672.7
743.4
356.7
2084
469.9
105799.4 4
12001.5
3661.5
261.3
2.9
18.5
10.7
1049.0
743.4
28039.7
2295.7
0.05
1.5
21.3
113.0
42.6
606.5
3.0
0.71
25,7
Thela
171458.1
2282.7
2564.9
1106.1
567.7
754.0
242.1
673.2
683.7
405.2
210.4
1483.7
99914.2
9254.6
2702.0
228.2
3.3
18.7
7.2
1341.6
1026.4
76935.6
3898.7 7
0.04
1.5
22.6
370.
39.8
735.5
5.0
0.65
37.4
SES
193829.5
2545.7
2725.8
1172.9
670.6
893.9
309.7
732.8
755.5
412.6
209.8
1476.8
106665.2
15000.0
7039.4 44
228.1
2.9
18.8
7.2
1331.3
1026.3
77067.9
3103.0
0.04
1.5
26.6
403.4
39.8
1369.5
5.0
0.71
36.5
SPACETINE TIME
135508.3
2200.3
2601.1
1412.4
676.1
897.12
263.56
718.2
1092.2
348.3
183.2
255.2
74799.2
11608.33 32
3181.2
69.6
2.7
16.9
7.4
1423.7
475.1
37802.2
3743.6
0.03
1.3
40.1
490.
24.0
630.2
3.1
0.64
30.4
Tourism Quarterly
Tourism Monthly
Electricity Hourly
Electricity Weekly
Dataset
M1 Yearly
MI Quarterly
MI Monthly
M3 Yearly
M3 Quarterly
M3 Monthly
M3 Other
M4 Quarterly
M4 Monthly
M4 Weekly
M4 Daily
M4 Hourly
Tourism Yearly
Weather Pedestrian
NN5 Weekly
Solar 10 min
Solar Weekly
Fred-MD MD
Traffic Hourly
Traffic Weekly
Hospital
Covid
Saugeen
US Births
Sunspot
Car Parts
Vehicle Trips
38


Reviews:
[{"review_id": "qP3BBoD0O2", "reviewer": "Reviewer_61Ty", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new architecture called SpaceTime, which is a state-space model (SSM) architecture that is based on the companion matrix, and is able to learn to model autoregressive processes. This architecture is applied to both time series classification and forecasting. A \"closed-loop\" variation of the approach is introduced for long-term forecasting where the architecture generates its own layer-wise inputs. The paper also proposes an algorithm for efficient training and inference for the architecture, which reduces computation from the naive O(dl) to O(d + l), where d is state-space size and l is sequence length.\n\nThe SpaceTime approach achieves competitive performance on ECG and speech time series classification and on Informer forecasting tasks. Better qualitative forecasting results are shown for long time horizons, and the architecture is shown to achieve 73% and 80% faster training in terms of wall-clock time compared to Transformers and LSTMs, respectively. ", "summary_of_the_review": "A new SSM-based architecture that has improved modeling capability compared to previous SSM-based models. Claims are well-supported with extensive proofs, and evaluation on time series classification and forecasting are substantial and convincing. The paper is very clear, and code is provided for reproducibility. Overall, an excellent paper that contributes a novel improved SSM architecture that improves on state-of-the-art for time series analysis.", "strength_and_weaknesses": "### Strengths\n\n1. The paper is very clear, thorough, and theoretically sound. Proofs are provided for all propositions, and they seem correct as far as I can tell.\n\n2. The proposed architecture seems quite effective, and the results show that the approach has impressive empirical performance. The architecture has desirable qualities compared to other similar architectures, and the paper does a good job about discussing these comparisons.\n\n### Weaknesses\n\n1. It would have been nice to see the approach applied to some other datasets, e.g. more speech or audio-based datasets. That said, the existing evaluations are quite extensive, so this is a good direction for future work.\n\n2. I have a few minor comments (see below)\n\n### Minor comments\n\n1. Using m for the number of SSMs per SpaceTime layer is a little confusing (Architecture section of section 3.1.2), because it is also used as the feature dimension earlier.\n\n2. It would be good to provide a few more details about the speech classification task in the \"Speech Audio (single-label classification)\" section in section 4.1. Also, the section mentions that \"SpaceTime outperforms prior SSMs (LSSL), Transformers, and specialized architectures (Table 3)\", but LSSL and Transformer don't have results listed in Table 3.\n\n3. \"length-wise pooling\": maybe say mean-pooling over length?\n\n4. In section 4.2.2, NLinear is mentioned without much detail. Maybe good to add a little more detail?\n\n5. Caption in Figure 4 referes to \"DLinear\"; should this be \"NLinear\"?\n\n6. In proofs in Appendix B for Proposition 1, maybe better to use numbers or letters instead of bullets to refer to the different cases (i.e. ARIMA, Exponential smoothing, Controllable LTI systems)\n\n7. A couple additional references that might be good to add to the \"Transformers\" section in A.2:\n\n    Conformer (https://arxiv.org/abs/2005.08100)\n\n    Perceiver AR (https://arxiv.org/abs/2202.07765)\n\n8. \"Moreover, embarrassingly simple alternatives, such as a single linear layer, have shown to outperform these highly specialized transformers\": is there a citation for this?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FpUd-NuXGB", "reviewer": "Reviewer_hBe1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a layer named SpaceTime. The proposed layer uses a State Space Model-based latent space structure. SpaceTime layers could achieve better performance in the task of time series forecasting and classification. ", "summary_of_the_review": "1. Some claims are wired and maybe even incorrect. For example, some claims like”  “Recent Transformer-based models reduce this complexity to O(ℓ+h), but do not always outperform the above FCNs on forecasting benchmarks”, but authors only cited an arXiv paper with limited citation. Moreover, the cited article seems like only focuses on ablation tests. \n\n2. Every component in the proposed model is ad-hoc and lacks a reason to do that. For example, the model used SSM embedded into the autoregressive model. But what about using Kalman Filter structure? Or any other things such as gaussian process. I barely seen the reason why SSM is used here compared with using classical embedding space.\n\n3. The simplified approach is also ad-hoc. The model uses a special A. I understand that it may have the ability to approximate a lot of dynamic systems, but that neither explain if we can ``learn’’ this solution nor why any formula is not working.\n\n4. The Closed-loop Recurrent is a very interesting structure, but I have a hard time understanding why it could get a better fit compared with the original seq2seq-based decoding process. Is it increasing the ability to capture nonlinearity in the system? But transformers already have the ability to capture such nonlinearity. Then why did you achieve better performance? Any explanation?\n\n5. Is there any reason why you haven’t compared with time series classification approaches such as Rocket[1]?\n\n6. Besides, I believe an overall structure for the forecasting and classification model should be included in the supplement material.\n\n7. In the comparison experiment, the best baseline is NLinear (LSTM + Conv). Any code, reason, or explanation? Since this result is conflicted with existing published work (e.g. Informer, NBeats, FEDFormer results because they claim that they are better than LSTM because it cannot capture long-term data dependency). As a rule of thumb, I am led to believe existing published work compared with the result in this paper.\n\nOverall, the experiment is impressively good. But given all the concerns here, I believe the proposed structure is ad-hoc and how to get such an impressive performance is not convincing. ", "strength_and_weaknesses": "Strength\n1. The experimental result is impressively good\n\n\nWeaknesses\n1. Using SSM in the seq2seq is sort of ad-hoc, lacks intuition, and just applies some existing techniques. \n2. The simplified approach here (e.g. constrained A) is ad-hoc to me without sufficient explaination.\n3. Some claims seem incorrect and the experimental result seems weird to me.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FiwjhpN3es", "reviewer": "Reviewer_YRVv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new SSM formulation for time-series modeling that utilizes some explicit modeling of “companion” matrices to model the autoregressive nature of the time-series, and a “closed-loop” formulation that outputs the “input” for the next frames in forecasting. Experiments are conducted on several benchmark datasets for time-series forecasting and classification, in comparison to selected baseline models. \n", "summary_of_the_review": "This paper presents a SSM formulation for time-series forecasting that includes \"autoregressive\"m modeling in a state-space formulation and adds the prediction of the \"input\" to the forecasting model as an additional output. While improvements were demonstrated with respect to selected baselines, I found the premise for both of the two primary contributions unclear. Missing necessary baselines to either line of the related works, especially those in deep SSMs, further add to the concern.\n\nPost-rebuttal update: The extensive discussion and revision during the rebuttal phase helped me appreciate the contribution of the work better, because of which I'm raising my score.", "strength_and_weaknesses": "Strength\n\nThe formulation of the SSM presented appeared to be new. \n\nThe experiments considered a large number of forecasting and classification tasks. \n\nThe detailed breakdown analyses of each of the stated goals in 4.2 are appreciated. \n\nWeakness\n\n-The state-space formulation in (1-3) is somewhat unconventional. In existing SSM works, u_k in (1-2) would be corresponding to external controls applied to the system, and y_k the observations. In what presented in (1-3), u_k and y_k are essentially the same variables, one being what observed in the look-back windows, and one to be forecasted. This somehow mixes the “estimation” of the state-space modes with the “generative” formulation of the state-space models. The intuition for such a mix — as the premise of the work — is not clear.\n\n- The statement of existing SSMs not being able to capture the autoregressive process is not valid. Perhaps the authors are referring to when the state transition model is formulated as in (1-2) with a Markovian assumption? But the transition function can be much more general than by introducing a larger lag in time for the dependency. \n\n\n- The rationale behind the closed-loop modeling is also somewhat counter-intuitive. Essentially it is suggesting that the forecasting of y_K+1 is not going to be good enough for it to be used as the input for the future forecasting? What is the fundamental rationale to believe that (11) and (12) can be identified as two different models? If y_k’s are accurately forecasted, it should be sufficient to be used as inputs for future forecasting; the fact that another variable needs to be predicted to be used as \"input\" to the model means that y's would not be well estimated? \n\n- Some of the strong AR baselines such as N-BEATs and DeepAR are missed in the experiments. Since the paper has an emphasis of taking a SSM approach, comparison to existing SSM-based time-series forecasting models is also necessary. This again speaks to the need to demonstrate the benefits of the presented modeling that appears to be a unusual mix of a SSM and AR formulation. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "89GfvMGq45w", "reviewer": "Reviewer_BK6G", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to improve a linear state space model’s expressive power by maintaining a companion matrix, which encodes essential properties of the dynamics into a discrete set of quantities.", "summary_of_the_review": "I do not have the impression that the paper speaks to the target audience of ICLR. Its algorithmic details could be better fit to a signal processing venue. From a machine learning research perspective, what I can only see is a model that is not working charmingly well even in much simpler data sets than an average machine learning research paper uses.\n\nAfter rebuttal\n---\n\nI read the author response and decided to keep my score. The reply to the comparative strength of the proposed method to LSTM/GRU and the degree of difficulty of the studied data sets is not convincing. I still do not think the paper's content is relevant for the machine learning community.", "strength_and_weaknesses": "**Strengths:**\n\n  * The paper reports a rigorous empirical analysis of the proposed method, though the experiments are designed from a signal processing point of view rather than machine learning.\n\n**Weaknesses:**\n  \n  * The proposed method assumes a linear state space model, which is a major restriction for the wide-scope applicability of the method to general machine learning problems. It performs on par with an LSTM on the ECG problem probably because a preprocessing step on the signal, such as a Fourier Transform, makes the problem linearly solvable. This will not be the case for any other arbitrary machine learning problem. It is also likely that an LSTM cascade or GRU would give even better results, both of which are way below state of the art methods.\n  * The experimental results are rather inconclusive. The proposed method appears to work favorably on univariate forecasting tasks, which are not so much within the scope of machine learning research. In more challenging tasks such as ECG and speech audio classification, the proposed method does not appear to meet the state of the art. The paper also does not explain why a comparison against a transformer in Tables 2 and 3 was not possible. They are currently operational in much bigger processing pipelines than the ones used in this paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster