Liquid Structural State-Space Models
Authors: Ramin Hasani
Mathias Lechner
Tsun-Hsuan Wang
Makram Chahine
Alexander Amini
Daniela Rus
Venue: ICLR 2023 poster (2023)
Keywords: state-space models
liquid neural networks
time series. memory
recurrent neural networks

Abstract: A proper parametrization of state transition matrices of linear state-space models (SSMs) followed by standard nonlinearities enables them to efficiently learn representations from sequential data, establishing the state-of-the-art on an extensive series of long-range sequence modeling benchmarks. In this paper, we show that we can improve further when the structured SSM, such as S4, is given by a linear liquid time-constant (LTC) state-space model. LTC neural networks are causal continuous-time neural networks with an input-dependent state transition module, which makes them learn to adapt to incoming inputs at inference. We show that by using a diagonal plus low-rank decomposition of the state transition matrix introduced in S4, and a few simplifications, the LTC-based structured state-space model, dubbed Liquid-S4, improves generalization across sequence modeling tasks with long-term dependencies such as image, text, audio, and medical time-series, with an average performance of 87.32\% on the Long-Range Arena benchmark. On the full raw Speech Command recognition dataset, Liquid-S4 achieves 96.78\% accuracy with a 30\% reduction in parameter counts compared to S4. The additional gain in performance is the direct result of the Liquid-S4's kernel structure that takes into account the similarities of the input sequence samples during training and inference.

Published as a conference paper at ICLR 2023
LIQUID STRUCTURAL STATE-SPACE MODELS
Ramin Hasani CSAIL, MIT
Mathias Lechner CSAIL, MIT
Tsun-Hsuan Wang CSAIL, MIT
Makram Chahine CSAIL, MIT
Alexander Amini CSAIL, MIT
Daniela Rus CSAIL, MIT
ABSTRACT
A proper parametrization of state transition matrices of linear state-space mod- els (SSMs) followed by standard nonlincarities enables them to efficiently learn representations from sequential data, establishing the state-of-the-an on an exten- sive series of long-range sequence modeling benchmarks. In this paper, we show that we can improve further when the structured SSM, such as S4, is given by a linear liquid time-constant (LTC) state-space model. LTC neural networks are causal continuous-time neural networks with an input-dependent state transition module, which makes them learn to adapt to incoming inputs at inference. We show that by using diagonal plus low-rank decomposition of the state transition matrix introduced in S4, and few simplifications, the LTC-based structured state- space model, dubbed Liquid-S4, improves generalization across sequence model- ing tasks with long-term dependencies such as image, text, audio, and medical time-series, with an average performance of 87.32% on the Long-Range Arena benchmark. On the full raw Speech Command recognition dataset, Liquid-S4 achieves 78% accuracy with a 30% reduction in parameter counts compared S4. The additional gain in performance is the direct result of the Liquid-S4 's ker- nel structure that takes into account the similarities of the input sequence samples during training and inference.
INTRODUCTION
Learning representations from sequences of data requires expressive temporal and structural credit assignment. In this space, the continuous-timg neural network class of liquid time-constant networks (LTC) (Hasani et al., 2021b) has shown theoretical and empirical evidence for their expressivity and their ability to capture the cause and effect of given task from high- nigh-dimensiona sequential demon- strations (Lechner et al., 2020a; Vorbach et al., 2021; Wang et al., 2022; Hasani et al., 2022; Yin et al., 2022). Liquid networks are nonlinear state-space models (SSMs) with an input-dependent state transition module that enables them to learn adapt the dynamics of the model to incom- ing inputs, at inference, as they are dynamic causal models (Friston et al., 2003). Their complexity, however, is bottlenecked by their differential equation numerical solver that limits their scalability to longer-term sequences. How can we take advantage of LTC's generalization and causality capabil- ities and scale them to competitively learn long-range sequences without gradient issues, compared to advanced recurrent neural networks (RNNs) (Rusch & Mishra, 202 la; Erichson et al., 2021; Gu al., 2020a), convolutional networks (CNNs) (Lea al., 2016; Romero al., 202 2b: Cheng al., 2022), and attention- based models ( (Vaswani et al., 2017)?
In this work, we set out to leverage the elegant formulation of structured state-space models (S4) (Gu et al., 2022a) to obtain linear liquid network instances that possess the approximation capabilities of both $4 and L.Css. This is because structured SSMs are shown largely dominate advanced RNNs, CNNs, and Transformers across many data modalities such as text, sequence of pixels, audio, and time series (Gu et al., 2021: 2022a;b; Gupta, 2022). structured SSMs achieve such impressive per- formance by using three main mechanisms: 1) High-order polynomial projection operators (HiPPO)
correspondence to: rhasan i @mi edu indicates authors with equal contributions.
Code: https: / //g
rami nmh /1 i d-

---

Published as a conference paper at ICLR 2023
(Gu et al., 2020a) that are applied to state and input transition matrices memorize signals' history, 2) diagonal plus low-rank parametrization of the obtained HipPO (Gu et al., 2022a), and 3) an ef- ficient (convolution) kernel computation of an SSM's transition matrices in the frequency domain, transformed back in time via an inverse Fourier transformation (Gu et al., 2022a).
To combine S4 and LTCs, instead of modeling sequences by linear state-space models of the form é = A x + B u, = C æ; (as done in structured and diagonal SSMs (Gu et al., 2022a;b)), we propose to use linearized LTC state-space model (Hasani et al., 202Ib), given by the following dynamics: & (A + B u) a + B u, y = C æ. We show that this dynamical system can also be efficiently solved via the same parametrization of S4, giving rise to an additional convolutional Kernel that accounts for the similarities of lagged signals. We call the obtained model Liquid- S4. Through extensive empirical evaluation, we show that Liquid-S4 consistently leads to better generalization performance compared to all variants of $4, CNNs, RNNs, and Transformers across many time-series modeling tasks. In particular, we achieve SOTA performance on the Long Range Arena benchmark (Tay et al., 2020b). To sum up, we make the following contributions:
1. We introduce Liquid-S4, -S4, a new state-space model that encapsulates the generalization and causality capabilities of liquid networks well as the memorization and scalability of S4, performance on pixel-level sequence classification, text, speech
2. We achieve state-of-the recognition, and all six tasks of the long-range arena benchmark with an average accuracy 87 87.32%. On the full raw Speech Command recognition dataset, Liquid-S4 achieves 78%78% accuracy with 30% reduction parameters. Finally, on the BIDMC vital signs dataset, Liquid-S4 achieves SOTA in all modes.
2 RELATED WORKS
Learning Long-Rand Range Dependencies with RNNs. Sequence modeling can be performed autore- gressively with RNNs which possess persistent states (Little, 1974) originated from Ising (Brush, 1967) and Hopfield networks (Hopfield, 1982; Ramsauer et al., 2020). Discrete RNNs approxi- mate continuous dynamics step-by-steps via dependencies on the history of their hidden states, and continuous-time (CT) RNNs use ordinary differential equation (ODE) solvers to unroll their dynam- ies with more elaborate temporal steps (Funahashi & Nakamura, 1993).
CT-RNNs can perform remarkable credit assignment in sequence modeling problems both on reg- ularly sampled, trreullaly-samppler data (Pearson et al., 2003; Li & Marlin, 2016; Belletti et al., 2016; Roy & Yan, 2020; Foster, 1996; Amigó et al., 2012; Kowal et al., 2019), by turning the spatiotemproal dependencies into vector fields (Chen et al., 2018), enabling better generalization and expressivity (Massaroli et al., 2020; Hasani et al., 2021b). Numerous works have studied their characteristics to understand their applicability and limitations in learning sequential data and flows (Lechner et al., 2019; Dupont et al., 2019; Durkan et al., 2019; Jia & Benson, 2019; Grunbacher et al., 2021; Hanshu et al., 2020; Holl et al., 2020; Quaglino et al., 2020; Kidger et al., 2020; Hasani et al., 2020; Liebenwein et al., 2021; Gruenbacher et al., 2022).
However, when these RNNs are trained by gradient descent (Rumelhart et al., 1986; Allen-Zht & Li, 2019; Sherstinsky, 2020), they suffer from the vanishing/exploding gradients problem, which makes difficult the learning of long-term dependencies in sequences (Hochreiter, 1991; Bengio et al., 1994). This issue happens in both discrete RNNs such as GRU-D with its continuous delay mechanism (Che et al., 2018) and Phased-LSTMs (Neil et al., 2016), and continuous RNNs such as ODE-RNNs (Rubanova al., 2019), GRU-ODE (De Brouwer al., 2019), Log-ODE methods (Morrill et al., 2020) which compresses the input time-series by time-continuous path signatures (Friz & Victoir, 2010), and neural controlled differential equations (Kidger et al., 2020), and liquid time-constant networks (LTCs) (Hasani et al., 2021b).
Numerous solutions have been proposed to resolve these gradient issues to enable long-range depen- dency learning. Examples include discrete gating mechanisms in LSTMs (Hochreiter & Schmidhu- ber, 1997; Greff et al., 2016; Hasani et al., 2019), GRUs (Chung et al., 2014), continuous gating mechanisms such CICs (Hasani et al., 2021a), hawks LSTMs (Mei & Eisner, 2017), IndRNNs (Li et al., 2018), state regularization (Wang& & Niepert, 2019), unitary RNNs (Jing et al., 2019), dilated RNNs (Chang al., 2017), long memory stochastic processes (Greaves-Tuunell & Har- chaoui, 2019), recurrent kernel networks (Chen et al., 2019). Lipschitz RNNs (Erichson et al., 2021), symmetric skew decomposition (Wisdom et al., 2016), infinitely many updates in iRNNs
2

---

Published as a conference paper at ICLR 2023
(Kag et al., 2019), coupled oscillatory RNNs (coRNNs) (Rusch & Mishra, 2021a), mixed-memon RNNs (Lechner & Hasani, 2021), and Legendre Memory Units (Voelker et al., 2019).
Learning Long-range Dependencies with CNNs and Transformers. RNNs are not the only so- lution to learning long-range dependencies. Continuous convolutional kernels such as CKConv (Romero et al., 2021b) and (Romero et al., 2021a), and circular dilated CNNs (Cheng et al., 2022) have shown to be efficient in modeling long sequences faster than RNNs. There has also been a large series of works showing the effectiveness of attention-based methods for modeling spatiotemporal data. A large list of these models is listed in Table 6. These baselines have recently been largely outperformed by the structured state-space models (Gu et al., 2022a).
State-Space Models. SSMs are well-established frameworks to study deterministic and stochastic dynamical systems (Kalman, 1960). Their state and input transition matrices can be directly learned by gradient descent to model sequences of observations (Lechner et al., 2020b; Hasani et al., 202 lb; Gu al., 2021). In a seminal work, Gu al. (2022a) showed that with a couple of fundamental algorithmic methods most powerful sequence modeling framework to-date, outperforming advanced RNNs, temporal and continuous CNNs (Cheng et al., 2022; Romero et al., 2021b;a) and a wide variety of Transformers (Vaswani et al., 2017), available in Table by a significant margin.
memorization and computation of input sequences, SSMs can turn into the
The key their numerical performance is their derivation of higher-order polynomial projection (HiPPO) matrix (Gu et al., 2020a) obtained by a scaled Legendre measure (LegS) inspired by the Legendre Memory Units (Voelker et al., 2019) to memorize input sequences. Their efficient runtime and memory are derived from their normal plus-low rank representation.It was also shown recently that diagonal SSMs (S4D) (Gupta, 2022) could be as performant as S4 in learning long sequences when parametrized and initialized properly (Gu et al., 2022b;c). Concurrent with our work, there is also a new variant of S4 introduced simplified-S (S5) (Smith et al., 2022) that tensorizes the 1-D operations of S4 to gain a more straightforward realization of SSMs. Here, we introduce Liquid- S4, which is obtained by more expressive SSM, namely liquid time-constant (LTC) representation (Hasani et al., 2021b) which achieves SOTA performance across many benchmarks.
3 SETUP AND METHODOLOGY
In this section, we first revisit the necessary background to formulate our liquid structured state- space models. We then set up and sketch our technical contributions.
3.1 BACKGROUND: STRUCTURED STATE-SPACE MODELS (S4)
We aim to design an end-to-end sequence modeling framework built by SSMs. A continuous-timg
SSM representation of a linear dynamical system given by the following set of equations:
(t) (x)) + (t)), ytt = a+()) + D (t).
Here, ætt) is an N-dimensional latent state, receiving a I-dimensional input signal u(t), and com- puting a I-dimensional output signal y()). A (NxN), B(Nx1), ((INN) and D(Ix are system's parameters. For the sake of brevity, throughout our analysis, we set D 0 as can be added eventually after construction of our main results in the form of a skip connection (Gu et al., 2022a).
Discretization of SSMs. In order to create a sequence-to-sequenee neural network (RNN), we discretize the continuous-time representation of SSMs by the trapezoidal rule (bilinear transform) as follows (sampling step = öt) (Gu et al., 2022a):
model similar to a recurrent
= A I + B Ujs
yk = C Izk
This is obtained via the following modifications to the transition matrices:
8t A)
öt A),
8t A) S. C-C
A= (I
-
B =
With this transformation, we constructed a discretized seq-2-seq model that can map the input ua to output V/s, via the hidden state 2; E IR.. A is the hidden transition matrix, and C are input and output transition matrices, respectively.
3

---

Published as a conference paper at ICLR 2023
Creating Convolutional Representation of SSMs. The system described by Eq. and Eq. 3, can be trained via gradient descent to learn to model sequences, in a sequential manner which is not scalable. To improve this, we can write the discretized SSM in Eq. 2 as a discrete convolutional kernel. To construct the convolutional kernel, let us unroll the system of Eq. in time as follows, assuming zero initial hidden states 1- 0:
Zo Buo,
21 AB Buj
T2 A Buo + AB Bug
(4)
2 1/2 CA Bug + CABu: + CBu2,
CBuo,
V1 = CABuo + CBui,
The mapping u - yk can now be formulated into a convolutional kernel explicitly:
Uk = CA Buo + CA
Buj +... CABu,-1 + CBug:
K
(5)
K : (CA (E(1] BEi
(CB, CAB, CA
CA B)
(6)
Eq. 5 is a non-circular convolutional kernel. Gu et al. (2022a) showed that under the condition that K is known, it can be solved very efficiently by a black-box Cauchy kernel computation pipeline.
Computing S4 Kernel Efficiently: Gu et al. (2022a) showed that the S4 convolution kernel could be computed efficiently using the following elegant parameterization tricks:
To obtain hetter representations in sequence modeling schemes by SSMs, instead of ran- domly initializing the transition matrix A, we can use the normal plus low-Rank (NPLR) matrix below, called the Hippo Matrix (Gu al., 2020a) which is obtained by the scaled Legendre measure (LegS) (Gu et al., 2021; 2022a):
(2n + (2k + 1)1/2
if > k
(HiPPO Matrix)
Ank
n n+1
if =
(7)
if <
The NPLR representation of this matrix is the following (Gu et al., 2022a):
A VAV" PQ V (A (V*P) Q)") V
(8)
Here, V E CNNN is unitary matrix, A is diagonal, and P,Q Q RNr the low- rank factorization. Eq. 7 is normal plus low rank with r = 1 (Gu et al., 2022a). With the decomposition in Eq. we can obtain A over complex numbers in the form of diagonal plus low-rank (DPLR) (Gu et al., 2022a).
Vectors Bn and P. are initialized by B, (2n + 1) and P. (n + 1/2) (Gu et al., 2022b). Both vectors are trainable.
Furthermore, it was shown in Gu al. (2022b) that with Eq. 8, the eigenvalues of A might be on the right half of the complex plane, resulting in numerical instability. To resolve this, Gu al. (2022b) recently proposed use the parametrization A PP instead of A PQ"
Computing the powers of A in direct calculation of the S4 kernel K is computationally expensive. $4 computes the spectrum of K instead of direct computations, which reduces the problem of matrix powers to matrix inverse computation (Gu et al., 2022a). S4 then computes this convolution kernel via black-box Cauchy Kernel efficiently, and recovers K by inverse Fourier Transform (iFFT) (Gu et al., 2022a).
3.2 LIQUID STRUCTURAL STATE-SPACE MODELS
In this work, we construct convolutional kernel corresponding to a linearized version of LTCs (Hasani et al., 2021b); an expressive class of continuous-time neural networks that demonstrate attractive generalizability out-of-distribution and are dynamic causal models (Vorbach et al., 2021; Friston et al., 2003; Hasani et al., 2020). In their general form, the state of liquid time-constant network at each time-step is given by the set of ODEs described below (Hasani et al., 2021b):
dx((t) dl
A + B ② (x((1) (,),, Ox)(1 + BO f(x(1), u(),,,0).
(9)
Liquid time-constant
4

---

Published as a conference paper at ICLR 2023
In this expression, z (t) is the vector of hidden state of size N, (t) is input signal with features, A (Nx1) is time-constant state-transition mechanism, ((Xx1) is bias vector, and @ represents the Hadamard product. f(.) is bounded nonlinearity parametrized by 0.
Our objective is to show how the liquid time-constant (i.e., an input-dependent state transition mech- anism in state-space models can enhance its generalization capabilities by accounting for the covari- ance of the input samples. To do this, linearize the LTC formulation of Eq. 9 in the following to better connect the model to SSMs. Let's dive in:
Linear Liquid Time-Constant State-Space Model. A Linear LTC SSM be presented by the following coupled bilinear (first order bilinear Taylor approximation (Penny et al., 2005)) equation:
a (1) = [ + B v() (()) v(t),
v(() = C (())
(10)
Similar to Eq. 1, (t) is an N-dimensional latent state, receiving a I-dimensional input signal u(t), and computing I-dimensional output signal y(t)- A (NxN), ((Nx1), and C(IxN) Note that D is to zero for simplicity. In Eq. 10, JNv is x N unit matrix that adds utt) element-wise to A. This dynamical system allows the coefficient (state transition compartment) of state vector x(t) be input dependent which, as a result, allows to realize more complex dynamics.
Discretization of Liquid-SSMs. We can use forward Euler transformation to discretize Eq. 10 into the following discretization:
T = A + Buk) 2k-1 + Uk.
y/s C I
(11)
The discretized parameters would then correspond to: - A. B öt and C C, which are function of the continuous-time coefficients A, B, and C, and the discretization step öt. Given the properties of the transition matrices A and B, and ranges of ő1, we could use the stable bilinear discretization of matrices A and B, of Eq. 3 as well, as the Forward Euler discretization and the bilinear transformation of A and B presented in Eq. stay close each other (Appendix D).
Creating a Convolutional Representation Liquid-SSMs. Similar to Eq. 4, we first unroll the Liquid-SSM in time to construct a convolutional kernel of it. By assuming x = O, we have:
Zy = Buc,
yo
CBup
ABuo
Bui
B unuis
CABuo
CBui CB ugui
(12)
22 A'Ba + ABui Buz+
AB
uoui + AB unug B un ua
3/2 = CA " + CAB + CBug+ CAB ugu1 + CAB uou2 + CB u1 + CB Ng
The resulting expressions of the Liquid-SS -SMid at each time step consist of two types of weight figurations: 1. Weights corresponding to the mapping of individual time instances of inputs inde- pendently, shown in black in Eq. 12, and 2. Weights associated with all orders of auto-correlation of the input signal, shown in violet in Eq. 12. The first set of weights corresponds to the convolutional kernel of the simple SSM, shown by Eq. and Eq. 6, whereas the second set leads the design of an additional input correlation kernel, which we call the liquid kernel. These kernels generate the following input-output mapping:
U/
CA
Buo
CA
Bug + CAB CBug
Σ
CA (K+1 (+41-p-1)
Up
(13)
2=2 uiuttt ... Mp (.k+pppp
for E Z and i
y K
K
Rigudd Uepercattooss
Here, [[( I. represents (k+1) permuted indices. For instance, let us assume we have a 1- dimensional input signal u(t) of length = 100 on which we run the liquid- kernel. We set the hyperparameters P 4. This value represents the maximum order of the correlation terms
5

---

Published as a conference paper at ICLR 2023
Algorithm LIQUID-S4 KERNEL The S4 convolution kernel (highlighted in black) is used from Gu et al. (2022a) and et al. (2022b). Liquid kernel computation highlighted in purple.
Input: S4 parameters A, P, B, C E C.A, step size A, liquid kernel order P, inputs seq length I, liquid kernel sequence length L
Output: SSM convolution kernel K (A,, and SSM liquid kemell K Kisquid
Kg (A, B, C) for A = A PP* (Eq. 6)
1:
C
(I-A" A-AA
C
Truncate SSM generating function (SSMGF) length L
2: (oo(w)
Roz (w)
CPP
A A)
P]
Black-box Cauchy kernel
k1o(w)
kw)(w)
3: ((w)= + 14w (kuo(w) kon (w) kn(ww) [(m(w))
2
Woodbury Identity
4: K [R(w): :w]]:-- exp(2Ti
Evaluate SSMGF at all roots of unity w E S2
5: K < iFFT(K)
> Inverse Fourier Transform Liquid-S4 Kernel as shown Eq. 14
6: if Mode KB then
7:
for in 12.....77 P) do
8:
K iiquid=
* + K ((L-L, L,1) 3 B (1-1,11)
> Jg is backward identity matrix
9:
K apppid K liqui=p))
10: end for
11:
else if Mode == PB then
Liquid-S4 Kernel of Eq. 14 with A reduced to Identity.
12: for in ,2. P)- do
K liquid=p =C B B
13:
(L-LL)
14:
K liquaa-appe appped- K lquid=p)
end for
16:
end if
would want take into account output a decision. This means that the signal Mccrrelations in Eq. 13 will contain all combinations of 2 order correlation signals (4++)) u, order (~+)). uetty un and 4 order signals (L+t1), uuu The kernel weights corresponding to this auto-correlation signal given in Appendix A. This additional kernel takes the temporal similarities of incoming input samples into consideration. This way, Liquid-SSM gives rise to a more general sequence modeling framework. The liquid convolutional kernel, Kiguidd is as follows:
Kiggudd E IR K.C = (CA (L- (e(( PE(P) D')elDDDii = (CA
LBB ..... CB"
(14)
How to compute Liquid-S4 kernel efficiently? Kriquia possess similar structure to the S4 kernel. In particular, we have:
Proposition 1. The Liquid-S4 kernel for each order p E P, Küguid can be computed by the anti-diagonal transformation (flip operation) of the product of the S4 convolution kernel, K (CB,CAB ..... CAB...... CA B). and vector =-1 E R.....
The proof given in Appendix. Proposition indicates that the Liquid-s4 kernel can be obtained from the precomputed S4 kernel and Hadamard product of that kernel with the transition vector B powered by the chosen liquid order. This is illustrated in Algorithm 1, lines 6 to 10, corresponding mode we call KB, which stands for Kernel B.
Additionally, we introduce a simplified Liquid-S4 kernel that is easier to compute while is as ex- pressive as or even better performing than the KB kernel. To obtain this, we set the transition matrix A in Liquid-S4 of Eq. 14, with identity matrix, only for the input correlation terms. This way, the Liquid-s4 Kernel for a given liquid order E P reduces to the following expression:
(Liquid-S4- PB)
Kiguid=p
CR_ K2(C. B) == (CB")
iE(f.), pE(P)
(15)
We call this kernel Liquid-S4 PB, as it is obtained by powers of the vector B. The computational steps to get this kernel is outlined in Algorithm lines 11 to 15.

---

Published as a conference paper at ICLR 2023
Table 1: Performance on Long Range Arena Tasks. Numbers indicate validation accuracy (standard deviation). The accuracy of models denoted by is reported from (Tay et al., 2020b). Methods denoted by ** are reported from (Gu et al.. 2022a). The rest of the models' performance results are reported from the cited paper. See Appendix for accuracy on test set.
Model
ListOps 2048
IMDB
AAN 4000)
CITAR 1024
Pathfinder 1024
Path-X
Avg.
(iuput/ength)
21488
16:34
Random"
10.00
50,00
50.00
10.00
50.00
50,00
36.67
Transformer" (Vaswani ot 2017)
36.37
64.27
$7,46
42.44
71.40
54,39
Local Ant" (Tay et
15.82
Sparse Transformer (Child et al., 2019)
17.07
63.58
59.59
44.24
71.71
X
$1.24
Longformer" (Beltugy et 2020)
35.63
62.85
56.89
42,22
69.71
53,46
Linformer" (Wang
2020)
16.13
65,90
$3.09
75.30
50.55
Reformer (Kitacy ct al.,, 2019)
37.27
56.10
$3,40
38.07
68.50
50,56
Sinkhorn Trans." (Tay et 2020s)
33.67
61.20
53,83
41.25
67.45
51,23
BigBird" (Zaheer et 2020)
36.05
64.02
59,29
55.01
Lincur Trams." (Kutharooollos et al., 2020)
16.13
65,90
53.09
42,34
75.30
$0.46
Performer' (Choromanski al., 2020)
18.01
65,40
53.82
42.77
77.05
51,18
FNet" (Lee-Thorp et al., 2021)
35.33
65.11
59.61
38.67
77.80
54,42
Nyströmformer" (Kiong al. 2021)
37.15
65.52
79,56
41,58
70.94
57,46
Luna-256"
(2021)
37.25
79.29
59.37
H-Trunsfommrr-l ID" (Zhu Soricut, 2021)
49.53
78.69
63.99
46.05
68.78
61.41
CDIL (Cheag al., 2022)
44.05
86.78
85,36
66,91
91.70
74.96
DSS (Gupta, 2022)
57.6
76.6
87.6
85.8
84.1
85.0
79.45
S4 (original)" (Gu et 2022a)
$8.35
76.02
87.09
87.26
86.05
88. 10
$0.48
$4- S4-Leg8 (Gu al., 2022b)
59.60 (0.07)
86.82
90:90
88.65 (0.23)
94.20 (0.25)
96.35
86.09
S4-FoeT (Gu et al., 20226)
57.88 (1.90)
86.34 (0.31)
89.56 (0.88)
89.07 (0.19)
94.46 (0.26)
x
77.90
$4- s4g1egS/ (Gu al., 2022:)
60.45 (0.75)
86.78 (0.26)
90.30 (0.28)
89.00 (0.26)
94.44 (0.08)
78.50
S4D-LegS
20226)
60.47
(1.24)
84,
S4D-Inv (Gu et al., 20226)
60.18 (0.35)
87.34 (0.20)
91 (0.01)
87.83 (0.37)
93.78 (0.25)
92.80
85.50
S4D-Lin (Gu et 20226)
60.52 (0.51)
86.97 (0.23)
90.96 (0.09)
87.93 (0.14)
93.96 (0.00)
78.39
original (Smith al., 2022)
86.51
86.
87.57
85,25
82.46
$5 new (Smith al., 2023)
62. 15
89.31
91.40
88.00
95.33
98.58
87.46
Liquid-S4-KK (ours)
62,55 (0,10)
88. (0.02)
91.10 (0.05)
89.37 (0.22)
94.50 (0.08)
96.10
87.09
Liquid-44. (ours)
62.75 (0.20)
(0.04)
(0.01)
(0.40)
(0.20)
$7.32
p=5
pe6
p=2
p=3
p=2
p=2
Computational Complexity of the Liquid-S4 Kernel. The computational complexity of the S4- Legs Convolutional kernel solved via the Cauchy Kernel is Ö(N + L), where N the state-size, and L is the sequence length [Gu et al. (2022a), Theorem 3]. Liquid-S4 S4 both in KB and PB modes can be computed in Ö + L Pmma L). The added time complexity in practice is tractable. This is because we usually select the liquid orders, to be less than 10 (typically Pmaz 3, and L which the number of terms we use to compute the input correlation vector, Mcorrelation. is typically two orders of magnitude smaller than the sequence length.
4 EXPERIMENTS WITH LIQUID-S4
In this section, we present an extensive evaluation of Liquid-S4 on sequence modeling tasks with very long-term dependencies and compare its performance large series of baselines ranging from advanced Transformers and Convolutional networks to many variants of state-space models. In the following, we first outline the baseline models we compare against. We then list the datasets we evaluated these models on and finally present results and discussions.
Baselines. We consider broad range of advanced models compare Liquid-S4 with. These baselines include transformer variants such as vanilla and Sparse Transformers, Transformer model with local attention, Longformer, Linformer, Reformer. Sinkhorn, BigBird, Linear Transformer, and Performer. We also include architectures such as FNets, Nystromformer, Luna-256, H-TTranfformer- 1D, and Circular Diluted Convolutional neural networks (CDIL.) We then include a full series of state-space models and their variants such as diagonal SSMs (DSS), S4, S4-legS, S4-FouT, S4- LegS/FouT, S4D-LegS, S4D-Inv, S4D-Lin and the Simplified structured state-space models (S5).
Datasets. We first evaluate Liquid-S4's performance on the well-studied Long Range Arena (LRA) benchmark (Tay et al., 2020b), where Liquid-Sa outperforms other S4 and S4D variants in every task with average accuracy of 87.32%. LRA dataset includes six tasks with sequence lengths ranging from 1k to 16k. Concurrent work We then report Liquid-S4's performance compared to other S4, and S4D variants, well other models, the BIDMC Vital Signals dataset (Pimentel et al.,
7

---

Published as a conference paper at ICLR 2023
2016; Goldberger et al., 2000). BIDMC uses bio-marker signals of length 4000 to predict Heart rate (HR), respiratory rate (RR), and blood oxygen saturation (SpO2). We also experiment with the sCIFAR dataset that consists of the classification of flattened images in the form of 1024-long sequences into ten classes.
Finally, perform Raw Speech Command (SC) recognition with full 35 labels as conducted very re- cently in the updated S4 article (Gu al., 2022a).
Table 2: Performance on BIDMC Vital Signs dataset. Num- bers indicate RMSE on the test set. Models denoted by is reported from (Gu et al., 2022b). The rest of the models' performance results are reported from the cited paper.
is essential to de- note that there is a modified speech command dataset that restricted the dataset to only ten output classes and used in a couple of works (see for example Kidger et al. (2020); Gu et al. (2021); Romero et al. (2021b;a)). Aligned with the updated results reported in Gu et al. (2022a) and Gu et al. (2022b), we choose not to break down this dataset and use the full-sized benchmark. SC dataset contains sequences of length 16k to be classified into 35 commands. Gu et al. (2022a) introduced new test case setting to assess the performance of models (trained on 16kHz sequences) on sequences of length 8kHz. S4 and S4D perform excep-
BIDMC
UnICORNN (Rusch & Mishra, 2021b)
1.39
1.06
0.869
coRNN (Rusch & Mishrn, 2021a)
1.81
1.45
CKCrev"
2,05
1.214
1,051
NRDE (Merill et al., 2021)
2.97
1.49
1.29
LSTM (Rusch & Mishra, 2021b)
10.7
2.28
Transformer"
12.2
XGBoost (Tan al., 2021)
4.72
1.57
1.52
Random Forest (Tan et al., 2021)
5.00
1.89
1.74
Ridge Regress. (Tan al., 2021)
S4-LegS" (Gu et al., 20226)
0.332 (0.013)
0.247 (0.062)
0.090 (0.006)
S4-HouT" (Gu et al., 20226)
0,339 (0.020)
0:301 (0.050)
0.06% (0.003)
S4D- egS (Gu
20226)
0.367 (0.001)
0.248 (0.036)
0,102 (0.00))
S4-(ogg/FouT) (Gu at. 20226)
0.344 (0.032)
0.163 (0.008)
0,880 (0.007)
S4D-Iay (Gu et al., 2022b)
0.373 (0.024)
0.254 (0.022)
(0.00().001)
S4D-Lin" (Gu al., 2022h)
0.379 (0.006)
0.226 (0.008)
0.114
Liquid-S4-Ke SS-uid-S--KBB-- (ours)
0.310 (0.001)
0.162 (0.001)
0.068 (0.002)
Liquid-S4 PB (ours)
0.303 (0.002)
0.158 (0.001)
0.0666 (0.002)
p=i
tionally well in this zero-shot test scenario.
4.1 Results on Long Range Arena
ListOpp Long-Range Arena
IMDB Long-Ranne Arena
80.0
Table depicts a comprehensive list of baselines benchmarked against each other on six long-range se- quence modeling tasks in LRA. We observe that Liquid-S4 instances (all use the PB kernel with scaled Leg- endre (LegS) configuration) with a small liquid order, p. ranging from 2 to 6, consistently outperform all base- lines in all six tasks, establishing the new SOTA on LRA with an average performance of 87.32%. In particular, on ListOps, Liquid-S4 improves S4-LegS performance by more than 3%, on character-level IMDB by 2.2%, and on 1-D pixel-level classification (CIFAR) by 0.65%, while establishing the-state-of the-art on the hardest L.A task by gaining 96.66 % accuracy. Liquid-S4 performs on par with improved S4 and S4D instances on both AAN and Pathfinder tasks. The performance of SSM models is generally well-beyond what advanced Transformers, RNNs, and Convolutional networks achieve on LRA tasks, with the Liquid-S4 S4 variants standing on top.
88.5
Accursay
88.0
87.5
8
87.0
B4-lags
Upiin pat
pat
Lipii pag
Lipar the pot
Lppes th4 pid
Scilng Liaod pez
Utith pud
pet pas
Liees pn pre
Ubaks
Mode
Figure 1: Performance vs Liquid Order in Liquid-S4 for A) ListOps, and B) IMDB datasets. More in Appendix. (n=3)
The impact of increasing Liquid Order p. Figure I illustrates how increasing the liquid order, p. can improve performance on ListOps and IMDB tasks from LRA (More results in Appendix).
4.2 Results on BIDMC Vital Signs
Table 2 demonstrates the performance of variety of classical and advanced baseline models on the BIDMC dataset for all three heart rate (HR), respiratory rate (RR), and blood oxygen saturation (SpO2) level prediction tasks. We observe that Liquid-s4 with PB kernel of order p = 3. 2, and = 4, perform better than all S4 and S4D variants. It is worth denoting that Liquid-S4 is built by the same parametrization as S4-LegS (which is the official S4 model reported in the updated S4 report (Gu et al., 2022a)). In RR, Liquid-S4 outperforms S4-LegS by a significant margin of 36%. On SpO2, Liquid- S4 performs 26.67% better than S4-Legs. On HR, Liquid-S4 outperforms S4-Legs by 8.7% improvement in performance.
8

---

Published as a conference paper at ICLR 2023
4.3 Results Image Classification
Table 3: Performance on SCIFAR. Numbers indi-
cate Accuracy (standard deviation). The bascline models are from Table 9 of Gu et al. (2022b).
Similar to the previous tasks, a Liquid-S4 net- work with PB kemel of order = 3 outper- forms all variants of S4 and S4D while being significantly better than Transformer and RNN baselines as summarized in Table 3.
Model
Accuracy
Transformer (Trinh al., 2018)
62.2
FlexConv (Romero et al., 2021a)
80.82
TrellisNet (Bai et al., 2018)
73.42
4.4 Results Speech Commands
LSTM
63.01
T-LTTM (Trinh el al., 2018)
72.2
Table 4 demonstrates that Liquid-S4 with = 2 achieves the best performance amongst all benchmarks on the 16KHz testbed. Liquid- S4 also performs competitively on the half- frequency zero-shot experiment, while it does not realize the best performance. Although the task is solved to a great degree, the reason could be that liquid kernel accounts for covariance terms. This might influence the learned repre- sentations a way that hurts performance by small margin in this zero-shot experiment. The hyperparameters are given in Appendix.
UR UR-GRU (Gu et al., 2020b)
74.4
HiPPO-RNN (Gu al., 2020a)
61.1
LipschizRNN (Erichson et al., 2021)
64.2
S4-LegS (Gu et al., 2022b)
91.80 (0.43)
84-FouT (Gu et al., 20226)
91.22 (0.25)
S4-(LegS/FouT) (Gu et al., 2022b)
91.58 (0.17)
S4D-LegS (Gu et al., 2022h)
89.92 (1.69)
S4D-Inv (Gu el al., 2022b)
90.69 (0.06)
S4D-Lin (Gu ct al., 2022b)
90.42 (0.03)
S5 Smith et al. (2022)
89.66
Liquid-S4-K0 (ours) Liquid-S4-PB (ours)
91.86 (0.08)
92.02 (0.14)
p=3
It is essential to denote that there is modified speech command dataset that restricts the dataset to only ten output classes, namely SC10, and is used in a couple of works (see for example (Kidger et al., 2020; Gu et al., 2021: Romero et al., 2021b;a)). Aligned with the updated results reported in (Gu et al., 2022a) and (Gu et al., 2022b), we choose not to break down this dataset and report the full-sized benchmark in the main paper. Nevertheless, we conducted experiment with SC10 and showed that even on the reduced dataset, with the same hyperparameters, we solved the task with a SOTA accuracy of 98.51%. The results are presented in Table 7.
CONCLUSIONS
We showed that the performance of structured state-space models could be considerably improved if they are formulated by linear liquid time-constant kernel, namely Liquid-S4. Liquid-S4 kernels are obtainable with minimal effort, with their kernel computing the similarities between time-lags of the input signals in addition to the main S4 diagonal plus low-rank parametrization. Liquid-S4 kernels with smaller parameter counts achieve SOTA performance all six tasks of the Long-range arena dataset, on BIDMC heart rate, respiratory rate, and blood oxygen saturation, on sequential 1-D pixel- level image classification, and on Speech command recognition. As final note, experimental evaluations suggest that for challenging multivariate time series and modeling complex signals with long-range dependencies, SSM variants such Liquid-S4 dominate other baselines, while for image and text data, a combination of SSMs and attention might enhance model quality.
Table 4: Performance on Raw Speech Command dataset with Full 35 Labels. Numbers indicate Accuracy on test set. The baseline models are reported from Table 11 of (Gu et al., 2022b).
Model
Parameters
16000Hz
8000Hz
InceptionNet (Nonaka & Scita, 2021)
481K
61.24 (0.69)
05.18 (0.07)
ResNet-18
216K
77.86 (0.24)
08.74 (0.57)
XResNet-50
904K
83.01 (0.48)
11.72 (0.39)
ConvNet
26.2M
95.51 (0.18)
07.26 (0.79)
S4-LegS (Gu et al.. 2022b)
307K
96.08 (0.15)
91.32 (0.17)
S4-FouT (Gu et al., 2022b)
307K
95.27 (0.20)
91 59 (0.23)
S4-(1eggS/ooTT) (Giu et al., 2022b)
307K
95.32 (0.10)
90.72 (0.68)
S4D-Legg (Gu al., 20226)
306K
95.83 (0.14)
08 (0.16)
S4D-Inv (Gu et al., 2022b)
306K
96.18 (0.27)
91.80 (0.24)
S4D-Lin (Gu et al., 2022b)
306K
96.25 (0.03)
91 58 (0.33)
Liquid-S4-Ke KB (ours)
224K
96.52 (0.04)
91 30 (0.19) 90.00 (0.25)
Liquid-S4-PP Souid-S4-PB- (ours)
224K
96.78 (0.05)
p=2
p=2

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This research was supported in part by the A12050 program at Schmidt Futures (Grant G-22-63172) and the United States Air Force Artificial Intelligence Accelerator under Cooperative Agreement Number 1A8750-11-2-118
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed im- plied, of the United States Air Force or the U.S. Govermment. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein. We are very grateful.
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Can sgd learn recurrett neural networks with provable general- ization? In Advances in Neural Information Processing Systems, pp. 10331-10341, 2019.
José M Amigó, Roberto Monetti, Thomas Aschenbrenner, and Wolfram Bunk. Transcripts: An algebraic approach to coupled time series. Chaos: An Interdisciplinary Journal of Nonlinear Science, 22(1):013105, 2012.
Shaojie Bai. J Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling, In Inter- national Conference on Learning Representations, 2018.
Francois W Belletti, Evan R Sparks, Michael J Franklin, Alexandre M Bayen, and Joseph E Gon- zalez. Scalable linear causal inference for irregularly sampled time series with long range depen- dencies. arXiv preprint arXiv: 1603.0330033333 2016.
Beltagy, Mathew Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:200..05550. 2020.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Stephen G Brush. History of the lenz-ising model. Reviews of modern physics, 39(4):883, 1967.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. Advances in neural information processing systems, 30, 2017.
Benjamin Charlier, Jean Feydy, Joan Alexis Glaunès, François- David Collin, and Ghislain Durif. Kemel operations the gpu, with autodiff, without memory overflows. Journal of Machine Learning Research, 22(74):1-6, 2021.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
Dexiong Chen, Laurent Jacob, and Julien Mairal. Recurrent kernel networks. Advances Neural Information Processing Systems, pp. 13431-13442, 2019.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differ- ential equations. In Advances in neural information processing systems, pp. 6571-6583, 2018.
Lei Cheng. Ruslan Khalitov, Tong Yu, and Zhirong Yang. Classification of long sequential data using circular dilated convolutional neural networks. arXiv preprint arXiv:2201.02144 /43, 2022.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreca Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2023
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau, Gru-ode-bayer Continuous
modeling of sporadically-observed time series. In Advances Neural Information Processing Systems, pp. 7377-7388, 2019.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In Advances in Neural Information Processing Systems, pp. 3134 3144, 2019.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, pp. 7509-7520, 2019,
Benjamin N. Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W. Mahoney. Lipschitz recurrent neural networks. In International Conference on Learning Repre- sentations, 2021.
Grant Foster. Wavelets for period analysis of unevenly sampled time series. The Astronomical Journal, 1 112:1709, 1996.
Karl J Friston, Lee Harrison, and Will Penny. Dynamic causal modelling. Neuroimage, 19(4): 1273-1302, 2003.
Peter K Friz and Nicolas B Victoir. Multidimensional stochastic processes as rough paths: theory and applications, volume 120. Cambridge University Press, 2010.
Ken-ichi Funahashi and Yuichi Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural networks, 6(6):801-806, 1993.
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph Mietus, George Moody, Chung thung-Kang Peng, and Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23):e21--222) 2000.
Alexander Tunnell and Zaid Harchaoui. A statistical investigation of long memory in lan- guage and music. In International Conference on Machine Leurning, pp. 2394-2403 2403, 2019.
Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems, 28(10): 2222-2232, 2016.
Sophie Gruenbacher, Mathias Lechner, Ramin Hasani, Daniela Rus, Thomas A Henzinger, Scott A Smolka, and Radu Grosu, Gotube: Scalable statistical verification of continuous-depth models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, No 6, pp. 6755-6766 6764, 2022.
Sophie Grunbacher, Ramin Hasani, Mathias Lechner, Jacek Cyranka, Scott A Smolka, and Radu Grosu. On the verification of neural odes with stochastic guarantees. Proceedings of the AAAI Conference Artificial Intelligence, volume 35, No 13, pp. 11525-11535, 2021.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Re. Hippo: Recurrent memory with optimal polynomial projections. Advances in newral information processing systems, 33, 2020a.
Albert Gu. Caglar Gulcehre, Thomas Paine. Matt Hoffmman, and Razvan Pascanu. Improving the gat- ing mechanism of recurrent neural networks. In International Conference on Machine Learning, pp. 3800-3809. PMLR, 2020b.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Com- bining recurrent, convolutional, and continuous-time models with linear state space layers. Ad- vances Neural Information Processing Systems, 34, 2021.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference Learning Representations, 2022a.

---

Published as a conference paper at ICLR 2023
Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models. arXiv preprint arXhv:2206.11893, 2022b.
Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré. your hippo: State space models with generalized orthogonal basis projections. arXiv preprint arXiv:2206. 12037, 2022c.
How train
Ankit Gupta. Diagonal state spaces are as effective as structured state spaces. arXiv preprint arXiv:2203. 14343, 2022.
YAN Hanshu, DU Jiawei, TAN Vincent, and FENG Jiashi. On robustness of neural ordinary differ- ential equations. In International Conference on Learning Representations, 2020.
Ramin Hasani, Alexander Amini, Mathias Lechner, Felix Naser, Radu Grosu, and Daniela Rus. Response characterization for auditing cell dynamics long short-term memory networks. In 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2019.
Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu. The natural lottery ticket winner: Reinforcement learning with ordinary neural circuits. In Proceedings of the 2020 International Conference on Machine Learning. JMLR. org, 2020.
Ramin Hasani, Mathias Lechner, Alexander Amini, Lucas Liebenwein, Max Tschaikowski, Gerald Teschl, and Daniela Rus. Closed-form continuous-depthh models. arXiv preprint
arXiv:2106. 13898, 202
Ramin Hasani, Mathias Lechner, Alexander Amini, Danicla Rus, and Radu Grosu. Liquid time- constant networks. Proceedings of the AAAI Conference Artificial Intelligence, 35(9):7657- 7666, May 2021b.
Ramin Hasani, Mathias Lechner, Alexander Amini, Lucas Liebenwein, Aaron Ray, Max Tschaikowski, Gerald Teschl, and Daniela Rus. Closed-form continuous-time neural networks. Nature Machine Intelligence, pp. 1-12, 2022.
Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen netzen [in german] diploma thesis. TU Münich, 1991.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Philipp Holl, Vladlen Koltun, and Nils Thuerey, Learning to control pdes with differentiable physics. arXiv preprini arXiv:2001.07457 2020.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554 2558, 1982.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. In Advances in Neural Information Processing Systems, pp. 9843 9854, 2019.
Li Jing, Caglar Gulcehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin Soljacic, and Yoshua Bengio. Gated orthogonal recurrent units: On learning to forget. Neural computation, 31)) 765-783, 2019.
Anil Kag, Ziming Zhang, and Venkatesh Saligrama. Rnns incrementally evolving on an equilibrium manifold: A panacea for vanishing and exploding gradients? In International Conference Learning Representations, 2019.
RE Kalman. A new approach to linear filtering and prediction problems. J. Basic Eng.. Trans. ASME, D, 82:35 45, 1960.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. In International Conference on Ma- chine Learning, pp. 5156-5165. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2023
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equa- tions for irregular time series. Advances in Neural Information Processing Systems, 33:6696- 6707, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2019.
Daniel R Kowal, David S Matteson, and David Ruppert. Functional autoregression for sparsely sampled data. Journal of Business & Economic Statistics, 37(1):97-109, 2019.
Colin Lea, Rene Vidal, Austin Reiter, and Gregory Hager. Temporal convolutional networks:
A unified approach to action segmentation. In European Conference Computer Vision, pp- 47-54. Springer, 2016.
Mathias Lechner and Ramin Hasani. Mixed-memory rnns for learning long-term dependencies in irregularly sampled time series. OpenReview, 2021.
Mathias Lechner, Ramin Hasani, Manuel Zimmer, Thomas A Henzinger, and Radu Grosu. De- signing worm-inspired neural networks for interpretable robotic control. In 2019 International Conference on Robotics and Automation (ICRA). pp. 87-94. IEEE, 2019.
Mathias Lechner, Ramin Hasani, Alexander Amini, Thomas A Henzinger, Daniela Rus, and Radu Grosu. Neural circuit policies enabling auditable autonomy. Nature Machine Intelligence, 2(10): 642-652, 2020a.
Mathias Lechner, Ramin Hasani, Daniela Rus, and Radu Grosu. Gershgorin loss stabilizes the recur- rent neural network compartment of end-to-end robot learning scheme. In 2020 International Conference on Robotics and Automation (ICRA). IEEE, 2020b.
James Lee-Thorp Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint wrXiv:2105.03822 2021.
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indmnn): Building a longer and deeper rnn. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5457-5466, 2018.
Steven Cheng-Xiaa Xian Li and Benjamin M Marlin. A scalable end-to-end gaussian process adapter for irregularly sampled time series classification. In Advances in neural information processing systems, pp. 1804-1812 2016
Lucas Liebenwein, Ramin Hasani, Alexander Amini, and Daniela Rus. Sparse flows: Pruning continuous-depth models. Advances in Neural Information Processing Systems, 34:22628-2266 2021.
William A Little. The existence of persistent states in the brain. Mathematical biosciences, 19(1-)): 101-120, 1974.
Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettle- moyer. Luna: Linear unified nested attention. Advances in Neural Information Processing Sys- tems, 14:244112243,, 2021.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural odes. Advances in Neural Information Processing Systems, 33:3952-3963, 2020.
Hongyuan Mei and Jason M Eisner. The neural hawkes process: neurally self-modulating mul- tivariate point process. In Advances in Neural Information Processing Systems, pp. 6754-6764, 2017.
James Morrill, Patrick Kidger, Cristopher Salvi, James Foster, and Terry Lyons. Neural cdes for long time series via the log-ode method. arXiv preprint arXiv:2009.0822) 2020.
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equa- tions for long time series. In International Conference on Machine Learning, pp. 7839-7838. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2023
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased Istm: Accelerating recurrent network training for long or event-based sequences. In Advances in neural information processing systems, pp. 3182-38880 2016.
Naoki Nonaka and Jun Seita. In-depth benchmarking of deep neural network architectures for eog diagnosis. In Machine Learning for Healthcare Conference, pp. 414-439. PMLR, 2021.
Ronald Pearson, Gregory Goney, and James Shwaber. Imbalanced clustering for microarray time- series. In Proceedings of the ICML, volume 3, 2003.
Will Penny, Zoubin Ghahramani, and Karl Friston. Bilinear dynamical systems. Philosophical Transactions of the Royal Society B: Biological Sciences, 360(1457):983-999, 2005.
Minh Phan, Yunde Shi, Raimondo Betti, and Richard W Longman. Discrete-time. bilinear repre-
sentation of continuous-time bilinear state-space models, Advances in the Astronautical Sciences, 143:571-589, 2012.
Marco AF Pimentel, Alistair EW Johnson, Peter H Charlton, Drew Birrenkott, Peter Watkinson, Lionel Tarassenko, and David A Clifton. Toward robust estimation of respiratory rate from pulse oximeters. IEEE Transactions on Biomedical Engineering. 64(8):1914-1923, 2016.
Alessio Quaglino, Marco Gallieri, Jonathan Masci, and Jan Koutnkk. Snode: Spectral discretization of neural odes for system identification. In International Conference Leurning Representa- lions, 2020.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gru- ber, Markus Holzleimer, Thomas Adler, David Kreil, Michael K Kopp, et al. Hopfield networks is all you need. In International Conference on Learning Representations, 2020.
David W Romero, Robert-Jan Bruintjes, Jakub Mikolaj Tomezak, Erik J Bekkers, Mark Hoogen- doorn, and Jan van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. In International Conference on Learning Representations, 2021a.
David W Romero, Anna Kuzina, Erik Bekkers, Jakub Mikolaj Tomezak, and Mark Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. In International Conference on Learning Representations, 2021b.
DP Roy and L Yan. Robust landsat-basee crop time series modelling. Remote Sensing of Environ- ment, 238:1 10810, 2020.
Yulia Rubanova, Tian Qi Chen, and David K Duvenaud. Latent ordinary differential equations for arregularly-sampled time series. In Advances in Neural Information Processing Systems, pp- 5321-5331, 2019.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, 323(60888 1986.
Konstantin T. Rusch and Siddhartha Mishra. Coupled oscillatory recurrent neural network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies. In Interna- tional Conference on Learning Representations, 2021a
Konstantin Rusch and Siddhartha Mishra. UnICORNN: A recurrent model for learning very long time dependencies. In International Conference on Machine Learning, pp. 9168-9178. PMLR, 2021b.
Alex Sherstinsky. Fundamentals of recurrent neural network (rnn) and long short-term memory (Istm) network. Physica Nonlinear Phenomena, 404:132306, 2020.
Jimmy TH Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04993. 2022.
Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for quence modeling. In The Eleventh International Conference on Learning Representations, 2023. URL https:/ http:::://pppeeeeeeeee net forum d= 8Hw3Axqks.
14

---

Published as a conference paper at ICLR 2023
Chang Wei Tan, Christoph Bergmeir, François Petitjean, and Geoffrey Webb. Time series extrinsic regression. Data Mining and Knowledge Discovery, 35(3):1032-106)) 2021.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International Conference Machine Learning, pp. 9438-9447, PMLR, 2020a.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations, 2020b.
Trieu Trinh, Andrew Dai, Thang Luong, and Quoe Le. Learning longer-term dependencies in mns with auxiliary losses. In International Conference Machine Learning, pp. 4965 4974. PMLR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Aaron Voelker, Ivana Kajić, and Chris Eliasmith. Legendre memory units: Continuous-time repre- sentation recurrent neural networks. Advances in neural information processing systems, 32, 2019.
Charles Vorbach, Ramin Hasani, Alexander Amini, Mathias Lechner, and Daniela Rus. Causal navi- gation by continuous-time neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Cheng Wang and Mathias Niepert. State-regularized recurrent neural networks. In International Conference on Machine Learning, pp. 6596-6600 2019.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0470 2020.
Tsun-Hsuan Wang, Wei Xiao, Tim Seyde, Ramin Hasani, and Daniela Rus. Interpreting neural policies with disentangled tree representations. arXiv preprint 2rXiv2210.006533 2022.
Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Adas. Full-capacity unitary recurrent neural networks. Advances in neural information processing systems, 29:4880- 4888, 2016
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, No 16, pp. 14138-
14148, 2021.
Lianhao Yin, Tsun-Hsuan Wang, Makram Chahine, Tim Seyde, Mathias Lechner, Ramin Hasani, and Daniela Rus. Cooperative flight control using visual- aat--attteniiiii--nuaaae arXiv preprint arXiv:2212.11084, 2022.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 17:172 17297,
1729777787777 2020.
Zhenhai Zhu and Radu Soricut. H-transformer-I 1d: Fast -ieedinensional hierarchical attention for sequences. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3801-3815, 2021.
15

---

Published as a conference paper at ICLR 2023
A
EXAMPLE LIQUID-S4 KERNEL
Kiiguit Ucorrelations "B" CB
CA BB
"B'......CA"
CA B' CB' (16)
ugu Ug 1ik......
ug U2
Me 2
10
ug u2
3 2 18kk
Here, Mecorelations is vector of length (ct)) (++)) + (*+)), and the kernel Kiquid (+))))(+))++++*))
B PROOF OF PROPOSITION 1
Proposition. The Liquid-S4 kernel for each order P, Kiquid can be computed by the anti-diagonal transformation (flip operation) of the product of the S4 convolution kernel, K
(CB, CAB.
-A'--BB B), and a vector B E RR..
Proof. This can be shown by unrolling the S4 convolution kernel and multiplying its components with ==1 performing anti-diagonal transformation to obtain the corresponding liquid S4 kernel:
K (CB, CAB, CA CAAB. C.'B....,CC. CA
I-1
B)
For (correlations of order 2), S4 kernel should be multiplied by B. The resulting kernel would be:
CB
CAB
CA
CA
B
We obtain the liquid kernel by flipping the above kernel to be convolved with the 2-term correlation terms (p=2):
K Kiquid=2
(CAL-1
(CA L-1
CA"B" CAB" CB
Similarly, we can obtain liquid kernels for higher liquid orders and obtain the statement of the proposition.
C HYPERPARAMETERS
Learning Rate. Liquid-S4 generally requires a smaller learning rate compared to S4 and S4D blocks.
Setting Atmar and Atman We set Atmar for all experiments to 0.2, while the Atmin was set based on the recommendations provided in (Gu al., 2022c) to be proportional to seg length"
Causal Modeling vs. Bidirectional Modeling Liquid-S4 works better when used as causal model, i.e. with no bidirectional configuration.
dytale We observed that Liquid-S4 PB kernel performs best with smaller individual state sizes d,late. For instance, we achieve SOTA results in ListOps, IMDB, and Speech Commands by a state size set to 7, significantly reducing the number of required parameters to solve these tasks.
Choice of Liquid-S4 Kernel In all experiments, choose simplified PB kernel over the KB kernel due to the computational costs and performance. We recommend the use of PB kernel.
Choice of parameter in liquid kernel. In all experiments, start off by setting p the liquidity order to 2. This means that the liquid kernel is going to be computed only for correlation terms observe that higher p values consistently enhance the representation learning capacity of Liquid-S4 modules, as we showed in all experiments. We recommend p = 3 as norm perform experiments with Liquid-S4.
of order 2. In principle,
16

---

Published as a conference paper at ICLR 2023
Table 6: Performance on Long Range Arena Tasks. Numbers for Liquid-S4 kernels indicate test accuracy (standard deviation). The rest of the models' performance results are reported from the cited paper. Liquid-S4 is used with its PB kernel.
Model
ListOps 2048
IMDB
AAN 4000
CIFAR 1024
Pathfinder 1024
Path-X 16384
Avg.
(input length)
S4-LegS (Gu et al., 2022b)
59.60 (0.07)
86.82 (0.13)
90.90 (0.15)
88.65 (0.23)
94.20 (0.25)
96.35
86.09
S4D-LegS
2022b)
60.47 (0.34)
86.18 (0.43)
89.46 (0.14)
88.19 (0.26)
93.06 (1.24)
91,95
84.89
S4D-Inw (Gu al., 2022h)
60.18 (0.35)
87.34 (0.20)
91.09 (0.01)
87.83 (0.37)
93.78 (0.25)
92.80
85.50
Liquid-S4-KB (ours)
62.30 (0.10)
88.80 (0.05)
90.95 (0.10)
89.40 (0.15)
94.60 (0.10)
95.98
87.01
Liquid-54-PP (ours)
62.60 (0.20)
88.90 (0.10)
91.15 (0.09) p=
89.45 (0.33)
94.90 (0.25)
96.36
87.21
p=5
p=6
p=3
p=2
p=2
The kernel computation pipeline uses the PyKeops package (Charlier et al., 2021) for large tensor computations without memory overflow.
All reported results are validation accuracy (similar to Gu et al. (2022a)) performed with 2 to different random seeds, except for the BIDMC dataset, which reports accuracy on the test set.
Table 5: Hyperparameters for obtaining best performing models. BN= Batch normalization, LN
Layer normalization, WD= Weight decay.
Depth Features
State Size
Norm
Pre-norm
Dropout
LR
Batch Size
Epochs
WD
ListOps
128
BN
0.002
12
30
0.03
Text (IMDB)
4
128
7
BN
True
0.1
0.003
50
0,01
Retrieval (AAN)
6
256
False
0.2
0.005
16
20
Image (CIFAR)
6
512
512
EN
False
0.1
0.01
16
200
0.03
Pathfinder
256
0.0
0.03
Path-X
6
320
64
BN
True
0.0
0.001
60
0,05
Speech Commands
6
128
7
BN
True
0.0
0.008
10
50
0.05
BICMD (HR)
256
0.0
0.005
32
BICMD (RR)
6
128
256
LN
True
0.0
0.01
32
500
0.01
BICMD (SpO2)
128
256
LN
True
0.0
32
500
0.01
SCIFAR
6
512
512
LN
False
0.1
50
0.03
Table 7: Performance on Raw Speech Command dataset with the reduced ten classes (SC10) indicate validation accuracy. The accuracy of bascline models is reported from Table 5 of (Gu et al., 2022a). stands for infeasible computation on single GPU or not applicable as stated in Table 10 of (Gu et al., 2022a). The hyperparameters for Liquid-S4 are the same as the ones reported for Speech Commands Full Dataset reported in Table 5.
SC10
Model
16kHz
8kHz
Transformer
Performer
30.77
30.68
ODE-RNN
NRDE
16.49
15.12
ExpRNN
1
10.8
LipschitZRNN
CKConv
71.66
65.96
WaveGAN-D
96.25
x
LSSL (Gu et al., 2021)
x
S4-LegS (Gu et al., 2022a)
98.32
96.30
Liquid-S4 (ours)
98.51
95.9
p=2
p=2
17

---

Published as a conference paper at ICLR 2023
A
Long Range
B
CIFAR Long-Range Arena
91.4
89.8
Higher better
Acborary (h)
91.2
89.00
Higher is better
91.0
90.8
SHaps
Lipast Sepue pxe
Liqas-54 pud
Liqud-S4 pus
Liuié pid
B4-Lag3
Usui $4 pv2
Lisad-34 pad
Liquid 194 pos
Moder
BIDMC (HI))
Respiratory
BIDMC SPO2
0.54
Lower better
0.30
Lower better
Lower is better
1988E
RMME
HRSSE
0.08
0.31
0.20
0.30
11 S44eps
94-
Lipas-S4 pat
L4quet p&3
Liqudd S4 pas
Lquel ps2
Liqud Sie pag
Liqust pod
S4-eps
Lipas-S4 pat
Liques pad
Liqued pass
Model
Moda
Modyy
Figure 2: Performance vs Liquid Order in Liquid-S4. (n=3)
D ON THE DISCRETIZATION OF THE LIQUID KERNEL.
How do we perform the discretization A + Bu(t). The dynamical system presented in Eq. 10, is continuous-time (CT) bilinear state-space model (SSM). Ideally, want that the discretization of a CT bilinear SSM to 1) satisfy the first-order form of the model, and 2) preserve the bilinear model structure. This is challenging and only possible via limited number of methods:
1) The most straightforward approach is to use Forward Euler with first-order emror: + #k+1 (8t). Now by plugging this Eq. 10, we get A the conditions above. For this discretization to stay stable. for a + iw an eigenvalue of the continuous transition matrix A, and = + sôt, an eigenvalue of the discrete model, Re(s) S or A [1 + sôt < 1, thus (1 + odt)² + w 1. This condition implies that selecting small enough St ensures the system's 's stability, but for cases where St is large, the system might go unstable.
SLA, = StB. Such discretization satisfies
One can show that based on the properties of the transition matrices A and B, and the range of selected 56 bilinear transformation of discrete matrices A and B, would be very close that of our Forward Euler discretization. This means that:
81 2
A Forward Euler
Approox billnearl <
<
(17)
2) Adams-Bashforth Method: The second-order Allams-Bashford will apply the transformation Tk+1 Iz + / fkk) f(k 1) + O where f(k) is the right-hand-side of Eq. time t == köt. This method also satisfies the two conditions we required (Phan et al., 2012).
One must denote that computing a bilinear transform (https: / /n wikipedia. org/wiki/ Bil inear r_taaasssssarar for) of continuous-time bilinear SSM while preserving the first-order structure of the model an open problem. Ideally, we can apply this transformation on A+Bu(t). However, it is challenging to preserve the first-order form of the equation while keeping the bilinear (liquid) structure of the model described in Eq. 10.
In our case, we use the bilinear transform form of A, B. and C presented in (Eq. 3) for the discrete weights of the system of (Eq. 11), as this approximation is close to that of the Forward Euler. This implies that the continuous system in (Eq. 10) could be transformed directly Eq. 11 by forward Euler transformation. Furthermore, due to the range of št and properties of A and B, the bilinear transformed matrices presented in Eq. would be close to the direct forward Fuler system.
18

---

Published as a conference paper at ICLR 2023
E ON THE AUTOREGRESSIVE MODE OF PB KERNEL
In autoregressive (AR) mode, with PB (or any other conditioned kernel), we obtain A, B, and C no matter what the conditions are.
More specifically, in the autoregressive mode of PB we can use Eq. 13. In Eq. 13, for computing the black parts can reuse the AR mode of the plain S4 model and only have computed the new (violet) parts. As the violet parts consist of multiplications of input terms (and the corresponding matrices) computing the AR mode feasible. This adds a complexity of O(p) to the inference in the AR mode of PB, but because is much smaller than I. (past sequence length), it can significantly speed up the inference time compared the convolution counterpart.
Moreover, one of the properties of LTCs that was never studied and introduced before this work their ability account for the pairwise correlation of inputs which became apparent once we unrolled the system's is dynamics in this work. We believe that the pairwise correlation of inputs is property that the kernel also possesses. Whether the kernel loses the expressivity and robustness attributes of LTCs, we have to investigate in future work.
F WHY DOES PB KERNEL OUTPERFORM KB KERNEL?
One possible reason why PB outperforms KB could be the fact that limit the correlation terms with the truncation with order p. This limitation arises from how S4 blocks are constructed as stack of many ID blocks which does not computationally allow to exploit the benefit of higher-order correlation terms due to the high computational complexity. This limitation might also reduce the expressivity of the KB kernel, but PB, as they do have dependency on A for the correlation terms.
A potential solution would be Liquid-S5 instance, where could directly a parallel scan introduced the concurrent work S5 (Smith et al., 2023), over the linear LTC system (which is a time-varying SSM). This possible because we could precompute the state transitions at each time step. This way we would not need to truncate the kernel and obtain all correlation terms for free. This is exciting extension Liquid-S4 which we are exploring future work
19


Reviews:
[{"review_id": "Z4CyQ1MW9P", "reviewer": "Reviewer_WFdy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper combines liquid time constant models with the recently published S4 model.  This extends S4 to have input-dependent dynamics.  This can be cast back into the original convolutional framework, and the efficient Cauchy kernel can be re-used for computing the additional terms.  The method achieves good results on the standard benchmarks.  \n", "summary_of_the_review": "The results in this paper speak for themselves.  However, the paper offers limited additional intuition or insight beyond “existing LTC ideas can improve S4”.  I also have reservations that the LTC ideas are carried across correctly as is currently presented.  This, coupled with the below-par presentation mean I am scoring this paper as a reject.  I am willing to upgrade my review if that authors can allay my concerns.  \n\nI also note that this is one of as many as six S4 extensions submitted to ICLR.  I would expect to see these at least commented on (although certainly not directly compared to) in a revised draft.  \n", "strength_and_weaknesses": "\n## Strengths\nThe method appears to perform very strongly on a good range of standard benchmarks.  The method also combines two existing methods nicely.  The (short) derivation of how the truncated liquid kernel can be computed, and its parallels to the S4 kernel are also nice.  Highlighting how (effectively) time-varying kernels can be used in S4 is nice.\n\n## Weaknesses\nMy main criticisms with the paper are that it is well below the threshold in terms of presentation, some questions on the core of the model, and that the contribution feels a little incremental.\n\n(a) There are numerous typos, typographical errors, inconsistencies etc that make the paper feel incredibly rushed.  I have tried to outline some of these below.  \n\nThe paper is also very wordy for what is quite a simple idea.  It feels like the authors have engineered paragraphs to cram dozens of citations in to make the work feel more thoroughly referenced; when really the work is just LTC models + S4.  For instance, listing the baselines and their citations in the paragraph “Baselines” is totally unnecessary (since they are also in Table 1).  The same criticism goes for much of Section 2.  \n\nI think this paper could very easily be reduced to say six pages, and that a shorter manuscript would dramatically improve its impact.  \n\nI also wished the authors separated S4 into its own background section.  Although the core concepts predate S4, the math in eg. (4) clearly is drawn from S4.  In that sense, I think “Background” should actually be called “S4”.  Similarly, the bullet points in “How to compute LS4 Kernel Efficiently” should also be in this background section.  \n\nReally, I feel like your sections should be:  “S4”, “LTCMs”, and then the methods section is just “S4+LTCMs”.  This will dramatically simplify the exposition for the reader, and make it clear what is a novel contribution in **this** paper.  \n\n(b)  Can the authors clarify some things in the relation to LCTMs for me:  You actually use the \"PB\" model variant in all of your experiments, which means that you aren't actually using the discrete-time LTC model in (9).  This should be highlighted in the table, where your method is labelled as \"Liquid-S4\", as opposed to \"Liquid-S4-PB\".  Is there a reason why you don't evaluate \"KB\"?  Secondly, I am unconvinced that the discretisation from (8) to (9) is correct.  The bilinear (or ZOH) transform is non-linear, and so discretising (8) is not as simple as discretising $A$ and $B$ separately.  I think the correct discretisation is $\\overline{A} \\cdot \\overline{(Bu_k)}$.  This scheme would make computing the recurrence in (11) more difficult.  I am willing to entertain that I have mis-evaluated this, but this means that the model for which results are presented is actually quite a drastic deviation from the initially motivated LS4 model.  This is okay, but does mean that much of the prose up until (15) is a touch misleading.  I invite the authors to comment on this.  \n\nI also query if the learned PB model can be iterated in a recurrent mode for autoregressive generation?  If this is not the case, then a major plus-point of the original S4 work has been lost.  \n\n(c)  My final major criticism is that the work feels a little incremental.  If we believe the extra cross-terms in (linear) LTC models are useful, as prior work has demonstrated, then there is no reason to think they won’t improve S4 models.  Adding more flexibility into a model should improve performance.  The only “novel” work therefore is showing that these cross-terms can be cast in the same terms as the S4 update, which is nice, but is also not totally surprising given the nature of the cross-terms.  \n\nThe ablation presented in Figure 1 is useful.  I would like to see the ablation for all the datasets:  did increasing p degrade performance?  Or did performance plateau?  What happens if we consider p=100?  \n\nDo any of the theoretical results from S4/DSS/S4D carry across to LS4?  If not, why not?  Is tying the $B$ matrices in the transition and input important?  \n\nSide note on Figure 1:  why was p=4 not evaluated? \n\nAre you also able to clarify whether the figures in the table are the validation or test scores? \n\n\n\n## Minor Weaknesses / Typographical Comments.\n(i)  Throughout:  Inconsistent use of \\eqref{}, \\ref{}, Equ. \\ref{} etc. \n\n(ii)  Throughout:  The math is weirdly spaced, cf. (1) or the second line on page two.\n\n(iii)  Throughout:  Only proper nouns should be capitalized.  (i.e. “State” -> “state”)\n\n(iv)  Throughout:  $x$ (and other terms in select places) should probably be boldfaced as they are vectors.\n\n(v)  Reference:  “KALMAN” -> “Kalman”\n\n(vi)  Pg 2:  “parameter” -> “parameters”.\n\n(vii)  Pg 3:  The logic in attributing the performance of S4 isn’t quite correct.  Numerical performance is derived from HiPPO.  Memory and runtime performance is derived from the NPLR representation.  \n\n(viii)  Eqn. (8):  The element-wise addition is very non-standard.  I would include multiplication with a row vector of ones to signify this in math, as opposed to in the text.\n\n(ix)  Alg 1:  The subscript “liquid” was in \\mathrm elsewhere.\n\n(x)  Eqn. (11):  The statement of this is far too informal.  It obfuscates the message.  Please write out the term fully in math. \n\n(xi)  Throughout:  “Structural” -> “structured”  (in the context of S4).\n\n(xii)  Throughout:  Footnotes should be avoided at all costs.\n\n(xiii)  Throughout (cf. footnote 2):  \\citep and \\citet are used incorrectly in places.\n\n(xiv)  RAW and FULL do not need to be capitalised. \n\n(xv)  Citations are inconsistent in their use of initial, full first name, middle initial etc.  URLs should also be removed.  \n\n(xvi)  Appendix B:  it is nice to re-state the proposition prior to proving it.  \n\n(xvii)  Figure 1:  text-wrapped figures should be avoided. \n\n(xviii):  You used an extra layer (9), compared to S4d's eight, in the ListOps experiment.  Why?  Is it possible to re-run LS4 with eight layers? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XVi6mhTmRde", "reviewer": "Reviewer_xtka", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper generalizes the recently proposed S4 model with an input dependent time constant used in state transition matrix and conduct thorough experiments to illustrate its benefits.", "summary_of_the_review": "While the paper introduces very interesting ideas and impressive results, it can be made much stronger by also looking at computational costs attached to introducing these ideas.", "strength_and_weaknesses": "Strengths:\n- Input dependent transition mechanism in state space models is a novel, important and natural extention of the original S4 model. I really like that this paper thoroughly explores it.\n- The paper conducts thorough experiments on multiple datasets and get impressive improvements over the original S4 model.\n\nWeaknesses:\nThe main set of weaknesses I see in this paper is related to computational complexity and cost.\n\n- This paper can be much stronger with a simpler diagonal version of Liquid-S4. Some of the accelerators don't support (or are super expensive) the full set of operations required for implementing S4. \n- The paper also doesn't report relative increase in computational cost or wall clock time compared to S4 and other alternatives. I would be interested in knowing that.\n- [Optional] Some of the the baselines reported on this paper on LRA doesn't match what's reported in the original papers e.g. S5 paper. Although, its possible that they were updated recently. It's likely that some of the improvements are orthogonal can improve results of this paper and make it even stronger.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FVXCDSaLS2", "reviewer": "Reviewer_KBed", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an extension on the popular S4 model by allowing the state transition matrix to be input dependent. They show significant improvements on the state of the art on many long range sequence prediction problems. ", "summary_of_the_review": "The paper presents a signficant advance. Minor changes can make the paper clearer. ", "strength_and_weaknesses": "Significant advance on the state of the art. I only have minor comments. \n\n1. Why is the same B used for modifying the state transition matrix and additive input in Equation 8 ? \n2. It is not clear why the best linearization of $A + Bu(t)$ equal to $\\bar{A} + \\bar{B}u(t)$. Ideally, the linearization must be done on $A + Bu(t)$ directly. \n3. It seems that the empirical analysis only focuses on PB mode. Why? Since the paper presents two modes, it is best to present results with both modes in all tables (Liquid-S4-KB and Liquid-S4-PB). \n4. It will be useful to get insights on the relationship between optimal $p$ and the \"complexity\" of $u$. Why does ListOps and IMDB require higher $p$ compared to Pathfinder and path-X datasets? Add Pathfinder and path-X to Figure 1 for completeness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-UaePVDyEp", "reviewer": "Reviewer_qgqj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces an approach for capturing long-term dependencies of complex time series. The paper incorporates some ideas from the classical dynamical system modeling literature into machine learning.", "summary_of_the_review": "Solid piece of work. The approach is a bit different from the commonplace approaches of the core machine learning community, which could make its accessibility a bit questionable. On the other hand, this could also be an opportunity for the community to get exposed to ideas from different domains.", "strength_and_weaknesses": "Strengths:\ni) The proposed method is very comprehensive. It is tailored in fine details including computational feasibility.\nii) The paper reports a large list of experiments, compares against competitive modern machine learning approaches such as transformers, and appears to outperform all of them.\n\nWeaknesses:\ni) The method involves calculation of iFFT somewhere in the pipeline. I wonder how much it could preserve end-to-end differentiability of the eventual loss function if it is used together with encoders, for instance in high-dimensional sequences such as videos. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster