SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
v _ {t} = - a _ {1} v _ {t - 1} - \dots - a _ {d} v _ {t - d} + u _ {t} \Leftrightarrow \text {t i m e - d e l a y p r o p . o f} \mathcal {Z} \text {- t r a n s f o r m}
$$
$$
x _ {t + 1} ^ {1} = - a _ {1} x _ {t} ^ {1} - \dots - a _ {d} x _ {t} ^ {d} + u _ {t} \Leftrightarrow \text {b y d e f . o f s t a t e (A . 1 0)}.
$$
Thus, we have the overall recurrence
$$
x _ {t + 1} ^ {1} = - a _ {1} x _ {t} ^ {1} - \dots - a _ {d} x _ {t} ^ {d} + u _ {t}
$$
$$
x _ {t + 1} ^ {2} = x _ {t} ^ {1}
$$
$$
x _ {t + 1} ^ {d} = x _ {t} ^ {d - 1}
$$
which can be written in matrix form as
$$
x _ {t + 1} = \left[ \begin{array}{c c c c} - a _ {1} & - a _ {2} & \dots & - a _ {N} \\ 1 & 0 & \dots & 0 \\ 0 & 1 & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & 1 \end{array} \right] x _ {t} + \left[ \begin{array}{c} 1 \\ 0 \\ \vdots \\ 0 \\ 0 \end{array} \right] u _ {t}
$$
The output spectrum is then given by
$$
\begin{array}{l} Y (z) = H (z) U (z) = \frac {q (z)}{p (z)} U (z) + b _ {0} U (z) \\ = q (z) V (z) + b _ {0} U (z) \quad \text {b y} \\ \end{array}
$$
$$
\begin{array}{l} Y (z) = q (z) V (z) + b _ {0} U (z) = \left[ \begin{array}{c c c c} \beta_ {1} & \beta_ {2} & \dots & \beta_ {N} \end{array} \right] \left[ \begin{array}{c} z ^ {- 1} \\ z ^ {- 2} \\ \vdots \\ z ^ {- d} \end{array} \right] V (z) + b _ {0} U (z) \\ = \left[ \begin{array}{c c c c} \beta_ {1} & \beta_ {2} & \dots & \beta_ {d} \end{array} \right] X (z) + b _ {0} U (z) \\ \end{array}
$$
and the output equation in time-domain is given by
$$
y _ {t} = \left[ \begin{array}{c c c c} \beta_ {1} & \beta_ {2} & \dots & \beta_ {d} \end{array} \right] x _ {t} + b _ {0} u _ {t}.
$$
yielding state-space matrices (A.8).
A.6 From State-Space to Transfer Function
We detail an implementation oriented method to compute the coefficients  $(a_{n})_{n = 1}^{d}$ ,  $(b_{n})_{n = 0}^{d}$  of a SSM's transfer function. Recall that
$$
H (z) = \mathrm {C} [ z \mathrm {I} - \mathrm {A} ] ^ {- 1} \mathrm {B} + h _ {0} = \frac {\mathrm {C A d j} (z \mathrm {I} - \mathrm {A}) \mathrm {B} + \det  (z \mathrm {I} - \mathrm {A}) h _ {0}}{\det  (z \mathrm {I} - \mathrm {A})} \tag {A.11}
$$
Hence, the denominator coefficients  $(a_{n})_{n = 1}^{d}$  are simply the coefficients of the characteristic polynomial of matrix A. They can be easily obtained by 1. computing the eigenvalues of A and 2. calculating the coefficients of the polynomial whose roots are such eigenvalues. On the other hand, the numerator apparently involves more complex symbolic manipulation. This can be simplified recalling a classic matrix-determinant identity:
Lemma A.5 ([49]). Let  $\mathsf{M}$ ,  $\mathsf{B}$ , and  $\mathsf{C}$  respectively denote matrices of orders  $d \times d$ ,  $d \times 1$ , and  $1 \times d$ . Then,
$$
\det  (M + B C) = \det  (M) + C A d j (M) B.
$$
Applying Lemma A.5 to (A.11) we obtain
$$
H (z) = \frac {\mathsf {d e t} (z \mathsf {I} - \mathsf {A} + \mathsf {B C}) + \mathsf {d e t} (z \mathsf {I} - \mathsf {A}) (h _ {0} - 1)}{\mathsf {d e t} (z \mathsf {I} - \mathsf {A})}.
$$
Let  $\mathrm{poly}(r)$  denote the coefficients of the polynomials with roots  $r = (r_1, \ldots, r_d)$ . Then  $a = \mathrm{poly}(\mathrm{eig}(\mathsf{A}))$ . Since  $\mathsf{A}$  and  $\mathsf{A} - \mathsf{BC}$  are of equal dimension, their characteristic polynomials have equal order and therefore
$$
b = \operatorname {p o l y} (\operatorname {e i g} (\mathrm {A} - \mathrm {B C})) + \operatorname {p o l y} (\operatorname {e i g} (\mathrm {A})) (h _ {0} - 1)
$$
Listing 1: State-space  $\rightarrow$  transfer function conversion code
def get=tf_from_ss(A,B,C,h0):
    a = poly(eig(A))
    b = poly(eig(A - outer(B,C))) + (h0-1)*a
    return a, b
A.7 State-Space Representation of Truncated Filters.
A truncated filter  $h_0, \ldots, h_L$  - as the ones found in any standard convolutional neural network - can be represented by a  $L$ -dimensional companion canonical SSM. The filter's transfer function  $H(z) = h_0 + h_1 z^{-1} + \dots + h_L z^{-L}$  is polynomial, i.e., a rational function with the denominator's coefficients set to zero. Following the canonical realization process detailed in Section A.5, the truncated filter has state-space form:
$$
x _ {t + 1} = \left[ \begin{array}{c c c c c} 0 & 0 & \dots & 0 & 0 \\ 1 & 0 & \dots & 0 & 0 \\ 0 & 1 & \dots & 0 & 0 \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ 0 & 0 & \dots & 1 & 0 \end{array} \right] x _ {t} + \left[ \begin{array}{c} 1 \\ 0 \\ \vdots \\ 0 \\ 0 \end{array} \right] u _ {t}
$$
$$
y _ {t} = \left[ \begin{array}{c c c c} h _ {1} & h _ {2} & \dots & h _ {L} \end{array} \right] x _ {t} + h _ {0} u _ {t}.
$$
If  $x_0 = \mathbb{O}_L$  and  $u_t = 0$  for negative  $t$ , then at each  $t > 0$  the state is a shifted copy of the input sequence  $x_t = (u_{t-1}, \ldots, u_{t-L}) \in \mathbb{R}^L$ . Nonetheless, the asymptotic complexity of computing one recurrent step is  $\mathcal{O}(L)$  as it requires only a shift operation and a length- $L$  dot product
$$
x _ {t + 1} ^ {1} = u _ {t}
$$
$$
x _ {t + 1} ^ {2: L} = \operatorname {s h i f t} \left(x _ {t}\right) \tag {A.12}
$$
$$
y _ {t} = \left\langle h _ {1: L}, x _ {t} \right\rangle + h _ {0} u _ {t}.
$$
The memory footprint is also  $\mathcal{O}(L)$ . In [1] it is proposed the use of shift-type SSMs to parametrize one of the filters of the H3 block.
A.8 Efficient Computation of State-Space Models
A.8.1 Fast Evaluation of the Transfer Function
Computing  $H(z)$  at any point  $z \in \mathbb{C}$  concerns the evaluation of the  $d$ -order polynomial of numerator and denominator,
$$
H (z) = \frac {q (z)}{p (z)} = \frac {\sum_ {n = 1} ^ {d} b _ {n} z ^ {- n}}{1 + \sum_ {n = 1} ^ {d} a _ {n} z ^ {- n}}
$$
In practice, we are mainly interested in a fast algorithm that allows computing  $H$  on the  $L$  roots of unity to obtain the DFT of the filter. The DFT of the filter can be then readily used to perform a FFT-based convolution with a length-  $L$  input sequence  $u$  or to recover the impulse response function via inverse DFT. We prove the following:
Lemma A.6. Given the coefficients  $a$ ,  $b$  of the transfer function, the frequency and impulse response of the filter can be evaluated in  $\tilde{\mathcal{O}}(L)$  time.
Proof. The result is proven showing that the transfer function can be evaluated in  $\tilde{\mathcal{O}}(L)$  time on the  $L$  roots of unity. The fastest method to evaluate polynomials on  $L$  arbitrary points  $z$  of the complex plane is generally the Horner's scheme. This method is based on a sequence of nested multiplications and computes the polynomial from its vector of coefficients, delivering a time complexity of  $\mathcal{O}(dL)$ . More explicitly, Horner's scheme determines  $p(z)$  as  $p(z) = ((\dots((a_d z^{-1} + a_{d-1}) z^{-1} + a_{d-2}) \dots) z^{-1} + a_2) z^{-1} + a_1) z^{-1} + 1$ . Each step involves a multiplication and an addition, making a total of  $2d$  operations per evaluation point. Thus, for  $L$  points, the total number of operations amounts to  $\mathcal{O}(dL)$ .
Effectively, Horner's approach implements the matrix-vector product of an  $L$ -by- $(d + 1)$  Vandermonde matrix  $\mathsf{V} \in \mathbb{C}^{L \times (d + 1)}$  constructed by  $L$  evaluation points  $(z_0, \ldots, z_{L - 1})$  with the vector of coefficients
$$
\begin{array}{l} a = (1, a _ {1}, \dots , a _ {d}) ^ {\top}: \\ \left[ \begin{array}{c} p (z _ {0}) \\ p (z _ {1}) \\ \vdots \\ p (z _ {L - 1}) \end{array} \right] = \left[ \begin{array}{c c c c c} 1 & z _ {0} ^ {- 1} & z _ {0} ^ {- 2} & \dots & z _ {0} ^ {- d} \\ 1 & z _ {1} ^ {- 1} & z _ {1} ^ {- 2} & \dots & z _ {1} ^ {- d} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & z _ {L - 1} ^ {- 1} & z _ {L - 1} ^ {- 2} & \dots & z _ {L - 1} ^ {- d} \end{array} \right] \left[ \begin{array}{c} 1 \\ a _ {1} \\ \vdots \\ a _ {d} \end{array} \right] = \mathsf {V} a \\ \end{array}
$$
Significantly, if the polynomial is required to be evaluated at the roots of unity, the Vandermonde matrix simplifies corresponds to the  $L \times (d + 1)$  DFT matrix. Further, zero-padding the coefficient vector to length  $L$ , enables the use a single length-  $L$  FFT to compute the matrix-vector product in  $\tilde{\mathcal{O}}(L)$  time. Thus, the numerator and denominator polynomials of the transfer function can be evaluated, on the roots of unity, in  $\tilde{\mathcal{O}}(L)$  time by taking the FFT of the padded numerator / denominator coefficients  $a, b$  and subsequently dividing element-wise the two sequences as  $\mathrm{FFT}_L[b] / \mathrm{FFT}_L[a]$ . The overall time complexity to obtain the impulse response is also  $\tilde{\mathcal{O}}(L)$  since  $h$  can be recovered taking an inverse FFT of the frequency response.
A.8.2 Fast Companion Recurrence
The recurrent step of a generic SSM (2.2) with dense system matrices usually requires  $\mathcal{O}(d^2)$  operations due to the matrix-vector product  $\mathrm{Ax}_t$ . We show how the recurrence of SSMs in companion canonical form, i.e. with system's matrices (A.8), requires only  $\mathcal{O}(d)$  operations.
Lemma A.7. The recurrent step of a state-space model in companion canonical form (A.8) can be evaluated in  $\mathcal{O}(d)$  time and memory.
Proof. The companion state matrix  $\mathsf{A}$  can be broken down into a lower shift matrix  $\mathsf{L}_N$  and a low-rank term. Particularly, with  $e_1$  the first element of the canonical basis of  $\mathbb{R}^N$  and  $\alpha = (a_1, \ldots, a_N)$ , we have
$$
\mathsf {A} = \mathsf {L} _ {N} - e _ {1} \otimes \alpha .
$$
It follows that the recurrent update can be simplified to
$$
x _ {t + 1} = \left(\mathrm {L} _ {N} - e _ {1} \otimes \alpha\right) x _ {t} + \mathrm {B} u _ {t}
$$
$$
y _ {t} = \mathrm {C} x _ {t} + b _ {0} u _ {t}
$$
The peculiarity of this formulation is that we never need to construct the matrices to perform the recurrence. In particular we have:
$$
x _ {t + 1} ^ {1} = u _ {t} - \alpha^ {\top} x _ {t}
$$
$$
x _ {t + 1} ^ {2: N} = \operatorname {s h i f t} (x _ {t})
$$
$$
y _ {t} = \beta^ {\top} x _ {t} + b _ {0} u _ {t}
$$
Thus, each step only requires two inner products ( $d$  multiplications and  $d$  sums each) and one shift operation, totaling  $\mathcal{O}(d)$  operations.
The proof of Lemma A.7 yields the practical implementation of the recurrence:
Listing 2: Python implementation of the companion canonical recurrence
def step(x, u, alpha, beta, b0):  
    y = dot(beta, x) + b0 * u  
    lr = u - dot(alpha, x)  
    x = roll(x)  
    x[0] = lr  
    return x, y
A.8.3 Canonization of State-Space Models
The companion canonical form discussed in Section A.5 is the ideal representation to deploy SSM-based convolutional layers:  $i$  ) it comes with a  $\mathcal{O}(d)$  fast recurrence and  $ii$  ) allows to swiftly switch between time and frequency domains with a direct mapping between state-space matrices and coefficient of the transfer function (which in turn allow  $\tilde{\mathcal{O}} (L)$  fast convolutions).
Aside from [50], which directly parametrizes S4 layers in companion canonical form, all the other parameterizations [12, 6, 32, 17, 33] can be converted (canonized), under mild assumptions.
Lemma A.8 (Canonization of SSMs). Any state-space model (2.2) with proper transfer function can be converted in companion canonical form.
Proof. The result can be proved following the two-step conversion process.
1. Get the coefficients of the transfer function: Given the original state-space matrices  $(\mathsf{A},\mathsf{B},\mathsf{C},h_0)$ , the transfer function is given by  $H(z) = \mathsf{C}(z|\mathsf{-A})^{-1}\mathsf{B} + h_0$ . A proper rational function has the form  $H(z) = q(z) / p(z)$  where the numerator  $q(z)$  has coefficients  $b = (b_n)^d_{n=0}$  and the denominator has coefficients  $a = (a_n)^d_{n=0}$  ( $a_0 = 1$  since  $p$  is monic). As shown in Section A.6, the coefficients of the transfer function can be extracted in closed-form as  $b = \text{poly}(\text{eig}(\mathsf{A} - \mathsf{BC})) + \text{poly}(\text{eig}(\mathsf{A})) (1 - h_0)$  and  $a = \text{poly}(\text{eig}(\mathsf{A}))^{14}$ ;
2. Construct companion matrices Given the coefficients  $a$  and  $b$  a new set of canonical state-space matrices which realize the transfer function can be obtained following the recipe of Section A.5.
The resulting companion SSM is equivalent the the original one since they share the same transfer function.
$^{14}$ eig(A) contains the eigenvalues of A. poly(r) yields the coefficients of the polynomial whose roots are the elements of  $r \in \mathbb{C}^d$ .
B LaughingHyena: Further Details
B.1 Parametrization of Modal Interpolators
Complex-conjugate states Assuming even distilling dimension  $d$ , we pick poles  $\lambda_{n}$  and residues  $\mathbb{R}_n$  in complex-conjugate pairs:
$$
\mathsf {A} = \operatorname {d i a g} \left(\lambda_ {1}, \dots , \lambda_ {d / 2}, \lambda_ {1} ^ {*}, \dots , \lambda_ {d / 2} ^ {*}\right)
$$
$$
\mathsf {C} = \frac {1}{2} \left[ R _ {1}, \dots , R _ {d / 2}, R _ {1} ^ {*}, \dots , R _ {d / 2} ^ {*} \right] \tag {B.1}
$$
which allow partitioning the state-space matrices as
$$
\mathsf {A} = \left[ \begin{array}{l l} \lambda & \\ & \lambda^ {*} \end{array} \right], \quad \mathsf {C} = \frac {1}{2} \left[ \begin{array}{l l} R & R ^ {*} \end{array} \right], \tag {B.2}
$$
$$
\lambda = \operatorname {d i a g} \left(\lambda_ {1}, \dots , \lambda_ {d / 2}\right) \text {a n d} R = \left[ R _ {1}, \dots , R _ {d / 2} \right]. \tag {B.3}
$$
If we also partition the state as  $x = (\bar{x},\tilde{x})$ $\bar{x},\tilde{x}\in \mathbb{C}^{d / 2}$  , the resulting recurrence has the form
$$
\bar {x} _ {t + 1} = \lambda \bar {x} _ {t} + \mathbb {1} _ {d / 2} u _ {t} \tag {B.4}
$$
$$
\tilde {x} _ {t + 1} = \lambda^ {*} \tilde {x} _ {t} + \mathbb {1} _ {d / 2} u _ {t}
$$
$$
\bar {x} _ {t} = \lambda^ {t} \bar {x} _ {0} + \sum_ {j = 0} ^ {t - 1} \lambda^ {t - j - 1} \mathbb {1} _ {d / 2} u _ {t}, \quad \tilde {x} _ {t} = [ \lambda^ {*} ] ^ {t} \tilde {x} _ {0} + \sum_ {j = 0} ^ {t - 1} [ \lambda^ {*} ] ^ {t - j - 1} \mathbb {1} _ {d / 2} u _ {t} \tag {B.5}
$$
Thus, if  $\tilde{x}_0 = \bar{x}_0^*$ , then  $\tilde{x}_t = \bar{x}_t^*$  for all  $t > 0$ . Hence, at inference time we only need to propagate forward half of the state - say  $\bar{x}$  - and then compute the output as
$$
\begin{array}{l} y _ {t} = \mathrm {D} u _ {t} + \frac {1}{2} (R \bar {x} _ {t} + R ^ {*} \bar {x} _ {t} ^ {*}) \\ = \mathrm {D} u _ {t} + \mathfrak {R} \{R \bar {x} _ {t} \} \tag {B.6} \\ = \mathrm {D} u _ {t} + \Re \{R \} \Re \{\bar {x} _ {t} \} - \Im \{R \} \Im \{\bar {x} _ {t} \} \\ \end{array}
$$
This parametrization allows to update only half of the state, reducing the time and memory cost compared to an unconstrained linear system with complex coefficients. However, the implicitly achieved realizness of the output (assuming  $\mathsf{D} = h_0 \in \mathbb{R}$  and  $u_{t} \in \mathbb{R}$ ) comes at a cost of expressivity: such a system is equivalent to an unconstrained complex linear system of dimension  $d / 2$  of which we only keep the real part of the output.
Poles and residues For the modal interpolation, the parametrization is analogous to the one of a diagonal state space model [33]. Poles  $\lambda_{n}$  and residues  $R_{n}$  need both to be complex numbers. In [33] the authors suggest parametrizing real and imaginary components of B and C matrices while representing the eigenvalues  $\lambda_{n}$  in polar form,  $\lambda_{n} = r_{n}e^{i\alpha_{n}}$  with  $r_n$  and  $\alpha_{n}$  being themselves exponential functions of the actual trainable parameters,  $r_n = e^{-e^{\nu_n}}$ ,  $\alpha_{n} = e^{\zeta n}$  leading to
$$
\lambda_ {n} = \exp \left\{- \exp \left\{\nu_ {n} \right\} + i \exp \left\{\zeta_ {n} \right\} \right\}, \quad \nu_ {n}, \zeta_ {n} \in \mathbb {R} \tag {B.7}
$$
This ensures stability of the poles  $|\lambda_n| < 1$  and positive-only phases  $\alpha_n$ . For the purpose of distillation we propose a simplified parametrization as follows:
1. We only parametrize the  $C$  vector. Parametrizing both  $B$  and  $C$  is redundant and increases the computational cost of performing each step of the recurrence. The residues  $R_{n}$  correspond in fact to  $R_{n} = \mathsf{C}_{n}\mathsf{B}_{n}$  of a diagonal state space model. Setting  $\mathsf{B} = \mathbb{1}_d$  saves parameters without harming expressivity. Further if  $B$  is different from  $\mathbb{1}_k$  it needs to be multiplied to  $u_{t}$  at each recurrence step.  $C_n = R_n = \Re [R_n] + i\Im [R_n]$  and  $\Re [R_n],\Im [R_n]$  are the trainable parameters of the residue.
2. For the purpose of distillation we have no benefit in forcing the eigenvalues of the model to be stable, i.e. constrained to lie strictly inside the unit circle. Instead, such constrain may actually harm the expressivity of the approximant. We choose the simpler parametrization  $\lambda_{n} = r_{n}e^{i\alpha_{n}}$ ,  $r_n,\alpha_n\in \mathbb{R}$ .
B.2 Distillation as Rational Interpolation
Distillation as rational interpolation Approximating a filter with an SSM can be thus achieved by fitting a proper rational function to the (truncated) transfer function of the original filter  $H_{L}(z)\coloneqq \sum_{t = 0}^{L}h_{t}z^{-t}$ . That is,
$$
\text {F i n d} a, b \text {s u c h t h a t} h _ {0} + h _ {1} z ^ {- 1} + \dots + h _ {L} z ^ {- L} \approx h _ {0} + Q _ {b} (z) / P _ {a} (z). \tag {B.8}
$$
A modern<sup>15</sup> way to solve this problem by  $\mathcal{H}_2$  error minimization via gradient descent<sup>16</sup>. We can use the Fast Fourier Transform (FFT) to evaluate both the target and distilled transfer functions and solve:
$$
\min  _ {a, b \in \mathbb {R} ^ {d}} \sum_ {k = 0} ^ {L} \left| \mathrm {F F T} _ {L} [ h ] _ {k} - h _ {0} - \mathrm {F F T} _ {L} [ b ] _ {k} / \mathrm {F F T} _ {L} [ a ] _ {k} \right| ^ {2}. \tag {B.9}
$$
To ensure stability of the distilled filters and well-conditioned gradient descent dynamics, the roots of the denominator polynomial must strictly lie inside the unit circle  $(\rho(\mathsf{A}) < 1)$ . This, in turn, requires constraining the coefficients  $a$  into the region  $\{a : \mathrm{poly}(a)$  is stable\} which is by itself an open research problem [52, 53]. Experimentally, we observe that standard coefficient normalization techniques overly restrict the parameters space and lead to poor distillation performances at reasonable order.
In the late 19th century, Henri Padé had already proposed a closed-form solution of the above problem that achieves  $o(z^{-L})$  error for  $z \to \infty$  using  $L = 2d$  samples of the impulse response. His method [31] solves a  $L$ -dimensional linear problem that, however, is known to often become numerically ill-conditioned even with small  $d$  [51]
In the case of finite sequences, the  $\mathcal{H}_2$  norm becomes the standard Euclidean metric evaluated on the  $L + 1$  roots of unity, i.e.  $(\sum_{k = 0}^{L}|H(e^{i2\pi k / (L + 1)})|^{2})^{1 / 2}$ .
C.1 Proof of Lemma 2.1
Generating  $K$  tokens with a long convolution layer (2.1) from a length- $T$  prompt has time complexity  $\mathcal{O}(T\log_2T + TK + K^2)$  and requires  $\mathcal{O}(L)$  memory.
Proof. We compute the time complexity memory of a length- $T$  prompt processing (pre-filling) and subsequent auto-regressive decoding of  $K$  tokens. The auto-regressive generation of long convolution computes the next token as by
$$
t = T, \dots , T + K - 1 \Rightarrow y _ {t} = \sum_ {j = 0} ^ {t - 1} h _ {t - j} y _ {j} \tag {C.1}
$$
The pre-filling step is needed to prime this recurrence by computing the first  $T$  outputs till  $y_{T - 1}$  from the length-  $T$  prompt  $u$ . This is just a convolution between two length-  $T$  signal and requires  $\mathcal{O}(T\log_2T)$  time and linear memory. The auto-regressive decoding of  $K$  tokens requires  $K$  steps (C.1) with the length of the sequences increasing by 1 at each step. Thus we have a total asymptotic complexity of
$$
\sum_ {k = 0} ^ {K - 1} (T + k) = T K + \frac {1}{2} K (K + 1). \tag {C.2}
$$
and requires at worst  $(k = K - 1)$  to store the length  $T + K = L$  generated output sequence, i.e.  $\mathcal{O}(L)$  memory. In the limit we thus have a total time complexity of  $\mathcal{O}(T\log_2T + TK + K^2)$  and  $\mathcal{O}(L)$  memory.
C.2 Proof of Lemma 2.2
Generating  $K$  tokens with a SSM (2.2) from a length- $T$  prompt has time complexity  $\mathcal{O}(T\log_2T + dK)$  and requires  $\mathcal{O}(d)$  memory.
Proof. In autoregressive mode, the cost of generating one token is the cost of evaluating the state recurrence (2.2). Each step then requires  $\mathcal{O}(d)$  time and memory for the class of SSMs considered in this work (see Lemma A.8). Hence, generating  $K$  tokens costs  $\mathcal{O}(dK)$  time and constant  $\mathcal{O}(d)$  memory (we only need to store the current state).
The recurrence is initialized for autoregressive generation with the post-prompt state  $x_{T-1}$  and output  $y_{T-1}$ . The latter can be recovered in linear time and memory  $\mathcal{O}(T)$  by definition  $y_{T-1} = \sum_{j=0}^{T-1} h_{t-j} u_j$  (assuming to have the impulse response  $h$  available) and state  $x_{T-1}$  in  $\mathcal{O}(dT)$  time and  $d$  memory through the recurrence. The overall asymptotic cost is therefore  $\mathcal{O}(dL)$  time and  $\mathcal{O}(d)$  memory.
Note that, for prompts and SSMs of practical sizes we usually have  $d > \log_2 T$ . In such a case the state  $x_{T-1}$  can be computed in  $T \log_2 T$  time rather than  $dT$  by Proposition 3.2.
C.3 Proof of Lemma 2.3
Generating  $K$  tokens with self-attention from a length- $T$  prompt has time complexity  $\mathcal{O}(T^2 + TK + K^2)$  and requires  $\mathcal{O}(L)$  memory.
Proof. The proof is identical to the one of Lemma 2.1, with the only difference of a quadratic asymptotic cost  $\mathcal{O}(T^2)$  to process the prompt obtain the  $kv$  cache.
Self-attention suffers with long contexts: it is significantly more expensive in prefilling than long convolutions and SSMs due to its quadratic cost. Nonetheless, in autoregressive mode, self-attention reaches the same overall asymptotic complexity  $\mathcal{O}(TK + K^2)$  as long convolutions (with the memory overhead of having to cache  $k$  and  $v$ ).
C.4 Proof of Proposition 3.1
If  $\mathsf{A}$  has semi-simple eigenvalues  $\lambda_{n} \in \mathbb{C}$ , then the transfer function of the system can be decomposed as  $\hat{H}(z) = \sum_{n=1}^{d} R_{n} / (z - \lambda_{n})$  where  $R_{n} \in \mathbb{C}$  is the residue associated with the pole  $\lambda_{n}$ .
Proof. If  $\mathsf{A}$  is semi-simple, then it is diagonalized by a basis  $\mathsf{V}$  of eigenvectors; it admits an eigenvalue decomposition  $\mathrm{diag}(\lambda) = \mathsf{VAV}^{-1}$  where  $\lambda = (\lambda_1, \dots, \lambda_d) \in \mathbb{C}^d$  contains the eigenvalues of  $\mathsf{A}$ . Projecting the state onto the basis of eigenvectors,  $s := \mathsf{V}x$ , the state space model is transformed into modal form:
$$
\begin{array}{c c} s _ {t + 1} = \mathsf {V A V} ^ {- 1} s _ {t} + \mathsf {V B} u _ {t} & s _ {t + 1} = \mathsf {d i a g} (\lambda) s _ {t} + \tilde {\mathsf {B}} u _ {t} \\ y _ {t} = \mathsf {C V} ^ {- 1} s _ {t} & y _ {t} = \tilde {\mathsf {C}} s _ {t} \end{array}
$$
where  $\tilde{\mathsf{B}}\coloneqq \mathsf{VB} = (\tilde{b}_n)^d_{n = 1}$  and  $\tilde{\mathbb{C}}\coloneqq \mathsf{CV}^{-1} = (\tilde{c}_n)^d_{n = 1}$ . In modal form, the state equations are decoupled, i.e.
$$
s _ {t + 1} ^ {n} = \lambda_ {n} s _ {t} ^ {n} + \tilde {b} _ {n} u _ {t}
$$
$$
y _ {t} = \sum_ {n = 1} ^ {d} \tilde {c} _ {n} s _ {t} ^ {n}.
$$
Taking the  $z$ -transform of the output equation and each state equation yields
$$
S _ {n} (z) = \mathcal {Z} [ s ^ {n} ] (z) = \frac {\tilde {b} _ {n}}{z - \lambda_ {n}} U (z) n = 1, \ldots , d
$$
$$
Y (z) = \sum_ {n = 1} ^ {d} \tilde {c} _ {n} S _ {n} (z)
$$
Thus, the overall transfer function is
$$
H (z) = \frac {Y (z)}{U (z)} = \sum_ {n = 1} ^ {d} \frac {\tilde {b} _ {n} \tilde {c} _ {n}}{z - \lambda_ {n}}
$$
Letting  $R_{n} = \tilde{b}_{n}\tilde{c}_{n}$ , proves the result.
C.5 Proof of Lemma 3.1
The distilled filter  $\hat{h}$  in modal form (3.2) can be computed in  $\mathcal{O}(dL)$  time from its modal form and in  $\tilde{\mathcal{O}}(L)$  from its rational form.