SlowGuess/NeurIPS_2023_OCR huggingface dataset:
S.M. Conceptualized the research; coordinated collaborations; lead theory development; conducted distillation experiments.
M.P. Conceptualized the research; coordinated collaborations; lead the experimental (model pre-training, distillation, benchmarks, downstream evaluation) efforts; coordinated writing and conference submission; optimized inference stack.
D.Y.F. Assisted in development of MultiHyena; assisted in pre-training and subsequent benchmarking of distilled models; assisted in writing.
H.K. Developed benchmarking suite and interpreted results; assisted in writing.
R.N.P. Assisted in theory and algorithmic development; performed model-order reduction experiments of H3 models; assisted in writing.
A.T. Conceived and proved Theorem 4.1; assisted in writing.
D.W.R. Assisted in Hankel operator spectral analysis; Assisted in writing.
Q.M. Assisted in theory development.
B.C. Supervised development of benchmarking suite and model deployment.
A.R. Supervised theory development (solving associative recall with MultiHyena, Th. 4.1).
C.Z. Supervised research; secured compute resources.
C.R. Supervised research; reviewed manuscript; secured compute resources.
S.E. Supervised research; reviewed manuscript.
Y.B. Supervised research; reviewed manuscript.
Stefano Massaroli, Michael Poli, and Dan Fu contributed equally to this work. Christopher Ré, Stefano Ermon, and Yoshua Bengio share equal senior authorship.
All authors read and approved the final manuscript.
A Linear Systems
A.1 Extended Notation and System Theory Preliminaries
We first introduce the notation and some mathematical concepts that will be used throughout the paper. By  $\mathbb{Z}$  we denote the set of integers, by  $\mathbb{R}$  the set of reals, and by  $\mathbb{C}$  the set of complex numbers. The variable  $t$  stands for time.  $\ell_p(\mathbb{Z})$  denotes the Banach space of complex-valued sequences  $(x_{t})_{t\in \mathbb{Z}}$  with finite energy, i.e.  $\| x\| _p\coloneqq [\sum_{t\in \mathbb{Z}}|x_t|^p ]^{1 / p} < \infty$  for some  $1\leq p < \infty$ .  $\ell_{\infty}(\mathbb{Z})$  is instead the space of sequences for which  $\| x\|_{\infty}\coloneqq \sup_{t\in \mathbb{Z}}|x_t| < \infty$ . With  $\mathbb{S}$  denoting the unit circle in the complex plane,  $\mathbb{S}\coloneqq \{z\in \mathbb{C}:|z| = 1\}$  we define  $\mathcal{H}_p(\mathbb{S})$  as the space of functions  $X$  from  $\mathbb{C}$  to itself such that  $\| X\| _p\coloneqq [(1 / 2\pi)\int_{-\pi}^{\pi}|X(e^{i\omega})|^p\mathrm{d}\omega ]^{1 / p} < \infty$  and  $\mathcal{H}_{\infty}(\mathbb{S})$  the space for which  $\| X\|_{\infty}\coloneqq \sup_{z\in \mathbb{S}}|X(z)| < \infty$ . Particularly,  $\kappa_2(\mathbb{S})$  is a Hilbert space with inner product  $\langle X,Y\rangle \coloneqq (1 / 2\pi)\int_{-\pi}^{\pi}X(e^{i\omega})Y^{*}(e^{i\omega})\mathrm{d}\omega$  where “*” denotes complex conjugation. Although we acknowledge we are using the same notation for norms in both  $\ell_p(\mathbb{Z})$  and  $\mathcal{H}_p(\mathbb{S})$ , the correct meaning will always be made clear by the context. The  $\mathcal{Z}$ -transform of a sequence  $x = (x_{t})_{t\in \mathbb{Z}}$  is  $X(z) = Z[x](z)\coloneqq \sum_{t\in \mathbb{Z}}x_{t}z^{-t}$ . We embrace the system theory convention of using capital letters to identify transformed sequences. The  $\mathcal{Z}$ -transform is a projection of the sequence onto a basis of powers  $e_t = r^{-t}e^{i\omega t}$ . This basis is not orthogonal unless  $r = 1$ . That is the basis of the discrete-time Fourier transform  $\mathcal{F}$ . Hence,  $\mathcal{F}$  is defined as  $\mathcal{F}[x](e^{i\omega}) = X(e^{i\omega})\coloneqq \sum_{t\in \mathbb{Z}}x_{t}e^{-i\omega t}$ . The discrete-time Fourier transform is an isometric isomorphism between  $\ell_2(\mathbb{Z})$  and  $L_{2}(\mathbb{S})$ . We say that sequences live in the time domain and their  $\mathcal{Z}$  (or  $\mathcal{F}$ ) transforms in the frequency domain.
A linear system is a linear operator transforming an input sequence  $u$  to an output sequence  $y$ . If the sequences have continuous support, i.e.  $t$  ranges over a continuous set (e.g.  $\mathbb{R}$ ), we have a continuous-time system. Conversely, if the sequences have discrete support, i.e.  $t$  ranges over a discrete set (e.g.  $\mathbb{Z}$ ), we have a discrete-time or digital system. In this manuscript we restrict ourselves to discrete-time systems. Systems can be single-input single-output (SISO) if  $u$  and  $y$  are scalar functions or multi-input multi-output if either  $u$  or  $y$  are vector-valued. We limit our discussion to SISO systems. The impulse response of a system is the output sequence  $y$  when the input sequence  $u$  is the Kronecker delta function  $\delta_t$  and is usually denoted by the letter  $h$ . The values  $h_t$  of the impulse response sequence are also known as the Markov parameters of the system. The most common mathematical representation of a linear system is its convolution form:  $y = h*u$ , i.e.  $y_t = \sum_{j\in \mathbb{Z}}h_{t - j}u_j = \sum_{j\in \mathbb{Z}}h_ju_{t - j}$ ,  $t\in \mathbb{Z}$ . In matrix form the input-output relation is given by the Toeplitz operator  $\mathsf{T}_h$  corresponding to the (possibly infinitely long) sequence  $h$ , i.e.  $y = \mathsf{T}_h u$ . Taking the  $\mathcal{Z}$ -transforms, we can write the input-output relation as  $Y(z) = H(z)U(z)$  (this is just the Fourier convolution theorem extended outside the unit circle).  $H(z)$  is called the transfer function of the system. When  $z = e^{i\omega}$ ,  $H(e^{i\omega})$  is just the discrete-time Fourier transform of  $h$  which is called the frequency response of the system. A linear system is causal if  $h_t = 0$  for  $t < 0$ . A system is called stable if the  $\mathsf{T}_h$  is a bounded operator. If  $u, y\in \ell_2(\mathbb{Z})$ , then stability implies  $h\in \ell_{\infty}$ . In the following, we mainly focus on causal stable systems.
A.2 Systems Norms
When quantitatively characterizing linear systems, several norms play a crucial role. These norms provide measures of various characteristics of the systems, which are essential in both analysis and filter design.
The  $\ell_2$  and  $\mathcal{H}_2$  norms As defined above, the  $\ell_2$  norm represents the energy of a signal  $h$ ,
$$
\| h \| _ {2} := \left[ \sum_ {t \in \mathbb {Z}} | h | _ {t} ^ {2} \right] ^ {1 / 2}
$$
while  $\mathcal{H}_2$  is the energy of the (continuous) spectrum of  $h$ ,
$$
\| H \| _ {2} := \left[ \frac {1}{2 \pi} \int_ {- \pi} ^ {\pi} | X (e ^ {i \omega}) | ^ {2} \mathrm {d} \omega \right] ^ {1 / 2}
$$
By Parseval's theorem, the  $\ell_2$  and  $\mathcal{H}_2$  norms are equal,  $\| h\| _2 = \| H\| _2$ . Further these norms are useful to study the approximation of convolutional filter. The following holds:
Lemma A.1 ( $\ell_{\infty}$  output error). Consider the class of  $\ell_2$  measurable inputs such that  $\| u\| _2\leq \zeta$  then for all  $H,\hat{H}\in \mathcal{H}_2$
$$
\| y - \hat {y} \| _ {\infty} \leq \zeta \| H - \hat {H} \| _ {2}
$$
$$
\begin{array}{l} \sup  _ {t > 0} | y _ {t} - \hat {y} _ {t} | = \sup  _ {t > 0} \left| \frac {1}{2 \pi} \int_ {- \pi} ^ {\pi} \left[ Y (e ^ {i \omega}) - \hat {Y} (e ^ {i \omega}) \right] e ^ {i \omega t} d \omega \right| \\ \leq \frac {1}{2 \pi} \int_ {- \pi} ^ {\pi} | Y (e ^ {i \omega}) - \hat {Y} (e ^ {i \omega}) | \mathrm {d} \omega \\ = \frac {1}{2 \pi} \int_ {\pi} ^ {\pi} | H (e ^ {i \omega}) - \hat {H} (e ^ {i \omega}) | U (e ^ {i \omega}) | \mathrm {d} \omega \\ \leq \left[ \frac {1}{2 \pi} \int_ {- \pi} ^ {\pi} | H (e ^ {i \omega}) - \hat {H} (e ^ {i \omega}) | ^ {2} d \omega \right] ^ {1 / 2} \left[ \frac {1}{2 \pi} \int_ {- \pi} ^ {\pi} | U (e ^ {i \omega}) | ^ {2} d \omega \right] ^ {1 / 2} \quad \text {H o l d e r I n e q u a l i t y} \\ \leq \left[ \frac {1}{2 \pi} \int_ {- \pi} ^ {\pi} | H (e ^ {i \omega}) - \hat {H} (e ^ {i \omega}) | ^ {2} \mathrm {d} \omega \right] ^ {1 / 2} \| u \| _ {2} \quad \text {P a r s e v a l T h e o r e m} \\ \leq \zeta \| H - \hat {H} \| _ {\mathcal {H} _ {2}} \\ \end{array}
$$
If  $u$  is the unit impulse function  $u_{t} = \delta_{t}$  then  $\zeta = 1$ . The results also hold for finite sequences of length  $L$  using the discrete Fourier transform.
Lemma A.2 (Impulse response error on finite sequences). Consider filters  $h, \hat{h}$  with finite length  $L$ . Then, the following holds.
$$
\| h - \hat {h} \| _ {\infty} \leq \| H - \hat {H} \| _ {2}
$$
where  $H$  and  $\hat{H}$  denote the discrete Fourier transforms of  $h$  and  $\hat{h}$ , respectively.
$$
\begin{array}{l} \| y - \hat {y} \| _ {\infty} := \sup  _ {t > 0} | y _ {t} - \hat {y} _ {t} | = \sup  _ {t > 0} \left| \frac {1}{2 \pi} \sum_ {n = 0} ^ {L - 1} \left[ Y _ {n} - \hat {Y} _ {n} \right] e ^ {i 2 \pi n t / L} \right| \\ \leq \frac {1}{2 \pi} \sum_ {n = 0} ^ {L - 1} | Y _ {n} - \hat {Y} _ {n} | \\ = \frac {1}{2 \pi} \sum_ {n = 0} ^ {L - 1} \left| H _ {n} - \hat {H} _ {n} \right| \left| U _ {n} \right| \\ \leq \left[ \frac {1}{2 \pi} \sum_ {n = 0} ^ {L - 1} \left(H _ {n} - \hat {H} _ {n}\right) ^ {2} \right] ^ {1 / 2} \left[ \frac {1}{2 \pi} \sum_ {n = 0} ^ {L - 1} U _ {n} ^ {2} \right] ^ {1 / 2} \quad \text {H o l d e r I n e q u a l i t y} \\ \leq \left[ \frac {1}{2 \pi} \sum_ {n = 0} ^ {L - 1} \left(H _ {n} - \hat {H} _ {n}\right) ^ {2} \right] ^ {1 / 2} \| u \| _ {2} \quad \text {P a r s e v a l T h e o r e m} \\ = \| H - \hat {H} \| _ {2} \quad \text {u s i n g} \| u \| _ {2} = 1 \\ \end{array}
$$
A.3 Transfer Function of State-Space Models
The transfer function (3.1) is derived by taking the  $z$ -transform of input and state,  $U(z) = \mathcal{Z}[u](z), X(z) = \mathcal{Z}[x](z)$ . Plugging  $U(z), X(z)$  in the state equation (2.2), it holds
$$
z X (z) = \mathsf {A} X (z) + \mathsf {B} U (z) \Leftrightarrow X (z) = (z \mathsf {I} - \mathsf {A}) ^ {- 1} \mathsf {B} U (z)
$$
Substituting in the output equation yields
$$
Y (\tilde {z}) = \mathsf {C} (z \mathsf {I} - \mathsf {A}) ^ {- 1} \mathsf {B} U (z) + h _ {0} U (z)
$$
The transfer function is then defined as
$$
H (z) = \frac {Y (z)}{U (z)} = \mathrm {C} (z \mathrm {I} - \mathrm {A}) ^ {- 1} \mathrm {B} + h _ {0}. \tag {A.1}
$$
Alternative derivation The transfer function can also be derived by direct  $z$ -transform of the impulse response  $h_t$  of the system. This derivation is useful to highlight the region of convergence of the transfer
$$
\begin{array}{l} H (z) = h _ {0} + \sum_ {t = 1} ^ {\infty} z ^ {- t} \mathsf {C A} ^ {t - 1} \mathsf {B} \quad h _ {0} \text {i s p u l l e d o u t v i a} h _ {0} z ^ {0} = h _ {0} \\ = h _ {0} + \mathsf {C} \left[ \sum_ {t = 1} ^ {\infty} z ^ {- t} \mathsf {A} ^ {t - 1} \right] \mathsf {B} \quad \text {m u l t i p l i c a t i o n d i s t r i b u t e s o v e r s u m}. \tag {A.2} \\ = h _ {0} + z ^ {- 1} \mathrm {C} \left[ \sum_ {t = 1} ^ {\infty} z ^ {- (t - 1)} \mathrm {A} ^ {t - 1} \right] \mathrm {B} \quad \text {m u l t i p l y b y} z / z \\ = h _ {0} + z ^ {- 1} \mathrm {C} \left[ \sum_ {t = 0} ^ {\infty} \left(z ^ {- 1} \mathrm {A}\right) ^ {t} \right] \mathrm {B} \quad \text {c h a n g e o f i n d e x a n d c o l c e l t l i k e t e r m s} \\ \end{array}
$$
We look at the convergence of the series  $\sum_{t=0}^{\infty}\|z^{-1}\mathsf{A}\|_2^t$ . We have
$$
\begin{array}{l} \| z ^ {- 1} \mathsf {A} \| _ {2} \leq \| z ^ {- 1} \| _ {2} \| \mathsf {A} \| _ {2} \\ = \| r ^ {- 1} e ^ {- i \omega} \| _ {2} \| \mathbf {A} \| _ {2} \quad \text {u s i n g} z := r e ^ {i \omega} \in \mathbb {C}, r, \omega \in \mathbb {R} \\ \leq r ^ {- 1} \| \mathsf {A} \| _ {2} = r ^ {- 1} \rho (\mathsf {A}) \\ \end{array}
$$
The series converges to  $1 / (1 - r^{-1}\rho (\mathsf{A}))$  if and only if  $r^{-1}\rho (\mathsf{A}) < 1$  i.e. for  $r > \rho (\mathsf{A})$ . Thus, in the exterior of the disk with radius  $\rho (\mathsf{A})$ ,  $\mathbb{D}_{\rho (\mathsf{A})} := \{z\in \mathbb{C}:|z| > \rho (\mathsf{A})\}$ ,  $\sum_{t = 0}^{\infty}(z^{-1}\mathsf{A})^{t}$  converges to  $(1 - z^{-1}\mathsf{A})^{-1}$  and
$$
z \in \mathbb {D} _ {\rho (\mathsf {A})} \Rightarrow H (z) = h _ {0} + z ^ {- 1} \mathsf {C} (\mathsf {I} - z ^ {- 1} \mathsf {A}) ^ {- 1} \mathsf {B} = h _ {0} + \mathsf {C} (z \mathsf {I} - \mathsf {A}) ^ {- 1} \mathsf {B}
$$
The transfer function  $H(z) = h_0 + \mathsf{C}(z\mathsf{l} - A)^{-1}\mathsf{B}$  of a stable lumped discrete-time system is defined outside the disc in the complex plane that encloses all the eigenvalues of  $\mathsf{A}$ .
Invariance of the transfer function  $H(z)$  as defined in (A.1) is a proper $^{13}$  rational function of  $z$ . In case  $h_0 = 0$ ,  $H(z)$  is strictly proper and the denominator is monic:
$$
H (z) = \frac {b _ {1} z ^ {- 1} + \cdots + b _ {d} z ^ {- d}}{1 + a _ {1} z ^ {- 1} + \cdots + a _ {d} z ^ {- d}} \tag {A.3}
$$
Specifically, the denominator could be derived from A with  $\operatorname{det}(z\mathsf{I} - \mathsf{A})$ , and the numerator is  $\operatorname{det}(z\mathsf{I} - \mathsf{A} + \mathsf{BC}) + \operatorname{det}(z\mathsf{I} - \mathsf{A})$ . We provide a detailed derivation below in Section A.6. While state-space representation involves the analysis and synthesis of model matrices A, B, C, the transfer function is entirely characterized by the coefficients  $a = (a_{n})_{n=1}^{d}$ ,  $b = (b_{n})_{n=1}^{d}$  of numerator and denominator polynomials. Notably, the transfer function is an invariant of the system: if we apply a change of variables to the state, the transfer function remains unchanged.
Lemma A.3. Coefficients  $a, b$  are invariant under any invertible change of variables.
Proof. The proof can be found in [5, pp.95] and follows from the definition of equivalence transformation. Consider the state-space matrices of under change of variables  $\hat{x} = Kx$ ,
$$
\hat {\mathsf {A}} = \mathsf {K A K} ^ {- 1}, \quad \hat {\mathsf {B}} = \mathsf {K B}, \quad \hat {\mathsf {C}} = \mathsf {C K} ^ {- 1}, \quad \hat {h} _ {0} = h _ {0}.
$$
The resulting transfer function  $H(z)$  can then be computed as
$$
\hat {H} (z) = \hat {\mathbf {C}} (z \mathbf {I} - \hat {\mathbf {A}}) ^ {- 1} \hat {\mathbf {B}} + \hat {h} _ {0} = \mathsf {C K} ^ {- 1} [ \mathsf {K} (z \mathbf {I} - \mathsf {A}) \mathsf {K} ^ {- 1} ] ^ {- 1} \mathsf {K B} + h _ {0} = H (z)
$$
A.4 Truncated Transfer Functions
In the case of generic truncated (finite) impulse response filters, such that  $h_t = 0$  for all  $t$  greater than a certain value  $L$  (which we refer to as the length of the filter), the transfer function is simply a polynomial in the complex variable  $z$  of order  $L$ , i.e.
$$
H (z) = \sum_ {t = 0} ^ {\infty} h _ {t} z ^ {- t} = \sum_ {t = 0} ^ {L} h _ {t} z ^ {- t} = h _ {0} + h _ {1} z ^ {- 1} + \dots + h _ {L} z ^ {- L} \tag {A.4}
$$
In case the filter is generated by a finite dimensional (lumped parameters) system, i.e.  $h_t = \mathsf{CA}^{t-1}\mathsf{B}$ ,  $t = 1,\ldots,L$ , then (A.4) can still be represented exactly by a rational function of order  $d$ .
i.e. such that the denominator's order is not less than the numerator's one.
Lemma A.4 (Truncated rational transfer functions). Consider the  $L$ -truncated impulse response  $h_t \in \ell_2(\mathbb{N})$  of a lumped-parameter filter  $(\mathsf{A}, \mathsf{B}, \mathsf{C}, h_0)$ ,
$$
h _ {t} = \left\{ \begin{array}{l l} h _ {0} & t = 0 \\ \mathsf {C A} ^ {t - 1} \mathsf {B} & 1 \leq t \leq L  . \\ 0 & t > L \end{array} \right.
$$
Then its truncated transfer function is
$$
H _ {L} (z) = \mathcal {Z} \{h \} (z) = h _ {0} + \mathsf {C} (\mathsf {I} - z ^ {- L} \mathsf {A} ^ {L}) (z \mathsf {I} - \mathsf {A}) ^ {- 1} \mathsf {B}
$$
Proof. By definition of  $z$ -transform we have
$$
\begin{array}{l} H _ {T} (z) = \sum_ {t = 0} ^ {\infty} h _ {t} z ^ {- t} = h _ {0} + \sum_ {t = 1} ^ {L} z ^ {- t} \mathsf {C A} ^ {t - 1} \mathsf {B} \tag {A.5} \\ = h _ {0} + \mathsf {C} \left[ \sum_ {t = 1} ^ {L} z ^ {- t} \mathsf {A} ^ {t - 1} \right] \mathsf {B} = h _ {0} + z ^ {- 1} \mathsf {C} \left[ \sum_ {t = 0} ^ {L - 1} (z ^ {- 1} \mathsf {A}) ^ {t} \right] \mathsf {B} \\ \end{array}
$$
The sum  $\sum_{t=0}^{L-1}(z^{-1}\mathsf{A})^t$  is a partial Neumann series and can be manipulated as follows.
$$
\begin{array}{l} \sum_ {t = 0} ^ {L - 1} \left(z ^ {- 1} \mathsf {A}\right) ^ {t} (\mathsf {I} - z ^ {- 1} \mathsf {A}) = \sum_ {t = 0} ^ {L - 1} \left(z ^ {- 1} \mathsf {A}\right) ^ {t} - \sum_ {t = 0} ^ {L - 1} \left(z ^ {- 1} \mathsf {A}\right) ^ {t + 1} \\ = \mathbf {I} - (z ^ {- 1} A) ^ {L}. \\ \end{array}
$$
$$
\sum_ {t = 0} ^ {L - 1} (z ^ {- 1} \mathsf {A}) ^ {t} = (\mathsf {I} - z ^ {- L} \mathsf {A} ^ {L}) (\mathsf {I} - z ^ {- 1} \mathsf {A}) ^ {- 1},
$$
which plugged in (A.5) gives  $H_{L}(z) = h_{0} + \mathsf{C}(\mathsf{I} - z^{-L}\mathsf{A}^{L})(z\mathsf{I} - \mathsf{A})^{-1}\mathsf{B}$ , proving the result.
Because of truncation, evaluating the transfer function  $H_{L}(z)$  on the  $L$  roots of unity  $z = e^{i\omega_k}$ ,  $w_{k} = 2\pi k / T$  for  $k = 0,\dots L$  gives the length-  $L$  discrete Fourier transform (DFT) of the filter:
$$
\bar {H} _ {k} := H _ {L} \left(e ^ {i \omega_ {k}}\right) = \sum_ {t = 0} ^ {L - 1} h _ {t} e ^ {- i 2 \pi k / L}, \quad k = 0, \dots , L - 1.
$$
In practice, this means that  $\bar{H} \in \mathbb{C}^L$  is the FFT of  $h$ ,  $\bar{H} = \mathsf{FFT}_L[h]$ . If we can find an efficient and stable algorithm to evaluate  $\bar{H}$  from the system matrices  $(A, B, C, h_0)$ , then the FFT-based convolution of truncated filter with an input sequence  $u \in \mathbb{R}^L$  can be evaluated in  $\tilde{O}(L)$  time.
Reparametrization Assume training a LCSM equipped with SSM filters with input/target sequences to be all of length  $L$  (smaller sequences can be padded with zeros to the maximum length). Thus, for training purposes, we are only interested in evaluating  $\bar{H}$  for the FFT-based convolution.
The truncated transfer function  $H_{L}$  is equal to the original one with a correction term  $|\mathsf{l} - z^{-L}\mathsf{A}^{L}$  on the numerator polynomial. As already noted in S4 [6],  $z^{-L}$  is conveniently equal to one on the roots of unity,  $z^{i\omega_k L} = e^{-i2\pi k} = 1$  for all  $k = 0,\dots ,L - 1$ . Hence, the correction term due to truncation becomes constant:  $H_{k} = \mathsf{C}(\mathsf{l} - \mathsf{A}^{L})(\exp (-i2\pi k / L)\mathsf{l} - \mathsf{A})^{-1}\mathsf{B}$ ; in DFT domain the truncated filter behaves as the infinitely long one with a perturbed C matrix
$$
\bar {\mathsf {C}} = \mathsf {C} - \mathsf {C A} ^ {L}
$$
If -as assumed- the SSM is stable  $\rho (\mathsf{A}) < 1$ ,  $(i)$  the transfer function is defined on the unit circle, term  $\mathsf{CA}^L$  will go to zero exponentially fast as  $L\to \infty$  and  $\bar{\mathsf{C}} = \mathsf{C}$  (as expected). As advised in [6], it is desirable to parametrize directly  $\bar{\mathsf{C}}$ ; the expensive computation of the correction term  $\mathsf{C}(\mathsf{l} - \mathsf{A}^{L})$  is never carried out during training. Instead, the real  $\mathsf{C}$  matrix can be retrieved for recurrent inference by inverting the correction term  $\mathsf{C} = \bar{\mathsf{C}} (\mathsf{l} - \mathsf{A}^{L})^{-1}$ , always invertible for stable systems although possibly ill conditioned by eigenvalues too close to the stability margin (the unit circle).
A.5 From Transfer Function to State-Space
Suppose the coefficients of the numerator and denominator polynomials of a proper transfer function  $H$  is given:
$$
H (z) = \frac {b _ {0} + b _ {1} z ^ {- 1} + \cdots + b _ {d} z ^ {- d}}{1 + a _ {1} z ^ {- 1} + \cdots + a _ {d} z ^ {- d}}. \tag {A.6}
$$
A state-space representation of the form (2.2) can be rapidly realized in two steps:
1. Get delay-free path From (A.6) we first notice that the bias term  $h_0$  is  $h_0 = b_0$ . We thus want to isolate  $b_0$  from the rest of the numerator. This can be obtained via long division (see §A.5.1) and results in
$$
H (z) = \frac {\beta_ {1} z ^ {- 1} + \cdots + \beta_ {N} z ^ {- d}}{1 + a _ {1} z ^ {- 1} + \cdots + a _ {d} z ^ {- d}} + b _ {0}, \quad \beta_ {n} = b _ {n} - b _ {0} a _ {n} \tag {A.7}
$$
2. Get state-space matrices Given the transfer function  $H(z)$  with the isolated pass-through coefficient  $b_{0}$  as in (A.7), we can construct the state-space matrices by companion canonical realization:
$$
\left[ \begin{array}{c c} \mathsf {A} & \mathsf {B} \\ \hline \mathsf {C} & h _ {0} \end{array} \right] = \left[ \begin{array}{c c c c c c} - a _ {1} & - a _ {2} & \dots & - a _ {d - 1} & - a _ {d} & 1 \\ 1 & 0 & \dots & 0 & 0 & 0 \\ 0 & 1 & \dots & 0 & 0 & 0 \\ \vdots & \vdots & \ddots & \vdots & \vdots & \vdots \\ 0 & 0 & \dots & 1 & 0 & 0 \\ \hline \beta_ {1} & \beta_ {2} & \dots & \beta_ {d - 1} & \beta_ {d} & b _ {0} \end{array} \right] \tag {A.8}
$$
Details on the complete  $a$  la [5] derivation can be found in  $\S A.5.2$ . A linear system with finite-dimensional state can be equivalently characterized: by its state-space matrices  $(\mathsf{A},\mathsf{B},\mathsf{C},h_0)$ , by its impulse response function  $h$ , or by the coefficients  $a,b$  (or  $\beta$ ) of the transfer function. A fourth representation is its linear-constant-coefficients difference equation form
$$
y _ {t} = \sum_ {j = 0} ^ {d} b _ {j} u _ {t - j} - \sum_ {n = 1} ^ {d} a _ {j} y _ {t - j},
$$
typically used in signal processing literature in the theory of infinite impulse response filters (see [46]) and known, in the context of system identification of error-in-variables models, as auto-regressive moving-average filters [47, 48].
A.5.1 Isolating the  $h_0$ -term from Transfer Function by Long division
If the rational transfer function  $H(z)$  accounts for the  $h_0$  term, then it is simply proper (order of numerator equals the order of denominator),  $h_0$  is necessarily  $h_0 = b_0$  (the delay-free path). Given the transfer function in this form, we can isolate the  $b_0$  term and the strictly rational term of (A.3) by long division. We start by expanding the fraction as
$$
H (z) = \frac {q (z)}{p (z)} = \frac {b _ {0}}{p (z)} + \frac {b _ {1} z ^ {- 1} + \cdots + b _ {d} z ^ {- d}}{p (z)}.
$$
$$
\frac {b _ {0}}{p (z)} = \frac {b _ {0} z ^ {d}}{z ^ {d} + a _ {1} z ^ {d - 1} + \cdots + a _ {d}}
$$
We then use the long division method to compute  $b_{0} / p(z)$ :
$$
\begin{array}{l} z ^ {d} + a _ {1} z ^ {d - 1} + \dots + a _ {d} \overline {{b _ {0} z ^ {d}}} \\ \begin{array}{c} \frac {b _ {0} z ^ {d} + b _ {0} a _ {1} z ^ {d - 1} + \cdots + b _ {0} a _ {d}}{- b _ {0} a _ {1} z ^ {d - 1} - \cdots - b _ {0} a _ {d}} \quad (\text {r e m i n d e r}) \end{array} \\ \end{array}
$$
to finally get
$$
\begin{array}{l} H (z) = b _ {0} - \frac {b _ {0} a _ {1} z ^ {d - 1} + \cdots + b _ {0} a _ {d}}{z ^ {d} + a _ {1} z ^ {d - 1} + \cdots + a _ {d}} + \frac {b _ {1} z ^ {- 1} + \cdots + b _ {d} z ^ {- d}}{p (z)} \\ = b _ {0} + \frac {\left(b _ {1} - b _ {0} a _ {1}\right) z ^ {- 1} + \cdots + \left(b _ {d} - b _ {0} a _ {d}\right) z ^ {- d}}{1 + a _ {1} z ^ {- 1} + \cdots + a _ {d} z ^ {- d}} \\ \end{array}
$$
Note that the coefficients  $b_{n}$  in (A.3) correspond to  $b_{n} - b_{0}a_{n}$  in (A.6),  $b_{n} \gets b_{n} - b_{0}a_{n}$ . It is indifferent to parameterize the coefficients of the transfer function in either forms. However, if we choose the simply proper representation (A.6), we need to apply the derived correction factor to the numerator coefficients when we separate the  $h_0$  term and strictly proper part of  $H(z)$ .
A.5.2 Construction of the State-Space from the Transfer Function
Chen's derivation The derivation is based on the steps reported for the continuous-time multi-input multi-output case in [5]. First, we define a pseudo-state  $v$  such that
$$
p (z) V (z) = U (z) \quad \Leftrightarrow \quad V (z) = \frac {1}{p (z)} U (z). \tag {A.9}
$$
Then, we define the state  $x_{t} \coloneqq (x_{t}^{1},\ldots ,x_{t}^{d}) \in \mathbb{R}^{d}$  as
$$
x _ {t} = \left(v _ {t - 1}, v _ {t - 2}, \dots , v _ {t - d}\right) \quad \Leftrightarrow \quad \mathcal {Z} \{x \} (z) = X (z) = \left[ \begin{array}{c} z ^ {- 1} \\ \vdots \\ z ^ {- d} \end{array} \right] V (z). \tag {A.10}
$$
From (A.9) we have
$$
V (z) + a _ {1} z ^ {- 1} V (z) + \dots + a _ {d} z ^ {- d} V (z) = U (z) \Leftrightarrow
$$
$$
V (z) = - a _ {1} z ^ {- 1} V (z) - \dots - a _ {d} z ^ {- d} V (z) + U (z) \Leftrightarrow
$$