SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Comparison of sequence models:  $T$  is the sequence length,  $L$  is the number of layers,  $m$  is the hidden dimension,  $C$  and  $K$  are the total number of channels and convolution kernel sizes in TCN. Input and output dimensions are  ${d}_{\text{input }}$  and  ${d}_{\text{output }}$  . Despite TCN’s inference cost is independent of sequence length, it depends on the kernel size  $K$  ,which typically bears a similar scale to  $T$  .
RNN	TCN	Transformer
Number of weights	Lm(m + dinput + doutput)	LCKdinput	3Lmdinput
Single-step inference cost	O(1)	O(1)	O(T2)
Memory pattern	exponential decay	low rank	no restriction
Memory improved version	SSM	CKConv	Linear Transformer
In this paper, we give a constructive proof for the universal approximation property of multi-layer state-space models. It is shown that the nonlinear recurrent activations in classical recurrent neural networks are not necessary when there are nonlinear activations across different hidden layers. This result implies state-space model is as powerful as the classical recurrent neural networks in the approximation sense. Furthermore, we study the memory decay in multi-layer state-space models, which is a notorious issue in classical recurrent neural networks. While empirical evidence suggests
Figure 6: Memory function of a randomly initialized S4. For each model, as the time increases, the memory function can be "capped" by a straight line, which indicate that the memories are decaying exponentially. Compared with Figure 5(d), the results indicate that the smart initialization from S4 provides the memory function with a slower decay.
that state-space models do not experience significant memory decay, they nonetheless exhibit a memory pattern that decays exponentially in the asymptotic sense.
Our research has exciting implications for future work in state-space models. By extending our work to the approximation rate of state-space models, we can obtain better understanding of state-space models' hypothesis space. Such result is important to further optimize the architecture in various real-world applications. We aim to unlock the full potential of state-space models by identifying the ideal network structure (including depth and hidden dimension) for specific tasks and applications.
[17] A. N. Kolmogorov. On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition, volume 28, pages 55-59. American Mathematical Society, Providence, Rhode Island, 1963. ISBN 978-0-8218-1728-5 978-1-4704-3239-3. doi: 10.1090/trans2/028/04.
[18] Lothar Collatz and Werner Krabs. Approximationstheorie: Tschebyscheffsche Approximation mit Anwendungen. Springer-Verlag, April 2013. ISBN 978-3-322-94885-4.
[19] David A Sprecher. ON THE STRUCTURE OF REPRESENTATIONS OF CONTINUOUS FUNCTIONS OF SEVERAL VARIABLES AS FINITE SUMS OF CONTINUOUS FUNCTIONS OF ONE VARIABLE. Proceedings of the American Mathematical Society, 1965.
[20] Jürgen Braun and Michael Griebel. On a Constructive Proof of Kolmogorov's Superposition Theorem. Constructive Approximation, 30(3):653-675, December 2009. ISSN 0176-4276, 1432-0940. doi: 10.1007/s00365-009-9054-2.
[21] Stephen Boyd, L. O. Chua, and C. A. Desoer. Analytical Foundations of Volterra Series. IMA Journal of Mathematical Control and Information, 1(3):243-282, January 1984. ISSN 0265-0754. doi: 10.1093/imamci/1.3.243.
[22] David W. Romero, Anna Kuzina, Erik J. Bekkers, Jakub Mikolaj Tomczak, and Mark Hoogendoorn. CKConv: Continuous Kernel Convolution For Sequential Data. In International Conference on Learning Representations, January 2022.
[23] Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. arXiv preprint arXiv:2303.06349, 2023.
[24] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, 323(6088):533-536, October 1986. ISSN 1476-4687. doi: 10.1038/323533a0.
[25] Eduardo D. Sontag. A learning result for continuous-time recurrent neural networks. Systems & Control Letters, 34(3):151-158, June 1998. ISSN 01676911. doi: 10.1016/S0167-6911(98)00006-1.
[26] Haotian Jiang, Qianxiao Li, Zhong Li, and Shida Wang. A Brief Survey on the Approximation Theory for Sequence Modelling. Journal of Machine Learning, 2(1):1-30, June 2023. ISSN 2790-203X, 2790-2048. doi: 10.4208/jml.221221.
[27] Andrew R. Barron. Approximation and estimation bounds for artificial neural networks. Machine Learning, 14(1):115-133, January 1994. ISSN 1573-0565. doi: 10.1007/BF00993164.
[28] G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4):303-314, December 1989. ISSN 1435-568X. doi: 10.1007/BF02551274.
[29] Ronen Eldan and Ohad Shamir. The Power of Depth for Feedforward Neural Networks. In Conference on Learning Theory, pages 907–940. PMLR, June 2016.
A Memory of multi-layer linear RNNs
In this section, we study the memory function of linear RNN. The depth of linear RNN does not directly expand the memroy pattern as the memory are still decaying exponentially. Nonetheless, although the depth does not increase the approximation capacity, a deeper model is endowed with more structural property.
Take the two-layer Linear RNN as an example:
$$
\begin{array}{l} \hat {y} _ {t} = \int_ {0} ^ {t} C e ^ {W _ {1} (t - s)} U _ {1} \int_ {0} ^ {s} e ^ {W _ {2} (s - r)} U _ {2} x _ {r} d r d s (27) \\ = \int_ {0} ^ {t} C P _ {1} e ^ {\Lambda_ {1} (t - s)} P _ {1} ^ {- 1} U _ {1} \int_ {0} ^ {s} P _ {2} e ^ {\Lambda_ {2} (s - r)} P _ {2} ^ {- 1} U _ {2} x _ {r} d r d s (28) \\ = \int_ {0} ^ {t} C e ^ {\Lambda_ {1} (t - s)} U _ {1} \int_ {0} ^ {s} e ^ {\Lambda_ {2} (s - r)} U _ {2} x _ {r} d r d s (29) \\ = \int_ {0 <   r <   s <   t} d r d s \left(C e ^ {\Lambda_ {1} (t - s)} U _ {1} e ^ {\Lambda_ {2} (s - r)} U _ {2} x _ {r}\right) (30) \\ = \int_ {0 <   r <   t} \int_ {r <   s <   t} d r d s \left(C e ^ {\Lambda_ {1} (t - s)} U _ {1} e ^ {\Lambda_ {2} (s - r)} U _ {2} x _ {r}\right) (31) \\ = \int_ {0} ^ {t} d r \int_ {r} ^ {t} d s \left(C e ^ {\Lambda_ {1} (t - s)} U _ {1} e ^ {\Lambda_ {2} (s - r)} U _ {2} x _ {r}\right) (32) \\ = \sum_ {i = 1} ^ {d _ {h 1}} \sum_ {j = 1} ^ {d _ {h 2}} \int_ {0} ^ {t} d r \int_ {r} ^ {t} d s \left(C e ^ {\lambda_ {1 i} (t - s)} U _ {1, i j} e ^ {\lambda_ {2 j} (s - r)} U _ {2} x _ {r}\right) (33) \\ = \sum_ {i = 1} ^ {d _ {h 1}} \sum_ {j = 1} ^ {d _ {h 2}} \int_ {0} ^ {t} d r \int_ {r} ^ {t} d s \left(e ^ {\lambda_ {1 i} (t - s)} U _ {1, i j} e ^ {\lambda_ {2 j} (s - r)} C U _ {2} x _ {r}\right) (34) \\ = \sum_ {i j} \int_ {0} ^ {t} d r \int_ {r} ^ {t} d s \left(e ^ {\lambda_ {1 i} (t - s) + \lambda_ {2 j} (s - r)} U _ {1, i j} C U _ {2} x _ {r}\right) (35) \\ = \sum_ {i j} \int_ {0} ^ {t} d r \left(\int_ {r} ^ {t} d s e ^ {\left(\lambda_ {2 j} - \lambda_ {1 i}\right) s}\right) \left(e ^ {\lambda_ {1 i} t - \lambda_ {2 j} r} U _ {1, i j} C U _ {2} x _ {r}\right) (36) \\ = \sum_ {i j} \int_ {0} ^ {t} d r \left(\frac {1}{\lambda_ {2 j} - \lambda_ {1 i}} \left(e ^ {\left(\lambda_ {2 j} - \lambda_ {1 i}\right) t} - e ^ {\left(\lambda_ {2 j} - \lambda_ {1 i}\right) r}\right)\right) \left(e ^ {\lambda_ {1 i} t - \lambda_ {2 j} r} U _ {1, i j} C U _ {2} x _ {r}\right) (37) \\ = \sum_ {i j} \int_ {0} ^ {t} d r \left(\frac {1}{\lambda_ {2 j} - \lambda_ {1 i}} \left(e ^ {\lambda_ {2 j} (t - r)} - e ^ {\lambda_ {1 i} (t - r)}\right) U _ {1, i j} C U _ {2} x _ {r}\right) (38) \\ = \int_ {0} ^ {t} d r \sum_ {i j} \left(\frac {1}{\lambda_ {2 j} - \lambda_ {1 i}} \left(e ^ {\lambda_ {2 j} (t - r)} - e ^ {\lambda_ {1 i} (t - r)}\right) U _ {1, i j} C U _ {2}\right) x _ {r} (39) \\ \end{array}
$$
The memory function of two-layer linear RNN is
$$
\hat {\rho} _ {t} = \sum_ {i j} \left(\frac {1}{\lambda_ {2 j} - \lambda_ {1 i}} \left(e ^ {\lambda_ {2 j} (t)} - e ^ {\lambda_ {1 i} (t)}\right) U _ {1, i j} C U _ {2}\right). \tag {40}
$$
The main advantage of two-layer linear RNN, compared with single-layer linear RNN, is that it can learn unbounded combination of exponential decay function as long as  $\lambda_{2j}$  and  $\lambda_{1i}$  are close enough. Notice that for simplicity we do not include the bias term in the evaluation of memory function. The multi-layer linear Recurrent Neural Network (RNN), including a bias term, can be perceived as a linear combination of various depth-specific linear RNNs, devoid of a bias term.
B.1 Universal approximation
Here we include the universal approximation from Barron [27], it is frequently used in the later constructive proof for state-space models.
Theorem B.1 (Cybenko [28], Barron [27]). For any continuous function  $f: \mathbb{R}^d \to \mathbb{R}$  and compact set  $K \subseteq \mathbb{R}^d$ . Assume the activation function  $\sigma: \mathbb{R} \to \mathbb{R}$  is bounded and sigmoidal:
$$
\lim  _ {z \rightarrow \infty} \sigma (z) = 1, \lim  _ {z \rightarrow - \infty} \sigma (z) = - 1. \tag {41}
$$
For any  $\epsilon > 0$ , there exists  $m \in \mathbb{N}$ ,  $A \in \mathbb{R}^{m \times d}$ ,  $b \in \mathbb{R}^k$ ,  $C \in \mathbb{R}^{m \times k}$  such that
$$
\sup  _ {x \in K} \| f (x) - g (x) \| \leq \epsilon , \tag {42}
$$
$$
g (x) = C \sigma (A x + b). \tag {43}
$$
B.2 Proof for Proposition 3.1
Proof. Fix the bounded continuous function  $f$ .
For any  $\epsilon > 0$ , according to the universal approximation theorem in Theorem B.1, there exists  $U_1, U_2, b_1$  such that
$$
\left| f (x) - U _ {2} \sigma \left(U _ {1} x + b _ {1}\right) \right| \leq \epsilon . \tag {44}
$$
Take  $W_{1} = W_{2} = b_{2} = 0$ , the two-layer state-space model can approximate element-wise continuous functions. (Here the element-wise function mean the sequences that for each  $k$ ,  $y_{k}$  only depends on  $x_{k}$ .)
B.3 Proof for Proposition 3.2
Proof. The original statement is given in the discrete index. We prove the result for the continuous index and the discrete case can be derived by discretization.
Without loss of generality, assume the input output sequence are all one-dimensional. Otherwise we shall stacking the  $C, W, U$  for different input-output channels.
Since the input and output are one-dimensional, we know  $c, u$  are vectors and  $W$  is a square matrix. To approximate the target convolution:
$$
y _ {t} = \int_ {0} ^ {t} \rho_ {t - s} x _ {s} d s \tag {45}
$$
by state-space model
$$
\hat {y} _ {t} = \int_ {0} ^ {t} c ^ {\top} e ^ {W (t - s)} u x _ {s} d s. \tag {46}
$$
Since the prediction error can be decomposed into following form
$$
y _ {t} - \hat {y} _ {t} = \int_ {0} ^ {t} \left(\rho_ {t - s} - c ^ {\top} e ^ {W (t - s)} u\right) x _ {s} d s. \tag {47}
$$
Approximating the temporal convolution with state-space model over all bounded inputs  $\mathbf{x}$  can be reduced to function approximating problem:
$$
\max  _ {s \in [ 0, T ]} \left| \rho_ {s} - \sum_ {i = 1} ^ {m} c _ {i} e ^ {- \lambda_ {i} s} \right| <   \epsilon , \quad \lambda_ {i} \geq 0. \tag {48}
$$
In other words, approximating a convolution layer with state-space model is equivalent to approximating a general integrable function by function with exponential form  $\hat{\rho}_s = \sum_{i=1}^m c_i e^{-\lambda_is}$ .
By change of variable, we have
$$
\max  _ {\tau \in \left[ e ^ {- T}, 1 \right]} | f (\tau) - \sum_ {i = 1} ^ {m} c _ {i} \tau^ {\lambda_ {i}} | <   \epsilon . \tag {49}
$$
Here  $f(\tau) = \rho_{-\log (\tau)}$ . Since polynomials are universal approximators on compact intervals, we know there exists  $c_{i},\lambda_{i}$  such that the aforementioned inequality is satisfied. (For example, if  $f$  is smooth, we can take the Taylor expansion of function  $f$ .)
B.4 Proof for Proposition 3.6
Proof. Our proof is based on Equation (18), for any sequence relationship  $H: \{x_k\}_{k=1}^T \to \{y_k\}_{k=1}^T$ , we need to approximate  $\phi$  and  $\Phi$ , and subsequently approximate the representation prescribed by the Kolmogorov-Arnold theorem.
Fix tolerance  $\epsilon$
As is shown in Figure 3, the first element-wise function  $\phi(\cdot)$  can be approximated by two-layer state-space model. This is a result from the direct application of Proposition 3.1. In math terms, there exists  $U_{1}, U_{2}, b_{1}$  such that two-layer state-space model approximate function  $\phi$ .
$$
\left\| \phi (x) - U _ {2} \sigma \left(U _ {1} x + b _ {1}\right) \right\| \leq \epsilon . \tag {50}
$$
We shall denote  $U_{2}\sigma (U_{1}x + b_{1})$  by  $\hat{\phi} (x)$ .
Next, according to Proposition 3.2, we know the (temporal) convolution can be approximated via single-layer state-space model. There exists weights  $C_3, W_3, U_3$  such that
$$
\left\| \sum_ {p = 1} ^ {T} \rho_ {T - p} \hat {\phi} \left(x _ {p} + q a\right) - \sum_ {p = 1} ^ {T} C _ {3} W _ {3} ^ {T - p} U _ {3} \hat {\phi} \left(x _ {p} + q a\right) \right\| \leq \epsilon . \tag {51}
$$
The last layer is again an element-wise function  $\Phi(\cdot)$ , which can be approximate by two-layer state-space model with weights  $U_4, U_5, b_4$ . (Figure 3)
$$
\left\| \Phi \left(\sum_ {p = 1} ^ {T} C _ {3} W _ {3} ^ {T - p} U _ {3} \hat {\phi} \left(x _ {p} + q a\right)\right) - U _ {5} \left(U _ {4} \left(\sum_ {p = 1} ^ {T} C _ {3} W _ {3} ^ {T - p} U _ {3} \hat {\phi} \left(x _ {p} + q a\right)\right) + b _ {4}\right) \right\| \leq \epsilon . \tag {52}
$$
Based on the above result, without loss of generality we assume the outer function  $\Phi$  is Lipschitz continuous with coefficient  $L$ , then the final approximation error is bounded by  $L\| \rho \|_{L^1}\epsilon +L\epsilon +\epsilon$ .
$$
\begin{array}{l} \left\| \Phi \left(\sum_ {p = 1} ^ {T} \rho_ {T - p} \phi \left(x _ {i} + q a\right)\right) - U _ {5} \left(U _ {4} \left(\sum_ {p = 1} ^ {T} C _ {3} W _ {3} ^ {T - p} U _ {3} \hat {\phi} \left(x _ {p} + q a\right)\right) + b _ {4}\right) \right\| (53) \\ \leq \left\| \Phi \left(\sum_ {p = 1} ^ {T} \rho_ {T - p} \phi \left(x _ {i} + q a\right)\right) - \Phi \left(\sum_ {p = 1} ^ {T} \rho_ {T - p} \hat {\phi} \left(x _ {p} + q a\right)\right) \right\| (54) \\ + \left\| \Phi \left(\sum_ {p = 1} ^ {T} \rho_ {T - p} \hat {\phi} \left(x _ {p} + q a\right)\right) - \Phi \left(\sum_ {p = 1} ^ {T} C _ {3} W _ {3} ^ {T - p} U _ {3} \hat {\phi} \left(x _ {p} + q a\right)\right) \right\| (55) \\ + \left\| \Phi \left(\sum_ {p = 1} ^ {T} C _ {3} W _ {3} ^ {T - p} U _ {3} \hat {\phi} \left(x _ {p} + q a\right)\right) - U _ {5} \left(U _ {4} \left(\sum_ {p = 1} ^ {T} C _ {3} W _ {3} ^ {T - p} U _ {3} \hat {\phi} \left(x _ {p} + q a\right)\right) + b _ {4}\right) \right\| (56) \\ \leq L \| \rho \| _ {L ^ {1}} \epsilon + L \epsilon + \epsilon . (57) \\ \end{array}
$$
To summarize, we achieve the approximation of general nonlinear sequence-to-sequence relationship with representation (with Lipschitz continuous  $\Phi$ ) via five-layer state-space model.
Since the Lipschitz continuous function is dense in the set of continuous function, therefore the universal approximation can be generalized to the representation with  $\Phi$  not necessarily Lipschitz continuous.
B.5 Proof for Proposition 3.9
Proof. For simplicity, we will only present the approximation of an  $n$ -th order component of the Volterra Series. The general approximation of nonlinear sequence-to-sequence relationship can be achieved by approximating different order components separately and taking the linear combination.
Consider the target functional
$$
y _ {n} (t) = \int_ {0} ^ {t} \dots \int_ {0} ^ {t} h _ {n} \left(\tau_ {1}, \dots , \tau_ {n}\right) \prod_ {j = 1} ^ {n} x \left(t - \tau_ {j}\right) d \tau_ {j}. \tag {58}
$$
The state-space model's  $n$ -th order term can be represented by
$$
\hat {y} _ {n} (t) = \sum_ {i = 1} ^ {m} \int_ {0} ^ {t} \dots \int_ {0} ^ {t} \left(\prod_ {j = 1} ^ {n} \hat {h} _ {j} ^ {(m)} (\tau_ {j})\right) \prod_ {j = 1} ^ {n} x (t - \tau_ {j}) d \tau_ {j}. \tag {59}
$$
It can be seen the kernel function of state-space model is  $\hat{h}_n(\tau_1,\dots ,\tau_n) = \sum_{i = 1}^m\prod_{j = 1}^n\hat{h}_j^{(m)}(\tau_j)$  while the original  $n$  -th order kernel is multi-variable function  $h_n(\tau_1,\ldots ,\tau_n)$ . For any tolerance  $\epsilon$ , there exists a sufficiently large hidden dimension  $m$  such that the multi-variable function is approximated by the single-variable function's product:
$$
\left. \left| \sum_ {i = 1} ^ {m} \prod_ {j = 1} ^ {n} \hat {h} _ {j} ^ {(m)} \left(\tau_ {j}\right) - h _ {n} \left(\tau_ {1}, \dots , \tau_ {n}\right) \right| \leq \epsilon . \right. \tag {60}
$$
For example, we may consider  $\hat{h}_n$  to be polynomial of  $\tau_{j}$  and take the Taylor expansion of kernel  $h_n$ .
Remark B.2. While the aforementioned proof for approximation is provided through a polynomial method, it's commonly accepted that polynomials may not be the most efficient means to parameterize kernel functions. Consequently, a variety of techniques have been empirically investigated to represent convolutional layers [22].
B.6 Proof for Proposition 3.11
Proof. As  $x_{t}$  converges to  $x^{*}$  exponentially and  $\rho$  is integrable, we know the limit of  $y_{t}$  exists
$$
\lim  _ {t \rightarrow \infty} y _ {t} = \lim  _ {t \rightarrow \infty} \int_ {0} ^ {\infty} \rho_ {s} x _ {t - s} d s = \int_ {0} ^ {\infty} \rho_ {s} \lim  _ {t \rightarrow \infty} x _ {t - s} d s = \int_ {0} ^ {\infty} \rho_ {s} x ^ {*} d s. \tag {61}
$$
Here  $\rho_{s} = C^{\top}e^{Ws}U, s \geq 0$  is the memory function for (continuous) state-space model.
By the dominated convergence theorem, as  $\rho$  is integrable and  $e^{c_0t}|x_{t - s} - x^*|$  is a bounded sequence,
$$
\begin{array}{l} \lim  _ {t \rightarrow \infty} e ^ {c _ {0} t} \| y _ {t} - y ^ {*} \| = \lim  _ {t \rightarrow \infty} e ^ {c _ {0} t} \left\| \int_ {0} ^ {\infty} \rho_ {s} \left(x _ {t - s} - x ^ {*}\right) d s \right\| (62) \\ \leq \left\| \int_ {0} ^ {\infty} \left| \rho_ {s} \right| \lim  _ {t \rightarrow \infty} e ^ {c _ {0} t} \left| x _ {t - s} - x ^ {*} \right| d s \right\| (63) \\ = \left\| \int_ {0} ^ {\infty} \left| \rho_ {s} \right| \cdot 0 d s \right\| = 0. (64) \\ \end{array}
$$
B.7 Proof for Proposition 3.12
Proof. Since  $\lim_{t\to \infty}x_t = x^*$ , by continuity of the activation function  $\sigma (\cdot)$
$$
\lim  _ {t \rightarrow \infty} y _ {t} = \lim  _ {t \rightarrow \infty} \sigma \left(x _ {t}\right) = \sigma \left(\lim  _ {t \rightarrow \infty} x _ {t}\right) = \sigma \left(x ^ {*}\right). \tag {65}
$$
Hereafter we define  $y^{*} = \sigma (x^{*})$
Since  $\sigma$  is Lipschitz continuous, let  $L$  be its Lipschitz constant
$$
\begin{array}{l} \lim  _ {t \rightarrow \infty} e ^ {c _ {0} t} \| y _ {t} - y ^ {*} \| = \lim  _ {t \rightarrow \infty} e ^ {c _ {0} t} \| \sigma \left(x _ {t}\right) - \sigma \left(x ^ {*}\right) \| (66) \\ \leq \lim  _ {t \rightarrow \infty} e ^ {c _ {0} t} L \| x _ {t} - x ^ {*} \| (67) \\ = L \lim  _ {t \rightarrow \infty} e ^ {c _ {0} t} \| x _ {t} - x ^ {*} \| (68) \\ = 0. (69) \\ \end{array}
$$
The Lipschitz continuity is a weak assumption that most of the activations such as ReLU, GeLU, tanh, hardtanh satisfy.
Theoretical results on approximation do not have quantitative results. Therefore it is not easy to directly compare the efficiency of different types of models over a specific task. The good news is that the general memory function servers as a guideline in constructing models. Take the Hyena [12] architecture as an example, as the implicit representation of convolution kernel is utilized in the network, Hyena is not a recurrent model. If we replace the convolution layer by state-space model, we can "translate" a non-recurrent neural network into a recurrent neural network without sacrificing the approximation capacity. The main advantage of recurrent networks, compared with implicit convolution form, lies in the lower inference memory cost.
This study primarily explores the qualitative attributes of the state space model within the context of approximation theory. To better delineate it from other architectures like linear transformers, a more in-depth investigation into the rate of approximation is necessary.
D Comparisons between RNNs, SSMs and S4
In Table 2, we compare different recurrent models in terms of the recurrence, universality, temporal parallel and the memory decay speed.
Our core findings are established for SSM; however, we argue that they are applicable to S4 as well. The primary differences between the vanilla state-space models (SSMs) and S4 involve model parameterisation, weight initialisation, discretisation, normalisation, dropout, and residual connections. The model architectures of SSMs and S4 are almost identical, as both alternately stack linear RNNs and nonlinear activation layers. Therefore, in terms of universal approximation, the approximation capacities of both SSMs and S4 are equivalent.
Linear RNN	Nonlinear RNN	SSMs	S4
Recurrence	Yes	Yes	Yes	Yes
Universality	No	Yes (hardtanh, tanh)	Yes (ours)	Yes (ours)
Temporal Parallel	Yes	No	Yes	Yes
Exponential decay	Yes [14]	Yes [16]	Yes (ours)	Yes, moderated (ours)
Table 2: Comparisons between RNNs, SSMs and S4
E Further discussion on the two proofs for universality of SSMs
We provided comparison between the two proof methods: An analogy between Kolmogorov-theorem and classical fully-connected neural networks can be drawn because of the finite number of layers in both. It is shown in Eldan and Shamir [29] that a simple function expressible by a small 3-layer feedforward neural networks cannot be approximated to a certain accuracy unless the network's width is exponential in the dimension. In contrast, Volterra-Series shares similarities with deep learning, and as a result, the advantages of deep learning over classical fully-connected neural networks carry over
to this approach. The authors are inclined to consider Volterra-Series based construction as relatively superior, and increasing the depth to be a more efficient way to scale up the model.