SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Samuel Lanthaler
slanth@caltech.edu
T. Konstantin Rusch
Siddhartha Mishra
Coupled oscillators are being increasingly used as the basis of machine learning (ML) architectures, for instance in sequence modeling, graph representation learning and in physical neural networks that are used in analog ML devices. We introduce an abstract class of neural oscillators that encompasses these architectures and prove that neural oscillators are universal, i.e., they can approximate any continuous and casual operator mapping between time-varying functions, to desired accuracy. This universality result provides theoretical justification for the use of oscillator based ML systems. The proof builds on a fundamental result of independent interest, which shows that a combination of forced harmonic oscillators with a nonlinear read-out suffices to approximate the underlying operators.
Oscillators are ubiquitous in the sciences and engineering [12, 30]. Prototypical examples include pendulums in mechanics, feedback and relaxation oscillators in electronics, business cycles in economics and heart beat and circadian rhythms in biology. Particularly relevant to our context is the fact that the neurons in our brain can be thought of as oscillators on account of the periodic spiking and firing of the action potential [29, 11]. Consequently, functional brain circuits such as cortical columns are being increasingly analyzed in terms of networks of coupled oscillators [29].
Given this wide prevalence of (networks of) oscillators in nature and man-made devices, it is not surprising that oscillators have inspired various machine learning architectures in recent years. Prominent examples include the CoRNN [27] and UnICORNN [28] recurrent neural networks for sequence modeling. CoRNN is based on a network of coupled, forced and damped oscillators, whereas UnICORNN is a multi-layer sequence model that stacks networks of independent undamped oscillators as hidden layers within an RNN. Both these architectures were rigorously shown to mitigate the exploding and vanishing gradient problem [20] that plagues RNNs. Hence, both CoRNN and UnICORNN performed very well on sequence learning tasks with long-term dependencies. Another example of the use of oscillators in machine learning is provided by GraphCON [26], a framework for designing graph neural networks (GNNs) [3], that is based on coupled oscillators. GraphCON was also shown to ameliorate the oversmoothing problem [25] and allow for the deployment of multi-layer deep GNNs. Other examples include Second Order Neural ODEs (SONODEs) [19], which can be interpreted as oscillatory neural ODEs, locally coupled oscillatory recurrent networks (LocoRNN) [17], and Oscillatory Fourier Neural Network (O-FNN) [13].
Another avenue where ML models based on oscillators arise is that of physical neural networks (PNNs) [35] i.e., physical devices that perform machine learning on analog (beyond digital) systems. Such analog systems have been proposed as alternatives or accelerators to the current paradigm of machine learning on conventional electronics, allowing us to significantly reduce the prohibitive energy costs of training state-of-the-art ML models. In [35], the authors propose a variety of physical neural networks which include a mechanical network of multi-mode oscillations on a plate and electronic circuits of oscillators as well as a network of nonlinear oscillators. Coupled with a novel physics aware training (PAT) algorithm, the authors of [35] demonstrated that their
nonlinear oscillatory PNN achieved very good performance on challenging benchmarks such as Fashion-MNIST [36]. Moreover, other oscillatory systems such as coupled lasers and spintronic nano-oscillators have also been proposed as possible PNNs, see [33] as an example of the use of thermally coupled vanadium dioxide oscillators for image recognition and [24, 32] for the use of spin-torque nano-oscillators for speech recognition and for neuromorphic computing, respectively.
What is the rationale behind the successful use of (networks of) oscillators in many different contexts in machine learning? The authors of [27] attribute it to the inherent stability of oscillatory dynamics, as the state (and its gradients) of an oscillatory system remain within reasonable bounds throughout the time-evolution of the system. However, this is at best a partial explanation, as it does not demonstrate why oscillatory dynamics can learn (approximate) mappings between inputs and outputs rather than bias the learned states towards oscillatory functions. As an example, consider the problem of classification of MNIST [18] (or Fashion-MNIST) images. It is completely unclear if the inputs (vectors of pixel values), outputs (class probabilities) and the underlying mapping possess any (periodic) oscillatory structure. Consequently, how can oscillatory RNNs (such as CoRNN and UnICORNN) or a network of oscillatory PNNs learn the underlying mapping?
Our main aim in this paper is to provide an answer to this very question on the ability of neural networks, based on oscillators, to express (to approximate) arbitrary mappings. To this end,
- We introduce an abstract framework of neural oscillators that encompasses both sequence models such as CoRNN and UnICORNN, as well as variants of physical neural networks as the ones proposed in [35]. These neural oscillators are defined in terms of second-order versions of neural ODEs [4], and combine nonlinear dynamics with a linear read-out.
- We prove a Universality theorem for neural oscillators by showing that they can approximate, to any given tolerance, continuous operators between appropriate function spaces.
- Our proof of universality is based on a novel theoretical result of independent interest, termed the fundamental Lemma, which implies that a suitable combination of linear oscillator dynamics with nonlinear read-out suffices for universality.
Such universality results, [1, 5, 14, 21] and references therein, have underpinned the widespread use of traditional neural networks (such as multi-layer perceptrons and convolutional neural networks). Hence, our universality result establishes a firm mathematical foundation for the deployment of neural networks, based on oscillators, in myriad applications. Moreover, our constructive proof provides insight into how networks of oscillators can approximate a large class of mappings.
2 Neural Oscillators
General Form of Neural Oscillators. Given  $u:[0,T]\to \mathbb{R}^p$  as an input signal, for any final time  $T\in \mathbb{R}_+$ , we consider the following system of neural ODEs for the evolution of dynamic hidden variables  $y\in \mathbb{R}^{m}$ , coupled to a linear read-out to yield the output  $z\in \mathbb{R}^q$ ,
$$
\left\{ \begin{array}{l l} \ddot {y} (t) = \sigma \left(W y (t) + V u (t) + b\right), & \\ y (0) = \dot {y} (0) = 0, & \\ z (t) = A y (t) + c. & \end{array} \right. \tag {2.1a}
$$
Equation (2.1) defines an input-/output-mapping  $u(t) \mapsto z(t)$ , with time-dependent output  $z: [0, T] \to \mathbb{R}^q$ . Specification of this system requires a choice of the hidden variable dimension  $m$  and the activation function  $\sigma$ . The resulting mapping  $u \mapsto z$  depends on tunable weight matrices  $W \in \mathbb{R}^{m \times m}$ ,  $V \in \mathbb{R}^{m \times p}$ ,  $A \in \mathbb{R}^{q \times m}$  and bias vectors  $b \in \mathbb{R}^m$ ,  $c \in \mathbb{R}^q$ . For simplicity of the exposition, we consider only activation functions  $\sigma \in C^\infty(\mathbb{R})$ , with  $\sigma(0) = 0$  and  $\sigma'(0) = 1$ , such as tanh or sin, although more general activation functions can be readily considered. This general second-order neural ODE system (2.1) will be referred to as a neural oscillator.
Multi-layer neural oscillators. As a special case of neural oscillators, we consider the following much sparser class of second-order neural ODEs,
$$
\left\{ \begin{array}{l} y ^ {0} (t) := u (t), \\ \ddot {y} ^ {\ell} (t) = \sigma \left(w ^ {\ell} \odot y ^ {\ell} (t) + V ^ {\ell} y ^ {\ell - 1} (t) + b ^ {\ell}\right), \quad (\ell = 1, \dots , L), \\ y ^ {\ell} (0) = \dot {y} ^ {\ell} (0) = 0, \\ z (t) = A y ^ {L} (t) + c. \end{array} \right. \tag {2.2a}
$$
In contrast to the general neural oscillator (2.1), the above multi-layer neural oscillator (2.2) defines a hierarchical structure; The solution  $y^{\ell} \in \mathbb{R}^{m_{\ell}}$  at level  $\ell$  solves a second-order ODE with driving force  $y^{\ell - 1}$ , and the lowest level,  $y^0 = u$ , is the input signal. Here, the layer dimensions  $m_1, \ldots, m_L$  can vary across layers, the weights  $w^{\ell} \in \mathbb{R}^{m_{\ell}}$  are given by vectors, with  $\odot$  componentwise multiplication,  $V^{\ell} \in \mathbb{R}^{m_{\ell} \times m_{\ell - 1}}$  is a weight matrix, and  $b^{\ell} \in \mathbb{R}^{m_{\ell}}$  the bias. Given the result of the final layer,  $y^L$ , the output signal is finally obtained by an affine output layer  $z(t) = Ay^L(t) + c$ . In the multilayer neural oscillator, the matrices  $V^{\ell}$ ,  $A$  and vectors  $w^{\ell}$ ,  $b^{\ell}$  and  $c$  represent the trainable hidden parameters. The system (2.2) is a special case of (2.1), since it can be written in the form (2.1), with  $y := [y^L, y^{L-1}, \ldots, y^1]^T$ ,  $b := [b^L, \ldots, b^1]^T$ , and a (upper-diagonal) block-matrix structure for  $W$ :
$$
W := \left[ \begin{array}{c c c c c} \operatorname {d i a g} \left(w ^ {L}\right) & V ^ {L} & 0 & \dots & 0 \\ 0 & \operatorname {d i a g} \left(w ^ {L - 1}\right) & V ^ {L - 1} & \ddots & \vdots \\ \vdots & \ddots & & \ddots & 0 \\ 0 & \dots & 0 & \operatorname {d i a g} \left(w ^ {2}\right) & V ^ {2} \\ 0 & \dots & 0 & 0 & \operatorname {d i a g} \left(w ^ {1}\right) \end{array} \right], \quad V := \left[ \begin{array}{c} 0 \\ \vdots \\ \vdots \\ 0 \\ V ^ {1} \end{array} \right] \tag {2.3}
$$
Given the block-diagonal structure of the underlying weight matrices, it is clear that the multi-layer neural oscillator (2.2) is a much sparser representation of the general neural operator (2.1). Moreover, one can observe from the structure of the neural ODE (2.2) that within each layer, the individual neurons are causally independent of each other.
Assuming that  $w_{i}^{\ell} \neq 0$ , for all  $1 \leq i \leq m_{\ell}$  and all  $1 \leq \ell \leq L$ , we further highlight that the multi-layer neural oscillator (2.2) is a Hamiltonian system,
$$
\dot {y} ^ {\ell} = \frac {\partial H}{\partial \dot {y} ^ {\ell}}, \quad \ddot {y} ^ {\ell} = - \frac {\partial H}{\partial y ^ {\ell}}, \tag {2.4}
$$
with the layer-wise time-dependent Hamiltonian,
$$
H \left(y ^ {\ell}, \dot {y} ^ {\ell}, t\right) = \frac {1}{2} \| \dot {y} ^ {\ell} \| ^ {2} - \sum_ {i = 1} ^ {m _ {\ell}} \frac {1}{w _ {i} ^ {\ell}} \widehat {\sigma} \left(w _ {i} ^ {\ell} y _ {i} ^ {\ell} + \left(V ^ {\ell} y ^ {\ell - 1}\right) _ {i} + b _ {i} ^ {\ell}\right), \tag {2.5}
$$
with  $\widehat{\sigma}$  being the antiderivative of  $\sigma$ , and  $\| \mathbf{x} \|^{2} = \langle \mathbf{x}, \mathbf{x} \rangle$  denoting the Euclidean norm of the vector  $\mathbf{x} \in \mathbb{R}^{m}$  and  $\langle \cdot, \cdot \rangle$  the corresponding inner product. Hence, any symplectic discretization of the multi-layer neural oscillator (2.2) will result in a fully reversible model, which can first be leveraged in the context of normalizing flows [23], and second leads to a memory-efficient training, as the intermediate states (i.e.,  $y^{\ell}(t_{0}), \dot{y}^{\ell}(t_{0}), y_{\ell}(t_{1}), \dot{y}_{\ell}(t_{1}), \dots, y_{\ell}(t_{N}), \dot{y}_{\ell}(t_{N})$ , for some time discretization  $t_{0}, t_{1}, \dots, t_{N}$  of length  $N$ ) do not need to be stored and can be reconstructed during the backward pass. This potentially leads to a drastic memory saving of  $\mathcal{O}(N)$  during training.
2.1 Examples of Neural Oscillators
(Forced) harmonic oscillator. Let  $p = m = q = 1$  and we set  $W = -\omega^2$ , for some  $\omega \in \mathbb{R}$ ,  $V = 1, b = 0$  and the activation function to be identity  $\sigma(x) = x$ . In this case, the neural ODE (2.1) reduces to the ODE modeling the dynamics of a forced simple harmonic oscillator [12] of the form,
$$
\ddot {y} = - \omega^ {2} y + u, \quad y (0) = \dot {y} (0) = 0. \tag {2.6}
$$
Here,  $y$  is the displacement of the oscillator,  $\omega$  the frequency of oscillation and  $u$  is a forcing term that forces the motion of the oscillator. Note that (2.6) is also a particular example of the multi-layer oscillator (2.2) with  $L = 1$ .
This simple example provides justification for our terminology of neural oscillators, as in general, the hidden state  $y$  can be thought of as the vector of displacements of  $m$ -coupled oscillators, which are coupled together through the weight matrix  $W$  and are forced through a forcing term  $u$ , whose effect is modulated via  $V$  and a bias term  $b$ . The nonlinear activation function mediates possible nonlinear feedback to the system on account of large displacements.
CoRNN. The Coupled oscillatory RNN (CoRNN) architecture [27] is given by the neural ODE:
$$
\ddot {j} = \sigma \left(W y + \mathcal {W} \dot {y} + V u + b\right) - \gamma y - \epsilon \dot {y}.
$$
We can recover the neural oscillator (2.1) as a special case of CoRNN by setting  $\mathcal{W} = 0$ ,  $\gamma = \epsilon = 0$ ; thus, a universality theorem for neural oscillators immediately implies a corresponding universality result for the CoRNN architecture.
UnICORNN. The Undamped Independent Controlled Oscillatory RNN (UnICORNN) architecture of [28, eqn. 1] recovers the multi-layer neural oscillator (2.2) in the case where the fundamental frequencies of UnICORNN are automatically determined inside the weight matrix  $W$  in (2.1).
Nonlinear oscillatory PNN of [35]. In [35, SM, Sect. 4.A], the authors propose an analog machine learning device that simulates a network of nonlinear oscillators, for instance realized through coupled pendula. The resulting mathematical model is the so-called simplified Frenkel-Kontorova model [2] given by the ODE system,
$$
M \ddot {\theta} = - K \sin (\theta) - C \sin (\theta) + f,
$$
where  $\theta = (\theta_{1},\dots,\theta_{N})$  is the vector of angles across all coupled pendula,  $M = \mathrm{diag}(\mu^1,\ldots ,\mu^N)$  is a diagonal mass matrix,  $f$  an external forcing,  $K = \mathrm{diag}(k^{1},\ldots ,k^{N})$  the "spring constant" for pendula, given by  $k^i = \mu^i g / \ell$  with  $\ell$  the pendulum length and  $g$  the gravitational acceleration, and where  $C = C^T$  is a symmetric matrix, with
$$
C _ {\ell \ell} = - \sum_ {\ell^ {\prime} \neq \ell} C _ {\ell \ell^ {\prime}}, \quad \text {s o t h a t} \quad [ C \sin (\theta) ] _ {\ell} = \sum_ {\ell^ {\prime} \neq \ell} C _ {\ell \ell^ {\prime}} \left(\sin \left(\theta_ {\ell^ {\prime}}\right) - \sin \left(\theta_ {\ell}\right)\right), \tag {2.7}
$$
which quantifies the coupling between different pendula. We note that this simplified Frenkel-Kontorova system can also model other coupled nonlinear oscillators, such as coupled lasers or spintronic oscillators [35].
We can bring the above system into a more familiar form by introducing the variable  $y$  according to the relationship  $Py = \theta$  for a matrix  $P$ . Substitution of this ansatz then yields  $MPij = -(K + C)\sin(Py) + f$ ; choosing  $P = M^{-1}(K + C)$ , we find
$$
\ddot {y} = - \sin (M ^ {- 1} (K + C) y) + f, \tag {2.8}
$$
which can be written in the form  $ij = \sigma(Wy) + f$  for  $\sigma = -\sin(\cdot)$  and  $W = M^{-1}(K + C)$ . We next make a particular choice of  $C$ , by choosing it in a block-matrix form
$$
C := \left[ \begin{array}{c c c c c} \gamma^ {L} I & C ^ {L} & 0 & \ldots & 0 \\ C ^ {L, T} & \gamma^ {L - 1} I & \ddots & & \vdots \\ 0 & \ddots & & \ddots & 0 \\ \vdots & \ldots & C ^ {3, T} & \gamma^ {2} I & C ^ {2} \\ 0 & \ldots & 0 & C ^ {2, T} & \gamma^ {1} I \end{array} \right],
$$
and with corresponding mass matrix  $M$  in block-matrix form  $M = \mathrm{diag}(\mu^L I, \mu^{L-1}I, \ldots, \mu^1I)$ , then with  $\rho^\ell \coloneqq \gamma^\ell / \mu^\ell$ , we have
$$
M ^ {- 1} C := \left[ \begin{array}{c c c c c} \rho^ {L} I & C ^ {L} / \mu^ {L} & 0 & \ldots & 0 \\ C ^ {L, T} / \mu^ {L - 1} & \rho^ {L - 1} I & \ddots & & \vdots \\ 0 & \ddots & & \ddots & 0 \\ \vdots & & \ddots & \rho^ {2} I & C ^ {2} / \mu^ {2} \\ 0 & \ldots & 0 & C ^ {2, T} / \mu^ {1} & \rho^ {1} I \end{array} \right],
$$
With the intent of ordering the masses of different layers, such that  $\mu^{\ell} / \mu^{\ell -1}\sim \epsilon$  is small, we now introduce an ordering parameter  $\epsilon >0$ . And we consider the following ordering  $\gamma^{\ell},C^{\ell},\mu^{\ell}\sim \epsilon^{\ell}$ . Assuming this ordering, it follows that  $\rho^{\ell}$ ,  $\frac{C^{\ell}}{\mu^{\ell}} = O(1)$ , and  $\frac{C^{\ell}}{\mu^{\ell - 1}} = O(\epsilon)$ . This ordering of the masses introduces an effective one-way coupling, making
$$
M ^ {- 1} C = \left[ \begin{array}{c c c c c} \rho^ {L} I & V ^ {L} & 0 & \ldots & 0 \\ 0 & \rho^ {L - 1} I & V ^ {L - 1} & \ddots & \vdots \\ \vdots & & \ddots & & 0 \\ 0 & \ldots & 0 & \rho^ {2} I & V ^ {2} \\ 0 & \ldots & 0 & 0 & \rho^ {1} I \end{array} \right] + O (\epsilon),
$$
upper triangular, up to small terms of order  $\epsilon$ . We note that the diagonal entries  $\rho^{\ell}$  in  $M^{-1}C$  are determined by the off-diagonal terms through the identity (2.7). The additional degrees of freedom in the (diagonal)  $K$ -matrix in (2.8) can be used to tune the diagonal weights of the resulting weight matrix  $W = M^{-1}(K + C)$ .
Thus, physical systems such as the Frankel-Kontorova system of nonlinear oscillators can be approximated (to leading order) by multi-layer systems of the form
$$
\ddot {y} ^ {\ell} = \sigma \left(w ^ {\ell} \odot y ^ {\ell} + V ^ {\ell} y ^ {\ell - 1}\right) + f ^ {\ell}, \tag {2.9}
$$
with  $f^{\ell}$  an external forcing, representing a tunable linear transformation of the external input to the system. The only formal difference between (2.9) and (2.2) is (i) the absence of a bias term in (2.9) and (ii) the fact that the external forcing appears outside of the nonlinear activation function  $\sigma$  in (2.9). A bias term could readily be introduced by measuring the angles represented by  $y^{\ell}$  in a suitably shifted reference frame; physically, this corresponds to tuning the initial position  $y^{\ell}(0)$  of the pendula, with  $y^{\ell}(0)$  also serving as the reference value. Furthermore, in our proof of universality for (2.2), it makes very little difference whether the external forcing  $f$  is applied inside the activation function, as in (2.2b) resp. (2.1a), or outside as in (2.9); indeed, the first layer in our proof of universality will in fact approximate the linearized dynamics of (2.2b), i.e. a forced harmonic oscillator (2.6). Consequently, a universality result for the multi-layer neural oscillator (2.2) also implies universality of variants of nonlinear oscillator-based physical neural networks, such as those considered in [35].
3 Universality of Neural Oscillators
In this section, we state and sketch the proof for our main result regarding the universality of neural oscillators (2.1) or, more specifically, multi-layer oscillators (2.2). To this end, we start with some mathematical preliminaries to set the stage for the main theorem.
Input signal. We want to approximate operators  $\Phi : u \mapsto \Phi(u)$ , where  $u = u(t)$  is a time-dependent input signal over a time-interval  $t \in [0, T]$ , and  $\Phi(u)(t)$  is a time-dependent output signal. We will assume that the input signal  $t \mapsto u(t)$  is continuous, and that  $u(0) = 0$ . To this end, we introduce the space
$$
C _ {0} ([ 0, T ]; \mathbb {R} ^ {p}) := \left\{u: [ 0, T ] \to \mathbb {R} ^ {p} \mid t \mapsto u (t) \text {i s c o n t i n u o u s a n d} u (0) = 0 \right\}.
$$
We will assume that the underlying operator defines a mapping  $\Phi : C_0([0, T]; \mathbb{R}^p) \to C_0([0, T]; \mathbb{R}^q)$ .
The approximation we discuss in this work are based on oscillatory systems starting from rest. These oscillators are forced by the input signal  $u$ . For such systems the assumption that  $u(0) = 0$  is necessary, because the oscillator starting from rest takes a (arbitrarily small) time-interval to synchronize with the input signal (to "warm up"); If  $u(0) \neq 0$ , then the oscillator cannot accurately approximate the output during this warm-up phase. This intuitive fact is also implicit in our proofs. We will provide a further comment on this issue in Remark 3.2, below.
Operators of interest. We consider the approximation of an operator  $\Phi : C_0([0,T];\mathbb{R}^p) \to C_0([0,T];\mathbb{R}^q)$ , mapping a continuous input signal  $u(t)$  to a continuous output signal  $\Phi(u)(t)$ . We will restrict attention to the uniform approximation of  $\Phi$  over a compact set of input functions  $K \subset C_0([0,T];\mathbb{R}^p)$ . We will assume that  $\Phi$  satisfies the following properties:
-  $\Phi$  is causal: For any  $t \in [0, T]$ , if  $u, v \in C_0([0, T]; \mathbb{R}^p)$  are two input signals, such that  $u|_{[0, t]} \equiv v|_{[0, t]}$ , then  $\Phi(u)(t) = \Phi(v)(t)$ , i.e. the value of  $\Phi(u)(t)$  at time  $t$  does not depend on future values  $\{u(\tau) \mid \tau > t\}$ .
-  $\Phi$  is continuous as an operator
$$
\Phi : \left(C _ {0} ([ 0, T ]; \mathbb {R} ^ {p}), \| \cdot \| _ {L ^ {\infty}}\right)\rightarrow \left(C _ {0} ([ 0, T ]; \mathbb {R} ^ {q}), \| \cdot \| _ {L ^ {\infty}}\right),
$$
with respect to the  $L^{\infty}$ -norm on the input-/output-signals.
Note that the class of Continuous and Causal operators are very general and natural in the contexts of mapping between sequence spaces or time-varying function spaces, see [7, 6] and references therein.
3.2 Universal approximation Theorem
The universality of neural oscillators is summarized in the following theorem:
Theorem 3.1. [Universality of the multi-layer neural oscillator] Let  $\Phi : C_0([0,T];\mathbb{R}^p) \to C_0([0,T];\mathbb{R}^q)$  be a causal and continuous operator. Let  $K \subset C_0([0,T];\mathbb{R}^p)$  be compact. Then for any  $\epsilon > 0$ , there exist hyperparameters  $L, m_1, \ldots, m_L$ , weights  $w^\ell \in \mathbb{R}^{m_\ell}$ ,  $V^\ell \in \mathbb{R}^{m_\ell \times m_{\ell-1}}$ ,  $A \in \mathbb{R}^{q \times m_L}$  and bias vectors  $b^\ell \in \mathbb{R}^{m_\ell}$ ,  $c \in \mathbb{R}^q$ , for  $\ell = 1, \ldots, L$ , such that the output  $z: [0,T] \to \mathbb{R}^q$  of the multi-layer neural oscillator (2.2) satisfies
$$
\sup  _ {t \in [ 0, T ]} | \Phi (u) (t) - z (t) | \leq \epsilon , \quad \forall u \in K.
$$
It is important to observe that the sparse, independent multi-layer neural oscillator (2.2) suffices for universality in the considered class. Thus, there is no need to consider the wider class of neural oscillators (2.1), at least in this respect. We remark in passing that Theorem 3.1 immediately implies another universality result for neural oscillators, showing that they can also be used to approximate arbitrary continuous functions  $F: \mathbb{R}^p \to \mathbb{R}^q$ . This extension is explained in detail in SM A.
Remark 3.2. We note that the theorem can be readily extended to remove the requirement on  $u(0) = 0$  and  $\Phi(u)(0) = 0$ . To this end, let  $\Phi : C([0,T];\mathbb{R}^p) \to C([0,T];\mathbb{R}^q)$  be an operator between spaces of continuous functions,  $u \mapsto \Phi(u)$  on  $[0,T]$ . Fix a  $t_0 > 0$ , and extend any input function  $u : [0,T] \to \mathbb{R}^p$  to a function  $\mathcal{E}(u) \in C_0([-t_0,T];\mathbb{R}^p)$ , by
$$
\mathcal {E} (u) (t) := \left\{ \begin{array}{l l} \frac {(t _ {0} + t)}{t _ {0}} u (0), & t \in [ - t _ {0}, 0), \\ u (t), & t \in [ 0, T ]. \end{array} \right.
$$
Our proof of Theorem 3.1 can readily be used to show that the oscillator system with forcing  $\mathcal{E}(u)$  and initialized at time  $-t_0 < 0$ , can uniformly approximate  $\Phi(u)$  over the entire time interval  $[0, T]$ , without requiring that  $u(0) = 0$ , or  $\Phi(u)(0) = 0$ . In this case, the initial time interval  $[-t_0, 0]$  provides the required "warm-up phase" for the neural oscillator.