SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Step 2: (Mollification) We now fix a smooth, non-negative and compactly supported function  $\rho : \mathbb{R} \to \mathbb{R}$ , such that  $\mathrm{supp}(\rho) \subset [0,1]$ ,  $\rho \geq 0$ ,  $\int_{\mathbb{R}} \rho(t) dt = 1$ , and we define a mollifier  $\rho_{\epsilon}(t) := \frac{1}{\epsilon} \rho(t / \epsilon)$ . In the following, we will assume throughout that  $\epsilon \leq T$ . We point out that  $\mathrm{supp}(\rho_{\epsilon}) \subset [0, \epsilon]$ , and hence, the mollification  $F_{\epsilon}(t) = (F * \rho_{\epsilon})(t)$  satisfies, for  $t \geq 0$ :
$$
\begin{array}{l} | F (t) - F _ {\epsilon} (t) | = \left| \int_ {0} ^ {\epsilon} (F (t) - F (t + \tau)) \rho_ {\epsilon} (\tau) d \tau \right| = \left| \int_ {0} ^ {\epsilon} (f (t) - f (t + \tau)) \rho_ {\epsilon} (\tau) d \tau \right| \\ \leq \left\{\sup  _ {\tau \in [ 0, \epsilon ]} | f (t) - f (t + \tau) | \right\} \int_ {0} ^ {\epsilon} \rho_ {\epsilon} (\tau) d \tau \leq \phi (\epsilon). \\ \end{array}
$$
In particular, this shows that
$$
\sup  _ {t \in [ 0, T ]} | F (t) - F _ {\epsilon} (t) | \leq \phi (\epsilon),
$$
can be made arbitrarily small, with an error that depends only on the modulus of continuity  $\phi$ .
Step 3: (Fourier inverse) Let  $\widehat{F}_{\epsilon}(\omega)$  denote the Fourier transform of  $F_{\epsilon}$ . Since  $F_{\epsilon}$  is smooth and compactly supported, it is well-known that we have the identity
$$
F _ {\epsilon} (\tau) = \frac {1}{2 \pi} \int_ {- \infty} ^ {\infty} \widehat {F} _ {\epsilon} (\omega) e ^ {- i \omega \tau} d \omega , \quad \forall t \in \mathbb {R},
$$
where  $\omega \mapsto \widehat{F}_{\epsilon}(\omega)$  decays to zero very quickly (almost exponentially) as  $|\omega| \to \infty$ . In fact, since  $F_{\epsilon} = F * \rho_{\epsilon}$  is a convolution, we have  $\widehat{F}_{\epsilon}(\omega) = \widehat{F}(\omega) \widehat{\rho}_{\epsilon}(\omega)$ , where  $|\widehat{F}(\omega)| \leq 2 \|f\|_{L^{\infty}} T$  is uniformly bounded, and  $\widehat{\rho}_{\epsilon}(\omega)$  decays quickly. In particular, this implies that there exists a  $L = L(\epsilon, T) > 0$  independent of  $f$ , such that
$$
\left| F _ {\epsilon} (\tau) - \frac {1}{2 \pi} \int_ {- L} ^ {L} \widehat {F} (\omega) \widehat {\rho} _ {\epsilon} (\omega) e ^ {- i \omega \tau} d \omega \right| \leq 2 T \| f \| _ {L ^ {\infty}} \int_ {| \omega | > L} | \widehat {\rho} _ {\epsilon} (\omega) | d \omega \leq \| f \| _ {L ^ {\infty}} \epsilon , \quad \forall \tau \in \mathbb {R}. \tag {B.2}
$$
Step 4: (Quadrature) Next, we observe that, since  $F$  and  $\rho_{\epsilon}$  are compactly supported, their Fourier transform  $\omega \mapsto \widehat{F}(\omega)\widehat{\rho}_{\epsilon}(\omega)e^{-i\omega\tau}$  is smooth; in fact, for  $|\tau| \leq T$ , the Lipschitz constant of this mapping can be explicitly estimated by noting that
$$
\begin{array}{l} \frac {\partial}{\partial \omega} \left[ \widehat {F} (\omega) \widehat {\rho_ {\epsilon}} (\omega) e ^ {- i \omega \tau} \right] = \frac {\partial}{\partial \omega} \int_ {\operatorname {s u p p} \left(F _ {\epsilon}\right)} \left(F * \rho_ {\epsilon}\right) (t) e ^ {i \omega (t - \tau)} d t \\ = \int_ {\operatorname {s u p p} \left(F _ {\epsilon}\right)} i (t - \tau) \left(F * \rho_ {\epsilon}\right) (t) e ^ {i \omega (t - \tau)} d t. \\ \end{array}
$$
We next take absolute values, and note that any  $t$  in the support of  $F_{\epsilon}$  obeys the bound  $|t|\leq T + \epsilon \leq 2T$ , while  $|\tau |\leq T$  by assumption; it follows that
$$
\operatorname {L i p} \left(\omega \mapsto \widehat {F} (\omega) \widehat {\rho_ {\epsilon}} (\omega) e ^ {- i \omega \tau}\right) \leq (2 T + T) \| F \| _ {L ^ {\infty}} \| \rho_ {\epsilon} \| _ {L ^ {1}} = 3 T \| F \| _ {L ^ {\infty}}, \quad \forall \tau \in [ 0, T ].
$$
It thus follows from basic results on quadrature that for an equidistant choice of frequencies  $\omega_{1} < \dots < \omega_{N}$ , with spacing  $\Delta \omega = 2L / (N - 1)$ , we have
$$
\left| \frac {1}{2 \pi} \int_ {- L} ^ {L} \widehat {F} (\omega) \widehat {\rho} _ {\epsilon} (\omega) e ^ {- i \omega \tau} d \omega - \frac {\Delta \omega}{2 \pi} \sum_ {j = 1} ^ {N} \widehat {F} (\omega_ {j}) \widehat {\rho} _ {\epsilon} (\omega_ {j}) e ^ {- i \omega_ {j} \tau} \right| \leq \frac {C L ^ {2} 3 T \| F \| _ {L ^ {\infty}}}{N}, \quad \forall \tau \in [ 0, T ],
$$
for an absolute constant  $C > 0$ , independent of  $F, T$  and  $N$ . By choosing  $N$  to be even, we can ensure that  $\omega_{j} \neq 0$  for all  $j$ . In particular, recalling that  $L = L(T, \epsilon)$  depends only on  $\epsilon$  and  $T$ , and choosing  $N = N(T, \epsilon)$  sufficiently large, we can combine the above estimate with (B.2) to ensure that
$$
\left| F _ {\epsilon} (\tau) - \frac {\Delta \omega}{2 \pi} \sum_ {j = 1} ^ {N} \widehat {F} \left(\omega_ {j}\right) \widehat {\rho} _ {\epsilon} \left(\omega_ {j}\right) e ^ {- i \omega_ {j} \tau} \right| \leq 2 \| f \| _ {L ^ {\infty}} \epsilon , \quad \forall \tau \in [ 0, T ],
$$
where we have taken into account that  $\| F\|_{L^{\infty}} = \| f\|_{L^{\infty}}$
Step 5: (Conclusion) To conclude the proof, we recall that  $\widehat{F}(\omega) = 2i\mathcal{L}_{t_0}u(\omega)$  can be expressed in terms of the sine transform  $\mathcal{L}_t u$  of the function  $u$  which was fixed at the beginning of Step 1. Recall also that  $f(\tau) = u(t_0 - \tau)$ , so that  $\| f \|_{L^{\infty}} = \| u \|_{L^{\infty}}$ . Hence, we can write the real part of  $\frac{\Delta\omega}{2\pi}\widehat{F}(\omega_j)\widehat{\rho}_{\epsilon}(\omega_j)e^{-i\omega_j\tau} = \frac{\Delta\omega}{2\pi}2i\mathcal{L}_{t_0}u(\omega_j)\widehat{\rho}_{\epsilon}(\omega_j)e^{-i\omega_j\tau}$ , in the form  $\alpha_j\mathcal{L}_{t_0}(\omega_j)\sin (\omega_j\tau - \vartheta_j)$  for coefficients  $\alpha_j \in \mathbb{R}$  and  $\theta_j \in \mathbb{R}$  which depend only on  $\Delta\omega$  and  $\widehat{\rho}_{\epsilon}(\omega_j)$ , but are independent of  $u$ . In
particular, it follows that
$$
\begin{array}{l} \sup  _ {\tau \in [ 0, \Delta t ]} \left| u (t _ {0} - \tau) - \sum_ {j = 1} ^ {N} \alpha_ {j} \mathcal {L} _ {t _ {0}} u (\omega_ {j}) \sin (\omega_ {j} \tau - \vartheta_ {j}) \right| = \sup  _ {t \in [ 0, \Delta t ]} \left| F (\tau) - \operatorname {R e} \left(\frac {\Delta \omega}{2 \pi} \sum_ {j = 1} ^ {N} \widehat {F} (\omega_ {j}) \widehat {\rho} _ {\epsilon} (\omega_ {j}) e ^ {- i \omega_ {j} \tau}\right) \right| \\ \leq \sup  _ {\tau \in [ 0, \Delta t ]} \left| F (\tau) - \frac {\Delta \omega}{2 \pi} \sum_ {j = 1} ^ {N} \widehat {F} \left(\omega_ {j}\right) \widehat {\rho} _ {\epsilon} \left(\omega_ {j}\right) e ^ {- i \omega_ {j} \tau} \right| \\ \leq \sup  _ {\tau \in [ 0, \Delta t ]} | F (\tau) - F _ {\epsilon} (\tau) | \\ + \sup  _ {\tau \in [ 0, \Delta t ]} \left| F _ {\epsilon} (\tau) - \frac {\Delta \omega}{2 \pi} \sum_ {j = 1} ^ {N} \widehat {F} \left(\omega_ {j}\right) \widehat {\rho} _ {\epsilon} \left(\omega_ {j}\right) e ^ {- i \omega_ {j} \tau} \right|. \\ \end{array}
$$
By Steps 1 and 3, the first term on the right-hand side is bounded by  $\leq \phi(\epsilon)$ , while the second one is bounded by  $\leq 2\sup_{u \in K} \|u\|_{L^{\infty}} \epsilon \leq C\epsilon$ , where  $C = C(K) < \infty$  depends only on the compact set  $K \subset C([0, T]; \mathbb{R}^p)$ . Hence, we have
$$
\sup _ {\tau \in [ 0, \Delta t ]} \left| u (t _ {0} - \tau) - \sum_ {j = 1} ^ {N} \alpha_ {j} \mathcal {L} _ {t _ {0}} u (\omega_ {j}) \sin (\omega_ {j} \tau - \vartheta_ {j}) \right| \leq \phi (\epsilon) + C \epsilon .
$$
In this estimate, the function  $u \in K$  and  $t_0 \in [0,T]$  were arbitrary, and the modulus of continuity  $\phi$  as well as the constant  $C$  on the right-hand side depend only on the set  $K$ . It thus follows that for this choice of  $\alpha_{j}, \omega_{j}$  and  $\vartheta_{j}$ , we have
$$
\sup  _ {u \in K} \sup  _ {t \in [ 0, T ]} \sup  _ {\tau \in [ 0, \Delta t ]} \left| u (t - \tau) - \sum_ {j = 1} ^ {N} \alpha_ {j} \mathcal {L} _ {t} u (\omega_ {j}) \sin (\omega_ {j} \tau - \vartheta_ {j}) \right| \leq \phi (\epsilon) + C \epsilon .
$$
Since  $\epsilon > 0$  was arbitrary, the right-hand side can be made arbitrarily small. The claim then readily follows.
The next step in the proof of the fundamental Lemma 3.6 needs the following preliminary result in functional analysis,
Lemma B.2. Let  $\mathcal{X},\mathcal{Y}$  be Banach spaces, and let  $K\subset \mathcal{X}$  be a compact subset. Assume that  $\Phi :\mathcal{X}\to \mathcal{Y}$  is continuous. Then for any  $\epsilon >0$  , there exists a  $\delta >0$  , such that if  $\| u - u^{K}\|_{\mathcal{X}}\leq \delta$  with  $u\in \mathcal{X},u^K\in K$  , then  $\| \Phi (u) - \Phi (u^K)\|_{\mathcal{Y}}\leq \epsilon$
Proof. Suppose not. Then there exists  $\epsilon_0 > 0$  and a sequence  $u_j, u_j^K$ ,  $(j \in \mathbb{N})$ , such that  $\|u_j - u_j^K\|_{\mathcal{X}} \leq j^{-1}$ , while  $\|\Phi(u_j) - \Phi(u_j^K)\|_{\mathcal{Y}} \geq \epsilon_0$ . By the compactness of  $K$ , we can extract a subsequence  $j_k \to \infty$ , such that  $u_{j_k}^K \to u^K$  converges to some  $u^K \in K$ . By assumption on  $u_j$ , this implies that
$$
\| u _ {j _ {k}} - u ^ {K} \| _ {\mathcal {X}} \leq \| u _ {j _ {k}} - u _ {j _ {k}} ^ {K} \| _ {\mathcal {X}} + \| u _ {j _ {k}} ^ {K} - u ^ {K} \| _ {\mathcal {X}} \stackrel {(k \rightarrow \infty)} {\longrightarrow} 0,
$$
which, by the assumed continuity of  $\Phi$ , leads to the contradiction that  $0 < \epsilon_0 \leq \| \Phi(u_{j_k}) - \Phi(u^K) \|_{\mathcal{V}} \to 0$ , as  $k \to \infty$ .
Proof of Lemma 3.6. Now, we can prove the fundamental Lemma in the following,
Proof. Let  $\epsilon > 0$  be given. We can identify  $K \subset C_0([0, T]; \mathbb{R}^p)$  with a compact subset of  $C((-\infty, T]; \mathbb{R}^p)$ , by extending all  $u \in K$  by zero for negative times, i.e. we set  $u(t) = 0$  for  $t < 0$ . Applying Lemma B.2, with  $\mathcal{X} = C_0([0, T]; \mathbb{R}^p)$  and  $\mathcal{Y} = C_0([0, T]; \mathbb{R}^q)$ , we can find a  $\delta > 0$  such that for any  $u \in C_0([0, T]; \mathbb{R}^p)$  and  $u^K \in K$ , we have
$$
\| u - u ^ {K} \| _ {L ^ {\infty}} \leq \delta \quad \Rightarrow \quad \| \Phi (u) - \Phi \left(u ^ {K}\right) \| _ {L ^ {\infty}} \leq \epsilon . \tag {B.3}
$$
By the inverse sine transform Lemma B.1, there exist  $N \in \mathbb{N}$ , frequencies  $\omega_1, \ldots, \omega_N \neq 0$ , phase-shifts  $\vartheta_1, \ldots, \vartheta_N$  and coefficients  $\alpha_1, \ldots, \alpha_N$ , such that for any  $u \in K$  and  $t \in [0, T]$ :
$$
\sup  _ {\tau \in [ 0, T ]} \left| u (t - \tau) - \sum_ {j = 1} ^ {N} \alpha_ {j} \mathcal {L} _ {t} u (\omega_ {j}) \sin (\omega_ {j} \tau - \vartheta_ {j}) \right| \leq \delta .
$$
Given  $\mathcal{L}_t u(\omega_1),\ldots ,\mathcal{L}_t u(\omega_N)$ , we can thus define a reconstruction mapping  $\mathcal{R}:\mathbb{R}^N\times [0,T]\to C([0,T];\mathbb{R}^p)$  by
$$
\mathcal {R} (\beta_ {1}, \dots , \beta_ {N}; t) (\tau) := \sum_ {j = 1} ^ {N} \alpha_ {j} \beta_ {j} \sin (\omega_ {j} (t - \tau) - \vartheta_ {j}).
$$
Then, for  $\tau \in [0,t]$ , we have
$$
\left| u (\tau) - \mathcal {R} \left(\mathcal {L} _ {t} u \left(\omega_ {1}\right), \dots , \mathcal {L} _ {t} u \left(\omega_ {N}\right); t\right) (\tau) \right| \leq \delta .
$$
We can now uniquely define  $\Psi : \mathbb{R}^N \times [0, T^2/4] \to C_0([0, T]; \mathbb{R}^p)$ , by the identity
$$
\Psi \left(\mathcal {L} _ {t} u \left(\omega_ {1}\right), \dots , \mathcal {L} _ {t} u \left(\omega_ {N}\right); t ^ {2} / 4\right) = \Phi \left(\mathcal {R} \left(\mathcal {L} _ {t} u \left(\omega_ {1}\right), \dots , \mathcal {L} _ {t} u \left(\omega_ {N}\right); t\right)\right).
$$
Using the short-hand notation  $\mathcal{R}_t u = \mathcal{R}(\mathcal{L}_t u(\omega_1), \ldots, \mathcal{L}_t u(\omega_N); t)$ , we have  $\sup_{\tau \in [0, t]} |u(\tau) - \mathcal{R}_t u(\tau)| \leq \delta$ , for all  $t \in [0, T]$ . By (B.3), this implies that
$$
\left| \Phi (u) (t) - \Psi (\mathcal {L} _ {t} u (\omega_ {1}), \ldots , \mathcal {L} _ {t} u (\omega_ {N}); t ^ {2} / 4) \right| = | \Phi (u) (t) - \Phi (\mathcal {R} _ {t} u) (t) | \leq \epsilon .
$$
B.3 Proof of Lemma 3.7
Proof. Let  $\omega \neq 0$  be given. For a (small) parameter  $s > 0$ , we consider
$$
\ddot {y} _ {s} = \frac {1}{s} \sigma (- s \omega^ {2} y _ {s} + s u), \quad y _ {s} (0) = \dot {y} _ {s} (0) = 0.
$$
Let  $Y$  be the solution of
$$
\dot {Y} = - \omega^ {2} Y + u, \quad Y (0) = \dot {Y} (0) = 0.
$$
Then we have, on account of  $\sigma(0) = 0$  and  $\sigma'(0) = 1$ ,
$$
\begin{array}{l} s ^ {- 1} \sigma (- s \omega^ {2} Y + s u) - [ - \omega^ {2} Y + u ] = \frac {\sigma (- s \omega^ {2} Y + s u) - \sigma (0)}{s} - \sigma^ {\prime} (0) [ - \omega^ {2} Y + u ] \\ = \frac {1}{s} \int_ {0} ^ {s} \frac {\partial}{\partial \zeta} [ \sigma (- \zeta \omega^ {2} Y + \zeta u) ] d \zeta - \sigma^ {\prime} (0) [ - \omega^ {2} Y + u ] \\ = \frac {1}{s} \left(\int_ {0} ^ {s} [ \sigma^ {\prime} (- \zeta \omega^ {2} Y + \zeta u) - \sigma^ {\prime} (0) ] d \zeta\right) [ - \omega^ {2} Y + u ]. \\ \end{array}
$$
It follows from Lemma 3.5 that for any input  $u \in K$ , with  $\sup_{u \in K} \| u \|_{L^{\infty}} =: B < \infty$ , we have a uniform bound  $\| Y \|_{L^{\infty}} \leq BT / \omega$ , hence we can estimate
$$
\left. \left| - \omega^ {2} Y + u \right| \leq B (\omega T + 1), \right.
$$
uniformly for all such  $u$ . In particular, it follows that
$$
\left| s ^ {- 1} \sigma (- s \omega^ {2} Y + s u) - [ - \omega^ {2} Y + u ] \right| \leq B (T \omega + 1) \sup _ {| x | \leq s B (T \omega + 1)} | \sigma^ {\prime} (x) - \sigma^ {\prime} (0) |.
$$
Clearly, for any  $\delta > 0$ , we can choose  $s \in (0,1]$  sufficiently small, such that the right hand-side is bounded by  $\delta$ , i.e. with this choice of  $s$ ,
$$
\left| s ^ {- 1} \sigma \left(- s \omega^ {2} Y (t) + s u (t)\right) - \left[ - \omega^ {2} Y (t) + u (t) \right] \right| \leq \delta , \quad \forall t \in [ 0, T ],
$$
holds for any choice of  $u \in K$ . We will fix this choice of  $s$  in the following, and write  $g(y,u) \coloneqq s^{-1}\sigma (-s\omega^2 y + su)$ . We note that  $g$  is Lipschitz continuous in  $y$ , for all  $|y| \leq BT / \omega$  and  $|u| \leq B$  with  $\mathrm{Lip}_y(g) \leq \omega^2\sup_{|\xi |\leq B(\omega T + 1)}|\sigma '(\xi)|$ .
To summarize, we have shown that  $Y$  solves
$$
\dot {Y} = g (Y, u) + f, \qquad Y (0) = \dot {Y} (0) = 0,
$$
where  $\| f\|_{L^{\infty}}\leq \delta$  . By definition,  $y_{s}$  solves
$$
\ddot {y} _ {s} = g (y _ {s}, u), \qquad y _ {s} (0) = \dot {y} _ {s} (0) = 0.
$$
It follows from this that
$$
\begin{array}{l} \left| y _ {s} (t) - Y (t) \right| \leq \int_ {0} ^ {t} \int_ {0} ^ {\tau} \left\{\left| g \left(y _ {s} (\theta), u (\theta)\right) - g (Y (\theta), u (\theta)) \right| + \left| f (\theta) \right| \right\} d \theta d \tau \\ \leq \int_ {0} ^ {t} \int_ {0} ^ {\tau} \left\{\operatorname {L i p} _ {y} (g) \left| y _ {s} (\theta) - Y (\theta) \right| + \delta \right\} d \theta d \tau \\ \leq T \omega^ {2} \sup  _ {| \xi | \leq B (\omega T + 1)} | \sigma^ {\prime} (\xi) | \int_ {0} ^ {t} | y _ {s} (\tau) - Y (\tau) | d \tau + T ^ {2} \delta . \\ \end{array}
$$
Recalling that  $Y(t) = \mathcal{L}_t u(\omega)$ , then by Gronwall's inequality, the last estimate implies that
$$
\sup  _ {t \in [ 0, T ]} | y _ {s} (t) - \mathcal {L} _ {t} u (\omega) | = \sup  _ {t \in [ 0, T ]} | y _ {s} - Y | \leq C \delta ,
$$
for a constant  $C = C(T,\omega ,\sup_{|\xi |\leq B(\omega T + 1)}|\sigma^{\prime}(\xi)|) > 0$  , depending only on  $T,\omega ,B$  and  $\sigma^\prime$  . Since  $\delta >0$  was arbitrary, we can ensure that  $C\delta \leq \epsilon$  . Thus, we have shown that a suitably rescaled nonlinear oscillator approximates the harmonic oscillator to any desired degree of accuracy, and uniformly for all  $u\in K$
To finish the proof, we observe that  $y$  solves
$$
\ddot {y} = \sigma (- \omega^ {2} y + s u), \qquad y (0) = \dot {y} (0) = 0,
$$
if, and only if,  $y_{s} = y / s$  solves
$$
\ddot {j} _ {s} = s ^ {- 1} \sigma (- s \omega^ {2} y _ {s} + s u), \qquad y _ {s} (0) = \dot {y} _ {s} (0) = 0.
$$
Hence, with  $W = -\omega^2$ ,  $V = s$ ,  $b = 0$  and  $A = s^{-1}$ , we have
$$
\sup  _ {t \in [ 0, T ]} | A y (t) - \mathcal {L} _ {t} u (\omega) | = \sup  _ {t \in [ 0, T ]} | y _ {s} (t) - \mathcal {L} _ {t} u (\omega) | \leq \epsilon .
$$
B.4 Proof of Lemma 3.8
Proof. Let  $\epsilon, \Delta t$  be given. By the sine transform reconstruction Lemma B.1, there exists  $N \in \mathbb{N}$ , frequencies  $\omega_1, \ldots, \omega_N$ , weights  $\alpha_1, \ldots, \alpha_N$  and phase-shifts  $\vartheta_1, \ldots, \vartheta_N$ , such that
$$
\sup  _ {\tau \in [ 0, \Delta t ]} \left| u (t - \tau) - \sum_ {j = 1} ^ {N} \alpha_ {j} \mathcal {L} _ {t} u \left(\omega_ {j}\right) \sin \left(\omega_ {j} \tau - \vartheta_ {j}\right) \right| \leq \frac {\epsilon}{2}, \quad \forall t \in [ 0, T ], \forall u \in K, \tag {B.4}
$$
where any  $u \in K$  is extended by zero to negative times. It follows from Lemma 3.7, that there exists a coupled oscillator network,
$$
\ddot {y} = \sigma (w \odot y + V u + b), \qquad y (0) = \dot {y} (0) = 0,
$$
with dimension  $m = pN$ , and  $w \in \mathbb{R}^m$ ,  $V \in \mathbb{R}^{m \times p}$ , and a linear output layer  $y \mapsto \widetilde{A}y$ ,  $\widetilde{A} \in \mathbb{R}^{m \times m}$ , such that  $[\widetilde{A}y(t)]_j \approx \mathcal{L}_t u(\omega_j)$  for  $j = 1, \dots, N$ ; more precisely, such that
$$
\sup  _ {t \in [ 0, T ]} \sum_ {j = 1} ^ {N} | \alpha_ {j} | \left| \mathcal {L} _ {t} u \left(\omega_ {j}\right) - \left[ \widetilde {A} y \right] _ {j} (t) \right| \leq \frac {\epsilon}{2}, \quad \forall u \in K. \tag {B.5}
$$
Composing with another linear layer  $B:\mathbb{R}^m\simeq \mathbb{R}^{p\times N}\to \mathbb{R}^p$ , which maps  $\beta = [\beta_{1},\dots ,\beta_{N}]$  to
$$
B \boldsymbol {\beta} := \sum_ {j = 1} ^ {N} \alpha_ {j} \beta_ {j} \sin (\omega_ {j} \Delta t - \vartheta_ {j}) \in \mathbb {R} ^ {p},
$$
we define  $A\coloneqq B\widetilde{A}$  , and observe that from (B.4) and (B.5):
$$
\begin{array}{l} \sup  _ {t \in [ 0, T ]} | u (t - \Delta t) - A y (t) | \leq \sup  _ {t \in [ 0, T ]} \left| u (t - \Delta t) - \sum_ {j = 1} ^ {N} \alpha_ {j} \mathcal {L} _ {t} u (\omega_ {j}) \sin (\omega_ {j} \Delta t - \vartheta_ {j}) \right| \\ + \sup  _ {t \in [ 0, T ]} \sum_ {j = 1} ^ {N} | \alpha_ {j} | \left| \mathcal {L} _ {t} u (\omega_ {j}) - [ \widetilde {A} y ] _ {j} (t) \right| | \sin (\omega_ {j} \Delta t - \vartheta_ {j}) | \\ \leq \epsilon . \\ \end{array}
$$
B.5 Proof of Lemma 3.9
Proof. Fix  $\Sigma, \Lambda, \gamma$  as in the statement of the lemma. Our goal is to approximate  $u \mapsto \Sigma \sigma (\Lambda u + \gamma)$ .
Step 1: (nonlinear layer) We consider a first layer for a hidden state  $y = [y_{1},y_{2}]^{T}\in \mathbb{R}^{p + p}$ , given by
$$
\left\{ \begin{array}{l} \ddot {y} _ {1} (t) = \sigma (\Lambda u (t) + \gamma) \\ \ddot {y} _ {2} (t) = \sigma (\gamma) \end{array} \right\}, \quad y (0) = \dot {y} (0) = 0.
$$
This layer evidently does not approximate  $\sigma (\Lambda u(t) + \gamma)$ ; however, it does encode this value in the second derivative of the hidden variable  $y_{1}$ . The main objective of the following analysis is to approximately compute  $\ddot{y}_1(t)$  through a suitably defined additional layer.
Step 2: (Second-derivative layer) To obtain an approximation of  $\sigma (\Lambda u(t) + \gamma)$ , we first note that the solution operator
$$
\mathcal {S}: u (t) \mapsto \eta (t), \quad \text {w h e r e} \ddot {\eta} (t) = \sigma (\Lambda u (t) + \gamma) - \sigma (\gamma), \quad \eta (0) = \dot {\eta} (0) = 0,
$$
defines a continuous mapping  $\mathcal{S}: C_0([0, T]; \mathbb{R}^p) \to C_0^2([0, T]; \mathbb{R}^p)$ , with  $\eta(0) = \dot{\eta}(0) = \ddot{\eta}(0) = 0$ . Note that  $\eta$  is very closely related to  $y_1$ . The fact that  $\dot{\eta} = 0$  is important to us, because it allows us to smoothly extend  $\eta$  to negative times by setting  $\eta(t) := 0$  for  $t < 0$  (which would not be true for  $y_1(t)$ ). The resulting extension defines a compactly supported function  $\eta: (-\infty, 0] \to \mathbb{R}^p$ , with  $\eta \in C^2((-\infty, T]; \mathbb{R}^p)$ . Furthermore, by continuity of the operator  $S$ , the image  $S(K)$  of the compact set  $K$  under  $S$  is compact in  $C^2((-\infty, T]; \mathbb{R}^p)$ . From this, it follows that for small  $\Delta t > 0$ , the second-order backward finite difference formula converges,
$$
\sup  _ {t \in [ 0, T ]} \left| \frac {\eta (t) - 2 \eta (t - \Delta t) + \eta (t - 2 \Delta t)}{\Delta t ^ {2}} - \ddot {\eta} (t) \right| = o _ {\Delta t \to 0} (1), \quad \forall \eta = \mathcal {S} (u), u \in K,
$$
where the bound on the right-hand side is uniform in  $u \in K$ , due to equicontinuity of  $\{\ddot{\eta} \mid \eta = S(u), u \in K\}$ . In particular, the second derivative of  $\eta$  can be approximated through linear combinations of time-delays of  $\eta$ . We can now choose  $\Delta t > 0$  sufficiently small so that
$$
\sup _ {t \in [ 0, T ]} \left| \frac {\eta (t) - 2 \eta (t - \Delta t) + \eta (t - 2 \Delta t)}{\Delta t ^ {2}} - \ddot {\eta} (t) \right| \leq \frac {\epsilon}{2 \| \Sigma \|}, \quad \forall y = \mathcal {S} (u), u \in K,
$$
where  $\| \Sigma \|$  denotes the operator norm of the matrix  $\Sigma$ . By Lemma 3.8, applied to the input set  $\widetilde{K} = S(K) \subset C_0([0,T];\mathbb{R}^p)$ , there exists a coupled oscillator
$$
\ddot {z} (t) = \sigma (w \odot z (t) + V \eta (t) + b), \quad z (0) = \dot {z} (0) = 0, \tag {B.6}
$$
and a linear output layer  $z \mapsto \widetilde{A} z$ , such that
$$
\sup  _ {t \in [ 0, T ]} \left| \left[ \eta (t) - 2 \eta (t - \Delta t) + \eta (t - 2 \Delta t) \right] - \widetilde {A} z (t) \right| \leq \frac {\epsilon \Delta t ^ {2}}{2 \| \Sigma \|}, \quad \forall \eta = \mathcal {S} (u), u \in K.
$$
Indeed, Lemma 3.8 shows that time-delays of any given input signal can be approximated with any desired accuracy, and  $\eta(t) - 2\eta(t - \Delta) - \eta(t - 2\Delta)$  is simply a linear combination of time-delays of the input signal  $\eta$  in (B.6).
To connect  $\eta(t)$  back to the  $y(t) = [y_1(t), y_2(t)]^T$  constructed in Step 1, we note that
$$
\ddot {\eta} = \sigma (A u (t) + b) - \sigma (b) = \ddot {y} _ {1} - \dot {y} _ {2},
$$
and hence, taking into account the initial values, we must have  $\eta \equiv y_1 - y_2$  by ODE uniqueness. In particular, upon defining a matrix  $\widetilde{V}$  such that  $\widetilde{V}y \coloneqq Vy_1 - Vy_2 \equiv V\eta$ , we can equivalently write (B.6) in the form,
$$
\ddot {z} (t) = \sigma (w \odot z (t) + \widetilde {V} y (t) + b), \quad z (0) = \dot {z} (0) = 0. \tag {B.7}
$$
Step 3: (Conclusion)
Composing the layers from Step 1 and 2, we obtain a coupled oscillator
$$
\ddot {y} ^ {\ell} = \sigma \left(w ^ {\ell} \odot y ^ {\ell} + V ^ {\ell} y ^ {\ell - 1} + b ^ {\ell}\right), \quad (\ell = 1, 2),
$$
initialized at rest, with  $y^{1} = y$ ,  $y^{2} = z$ , such that for  $A \coloneqq \Sigma \widetilde{A}$  and  $c \coloneqq \Sigma \sigma(\gamma)$ , we obtain
$$
\begin{array}{l} \sup  _ {t \in [ 0, T ]} \left| \left[ A y ^ {2} (t) + c \right] - \Sigma \sigma (\Lambda u (t) + \gamma) \right| \leq \| \Sigma \| \sup  _ {t \in [ 0, T ]} \left| \widetilde {A} z (t) - \left[ \sigma (\Lambda u (t) + \gamma) - \sigma (\gamma) \right] \right| \\ = \| \Sigma \| \sup  _ {t \in [ 0, T ]} \left| \widetilde {A} z (t) - \ddot {\eta} (t) \right| \\ \leq \| \Sigma \| _ {\sup  _ {t \in [ 0, T ]}} \left| \widetilde {A} z (t) - \frac {\eta (t) - 2 \eta (t - \Delta t) + \eta (t - 2 \Delta t)}{\Delta t ^ {2}} \right| \\ + \| \Sigma \| _ {\sup  _ {t \in [ 0, T ]}} \left| \frac {\eta (t) - 2 \eta (t - \Delta t) + \eta (t - 2 \Delta t)}{\Delta t ^ {2}} - \ddot {\eta} (t) \right| \\ \leq \frac {\epsilon}{2} + \frac {\epsilon}{2} = \epsilon . \\ \end{array}
$$
B.6 Proof of Theorem 3.1
Proof. Step 1: By the Fundamental Lemma 3.6, there exist  $N$ , a continuous mapping  $\Psi$ , and frequencies  $\omega_{1}, \ldots, \omega_{N}$ , such that
for all  $u\in K$  , and  $t\in [0,T]$  . Let  $M$  be a constant such that