SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| \mathcal {L} _ {t} u \left(\omega_ {1}\right) \right|, \dots , \left| \mathcal {L} _ {t} u \left(\omega_ {N}\right) \right|, \frac {t ^ {2}}{4} \leq M,
$$
for all  $u \in K$  and  $t \in [0, T]$ . By the universal approximation theorem for ordinary neural networks, there exist weight matrices  $\Sigma, \Lambda$  and bias  $\gamma$ , such that
$$
\left| \Psi \left(\beta_ {1}, \dots , \beta_ {N}; t ^ {2} / 4\right) - \Sigma \sigma (\Lambda \boldsymbol {\beta} + \gamma) \right| \leq \epsilon , \quad \boldsymbol {\beta} := \left[ \beta_ {1}, \dots , \beta_ {N}; t ^ {2} / 4 \right] ^ {T},
$$
holds for all  $t\in [0,T],|\beta_1|,\ldots ,|\beta_N|\leq M$
Step 2: Fix  $\epsilon_1 \leq 1$  sufficiently small, such that also  $\|\Sigma\| \|\Lambda\| \mathrm{Lip}(\sigma)\epsilon_1 \leq \epsilon$ , where  $\mathrm{Lip}(\sigma) := \sup_{|\xi| \leq \|\Lambda\| M + |\gamma| + 1} |\sigma'(\xi)|$  denotes an upper bound on the Lipschitz constant of the activation function over the relevant range of input values. It follows from Lemma 3.7, that there exists an oscillator network,
$$
\dot {y} ^ {1} = \sigma \left(w ^ {1} \odot y ^ {1} + V ^ {1} u + b ^ {1}\right), \quad y ^ {1} (0) = \dot {y} ^ {1} (0) = 0, \tag {B.8}
$$
of depth 1, such that
$$
\sup  _ {t \in [ 0, T ]} | \left[ \mathcal {L} _ {t} u \left(\omega_ {1}\right), \dots , \mathcal {L} _ {t} u \left(\omega_ {N}\right); t ^ {2} / 4 \right] ^ {T} - A ^ {1} y ^ {1} (t) | \leq \epsilon_ {1},
$$
for all  $u\in K$
Step 3: Finally, by Lemma 3.9, there exists an oscillator network,
$$
\dot {y} ^ {2} = \sigma \left(w ^ {2} \odot y ^ {2} + V ^ {2} y ^ {1} + b ^ {1}\right),
$$
of depth 2, such that
$$
\sup  _ {t \in [ 0, T ]} | A ^ {2} y ^ {2} (t) - \Sigma \sigma (\Lambda A ^ {1} y ^ {1} (t) + \gamma) | \leq \epsilon ,
$$
holds for all  $y^{1}$  belonging to the compact set  $K_{1} \coloneqq \mathcal{S}(K) \subset C_{0}([0,T];\mathbb{R}^{N + 1})$ , where  $\mathcal{S}$  denotes the solution operator of (B.8).
Step 4: Thus, we have for any  $u \in K$ , and with short-hand  $\mathcal{L}_t u(\omega) \coloneqq (\mathcal{L}_t u(\omega_1), \ldots, \mathcal{L}_t u(\omega_N))$
$$
\begin{array}{l} \left| \Phi (u) (t) - A ^ {2} y ^ {2} (t) \right| \leq \left| \Phi (u) (t) - \Psi (\mathcal {L} _ {t} u (\omega); t ^ {2} / 4) \right| \\ + \left| \Psi (\mathcal {L} _ {t} u (\omega); t ^ {2} / 4) - \Sigma \sigma (\Lambda [ \mathcal {L} _ {t} u (\omega); t ^ {2} / 4 ] + \gamma) \right| \\ + \left| \Sigma \sigma (\Lambda [ \mathcal {L} _ {t} u (\omega); t ^ {2} / 4 ] + \gamma) - \Sigma \sigma (\Lambda A ^ {1} y ^ {1} (t) + \gamma) \right| \\ + \left| \Sigma \sigma (\Lambda A _ {1} y _ {1} (t) + \gamma) - A ^ {2} y ^ {2} (t) \right|. \\ \end{array}
$$
By step 1, we can estimate
$$
\left| \Phi (u) (t) - \Psi (\mathcal {L} _ {t} u (\omega); t ^ {2} / 4) \right| \leq \epsilon , \quad \forall t \in [ 0, T ], u \in K.
$$
By the choice of  $\Sigma, \Lambda, \gamma$ , we have
$$
\left| \Psi (\mathcal {L} _ {t} u (\omega); t ^ {2} / 4) - \Sigma \sigma (\Lambda [ \mathcal {L} _ {t} u (\omega); t ^ {2} / 4 ] + \gamma) \right| \leq \epsilon , \quad \forall t \in [ 0, T ], u \in K.
$$
By construction of  $y^{1}$  in Step 2, we have
$$
\begin{array}{l} \left| \Sigma \sigma (\Lambda [ \mathcal {L} _ {t} u (\boldsymbol {\omega}); t ^ {2} / 4 ] + \gamma) - \Sigma \sigma (\Lambda A _ {1} y _ {1} (t) + \gamma) \right| \\ \leq \| \Sigma \| \operatorname {L i p} (\sigma) \| \Lambda \| | [ \mathcal {L} _ {t} u (\omega); t ^ {2} / 4 ] - A ^ {1} y ^ {1} (t) | \\ \leq \| \Sigma \| \operatorname {L i p} (\sigma) \| \Lambda \| \epsilon_ {1} \\ \leq \epsilon , \\ \end{array}
$$
for all  $t\in [0,T]$  and  $u\in K$  . By construction of  $y^{2}$  in Step 3, we have
$$
\left| \Sigma \sigma \left(\Lambda A ^ {1} y ^ {1} (t) + \gamma\right) - A ^ {2} y ^ {2} (t) \right| \leq \epsilon , \quad \forall t \in [ 0, T ], u \in K.
$$
$$
\left| \Phi (u) (t) - A ^ {2} y ^ {2} (t) \right| \leq 4 \epsilon ,
$$
for all  $t\in [0,T]$  and  $u\in K$ . Since  $\epsilon >0$  was arbitrary, we conclude that for any causal and continuous operator  $\Phi :C_0([0,T];\mathbb{R}^p)\to C_0([0,T];\mathbb{R}^q)$ , compact set  $K\subset C_0([0,T];\mathbb{R}^p)$  and  $\epsilon >0$ , there exists a coupled oscillator of depth 3, which uniformly approximates  $\Phi$  to accuracy  $\epsilon$  for all  $u\in K$ . This completes the proof.