SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
( \begin{array}{c c} I & - I \end{array} ) \circ \mathrm {R e L U} \circ \left( \begin{array}{c} I \\ - I \end{array} \right) = I.
$$
Thus, in both cases,  $H$  is injective.
In the case of (i), as  $\sigma \in A_0^L$ , we obtain the estimate
$$
\left\| \sigma (f) \right\| _ {L ^ {2} (D) ^ {d _ {i n}}} \leq \sqrt {2 | D | d _ {i n}} C _ {0} + \| f \| _ {L ^ {2} (D) ^ {d _ {i n}}}, f \in L ^ {2} (D) ^ {d _ {i n}},
$$
$$
C _ {0} := \sup  _ {x \in \mathbb {R}} \frac {| \sigma (x) |}{1 + | x |} <   \infty .
$$
Then we evaluate for  $a \in K(\subset B_R(0))$ ,
$$
\begin{array}{l} \left\| H (a) \right\| _ {L ^ {2} (D) ^ {d _ {i n} + d _ {o u t}}} \\ \leq \left\| \widetilde {G} (a) \right\| _ {L ^ {2} (D) ^ {d _ {\text {o u t}}}} + \| K _ {i n j} \circ K _ {i n j} \circ \sigma \circ K _ {i n j} \circ \dots \circ \sigma \circ K _ {i n j} \circ K _ {i n j} (a) \| _ {L ^ {2} (D) ^ {d _ {\text {i n}}}} \tag {C.7} \\ \leq 4 M + \sqrt {2 | D | d _ {i n}} C _ {0} \sum_ {\ell = 1} ^ {L} \| K _ {i n j} \| _ {\mathrm {o p}} ^ {\ell + 1} + \| K _ {i n j} \| _ {\mathrm {o p}} ^ {L + 2} R =: C _ {H}. \\ \end{array}
$$
In the case of (ii), we find the estimate, for  $a \in K$ ,
$$
\left\| H (a) \right\| _ {L ^ {2} (D) ^ {d _ {i n} + d _ {o u t}}} \leq 4 M + \left\| K _ {i n j} \right\| _ {\mathrm {o p}} ^ {L + 2} R <   C _ {H}. \tag {C.8}
$$
From (C.6) (especially,  $\operatorname{Ran}(\pi_1H) \subset C(D)$ ) and Remark 3, we can choose an orthogonal sequence  $\{\xi_k\}_{k \in \mathbb{N}}$  in  $L^2(D)$  such that (3.1) holds. By applying Lemma 1, as  $T = H$ ,  $n = d_{in}$ ,  $m = d_{in} + d_{out}$ ,  $\ell = d_{out}$ , we find that
$$
G := \underbrace {\pi_ {d _ {o u t}} \circ Q _ {\alpha} \circ P _ {V _ {\alpha} ^ {\perp}}} _ {=: B} \circ H: L ^ {2} (D) ^ {d _ {i n}} \to L ^ {2} (D) ^ {d _ {o u t}},
$$
is injective. Here,  $P_{V_{\alpha}^{\perp}}$  and  $Q_{\alpha}$  are defined as in Remark 4; we choose  $0 < \alpha << 1$  such that
$$
\left\| P _ {V _ {\alpha} ^ {\perp}} - P _ {V _ {0} ^ {\perp}} \right\| _ {\mathrm {o p}} <   \min \left(\frac {\epsilon}{1 0 C _ {H}}, 1\right) =: \epsilon_ {0},
$$
where  $P_{V_0^\perp}$  is the orthogonal projection onto
$$
V _ {0} ^ {\perp} := \{0 \} ^ {d _ {i n}} \times L ^ {2} (D) ^ {d _ {o u t}}.
$$
By the same argument as in the proof of Theorem 15 in Puthawala et al. [2022a], we can show that
$$
\left\| I - Q _ {\alpha} \right\| _ {\mathrm {o p}} \leq 4 \epsilon_ {0}.
$$
Furthermore, since  $B$  is a linear operator,  $B \circ K_{L + 1}$  is also a linear operator with integral kernel  $(Bk_{L + 1}(\cdot ,y))(x)$ , where  $k_{L + 1}(x,y)$  is the kernel of  $K_{L + 1}$ . This implies that
$$
G \in \mathrm {N O} _ {L} (\sigma ; D, d _ {i n}, d _ {o u t}).
$$
We get, for  $a \in K$ ,
$$
\left\| G ^ {+} (a) - G (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \leq \underbrace {\left\| G ^ {+} (a) - \widetilde {G} (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}}} _ {\text {(C . 5)} \leq \frac {\epsilon}{2}} + \left\| \widetilde {G} (a) - G (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}}. \tag {C.9}
$$
Using (C.7) and (C.8), we then obtain
$$
\begin{array}{l} \left\| \widetilde {G} (a) - G (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} = \left\| \pi_ {d _ {o u t}} \circ H (a) - \pi_ {d _ {o u t}} \circ Q _ {\alpha} \circ P _ {V _ {\alpha} ^ {\perp}} \circ H (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \\ \leq \left\| \pi_ {d _ {o u t}} \circ \left(P _ {V _ {0} ^ {\perp}} - P _ {V _ {\alpha} ^ {\perp}} + P _ {V _ {\alpha} ^ {\perp}}\right) \circ H (a) - \pi_ {d _ {o u t}} \circ Q _ {\alpha} \circ P _ {V _ {\alpha} ^ {\perp}} \circ H (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \\ \leq \left\| \pi_ {d _ {o u t}} \circ (P _ {V _ {0} ^ {\perp}} - P _ {V _ {\alpha} ^ {\perp}}) \circ H (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} + \left\| \pi_ {d _ {o u t}} \circ (I - Q _ {\alpha}) \circ P _ {V _ {\alpha} ^ {\perp}} \circ H (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \\ \leq 5 \epsilon_ {0} \| H (a) \| _ {L ^ {2} (D) ^ {d _ {i n} + d _ {o u t}}} \leq \frac {\epsilon}{2}. \tag {C.10} \\ \end{array}
$$
Combining (C.9) and (C.10), we conclude that
$$
\sup  _ {a \in K} \left\| G ^ {+} (a) - G (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \leq \frac {\epsilon}{2} + \frac {\epsilon}{2} = \epsilon .
$$
C.4 Remark following Theorem 1
Remark 5. We make the following observations using Theorem 1:
(i) ReLU and Leaky ReLU functions belong to  $\mathrm{A}_0^L \cap \mathrm{BA}$  due to the fact that  $\{\sigma \in C(\mathbb{R}) \mid \sigma \text{ is not a polynomial}\} \subseteq A_0$  (see Pinkus [1999]), and both the ReLU and Leaky ReLU functions belong to BA (see Lemma C.2 in Lanthaler et al. [2022]). We note that Lemma C.2 in Lanthaler et al. [2022] solely established the case for ReLU. However, it holds true for Leaky ReLU as well since the proof relies on the fact that the function  $x \mapsto \min(\max(x, R), R)$  can be exactly represented by a two-layer ReLU neural network, and a two-layer Leaky ReLU neural network can also represent this function. Consequently, Leaky ReLU is one of example that satisfies (ii) in Theorem 1.
(ii) We emphasize that our infinite-dimensional result, Theorem 1, deviates from the finite-dimensional result. Puthawala et al. [2022a, Theorem 15] assumes that  $2d_{in} + 1 \leq d_{out}$  due to the use of Whitney's theorem. In contrast, Theorem 1 does not assume any conditions on  $d_{in}$  and  $d_{out}$ , that is, we are able to avoid invoking Whitney's theorem by employing Lemma 1.
(iii) We provide examples that injective universality does not hold when  $L^2(D)^{d_{in}}$  and  $L^2(D)^{d_{out}}$  are replaced by  $\mathbb{R}^{d_{in}}$  and  $\mathbb{R}^{d_{out}}$ : Consider the case where  $d_{in} = d_{out} = 1$  and  $G^+ : \mathbb{R} \to \mathbb{R}$  is defined as  $G^+(x) = \sin(x)$ . We can not approximate  $G^+ : \mathbb{R} \to \mathbb{R}$  by an injective function  $G : \mathbb{R} \to \mathbb{R}$  in the set  $K = [0, 2\pi]$  in the  $L^\infty$ -norm. According to the topological degree theory (see Cho and Chen [2006, Theorem 1.2.6(iii)]), any continuous function  $G : \mathbb{R} \to \mathbb{R}$  which satisfies  $\| G - G^+ \|_{C([0, 2\pi])} < \varepsilon$  satisfies the equation on both intervals  $I_1 = [0, \pi]$ ,  $I_2 = [\pi, 2\pi]$ ,  $deg(G, I_j, s) = deg(G^+, I_j, s) = 1$  for all  $s \in [-1 + \varepsilon, 1 - \varepsilon]$ ,  $j = 1, 2$ . This implies that  $G : I_j \to \mathbb{R}$  obtains the value  $s \in [-1 + \varepsilon, 1 - \varepsilon]$  at least once. Hence,  $G$  obtains the values  $s \in [-1 + \varepsilon, 1 - \varepsilon]$  at least two times on the interval  $[0, 2\pi]$  and is thus not injective. It is worth noting that the degree theory exhibits significant differences between the infinite-dimensional and finite-dimensional cases [Cho and Chen, 2006]).
D Details in Section 3.3
D.1 Finite rank approximation
We consider linear integral operators  $K_{\ell}$  with  $L^2$  kernels  $k_{\ell}(x,y)$ . Let  $\{\varphi_k\}_{k\in \mathbb{N}}$  be an orthonormal basis in  $L^2 (D)$ . Since  $\{\varphi_k(y)\varphi_p(x)\}_{k,p\in \mathbb{N}}$  is an orthonormal basis of  $L^2 (D\times D)$ , integral kernels  $k_{\ell}\in L^{2}(D\times D;\mathbb{R}^{d_{\ell +1}\times d_{\ell}})$  in integral operators  $K_{\ell}\in \mathcal{L}(L^{2}(D)^{d_{\ell}},L^{2}(D)^{d_{\ell +1}})$  has the expansion
then integral operators  $K_{\ell}\in \mathcal{L}(L^{2}(D)^{d_{\ell}},L^{2}(D)^{d_{\ell +1}})$  take the form
$$
K _ {\ell} u (x) = \sum_ {k, p \in \mathbb {N}} C _ {k, p} ^ {(\ell)} (u, \varphi_ {k}) \varphi_ {p} (x), u \in L ^ {2} (D) ^ {d _ {\ell}},
$$
$$
c _ {k, p, i j} ^ {(\ell)} = \left(k _ {\ell , i j}, \varphi_ {k} \varphi_ {p}\right) _ {L ^ {2} (D \times D)}.
$$
Here, we write  $(u,\varphi_k)\in \mathbb{R}^{d_\ell}$
$$
(u, \varphi_ {k}) = \left((u _ {1}, \varphi_ {k}) _ {L ^ {2} (D)},..., (u _ {d _ {\ell}}, \varphi_ {k}) _ {L ^ {2} (D)}\right).
$$
We define  $K_{\ell, N} \in \mathcal{L}(L^2(D)^{d_\ell}, L^2(D)^{d_{\ell+1}})$  as the truncated expansion of  $K_\ell$  by  $N$  finite sum, that is,
$$
K _ {\ell , N} u (x) := \sum_ {k, p \leq N} C _ {k, p} ^ {(\ell)} (u, \varphi_ {k}) \varphi_ {p} (x).
$$
Then  $K_{\ell ,N}\in \mathcal{L}(L^{2}(D)^{d_{\ell}},L^{2}(D)^{d_{\ell +1}})$  is a finite rank operator with rank  $N$  . Furthermore, we have
$$
\left\| K _ {\ell} - K _ {\ell , N} \right\| _ {\mathrm {o p}} \leq \left\| K _ {\ell} - K _ {\ell , N} \right\| _ {\mathrm {H S}} = \left(\sum_ {k, p \geq N} \sum_ {i, j} \left| c _ {k, p, i j} ^ {(\ell)} \right| ^ {2}\right) ^ {1 / 2}, \tag {D.1}
$$
which implies that as  $N\to \infty$
$$
\left\| K _ {\ell} - K _ {\ell , N} \right\| _ {\mathrm {o p}} \rightarrow 0.
$$
D.2 Layerwise injectivity
We first revisit layerwise injectivity and bijectivity in the case of the finite rank approximation. Let  $K_{N}: L^{2}(D)^{n} \to L^{2}(D)^{m}$  be a finite rank operator defined by
$$
K _ {N} u (x) := \sum_ {k, p \leq N} C _ {k, p} (u, \varphi_ {k}) \varphi_ {p} (x), u \in L ^ {2} (D) ^ {n},
$$
where  $C_{k,p}\in \mathbb{R}^{m\times n}$  and  $(u,\varphi_p)\in \mathbb{R}^n$  is given by
$$
(u, \varphi_ {p}) = \left(\left(u _ {1}, \varphi_ {p}\right) _ {L ^ {2} (D)}, \dots , \left(u _ {n}, \varphi_ {p}\right) _ {L ^ {2} (D)}\right).
$$
Let  $b_{N}\in L^{2}(D)^{n}$  be defined by
$$
b _ {N} (x) := \sum_ {p \leq N} b _ {p} \varphi_ {p} (x),
$$
in which  $b_{p}\in \mathbb{R}^{m}$ . As analogues of Propositions 1 and 2, we obtain the following characterization.
Proposition 5. (i) The operator
$$
\operatorname {R e L U} \circ \left(K _ {N} + b _ {N}\right): (\operatorname {s p a n} \left\{\varphi_ {k} \right\} _ {k \leq N}) ^ {n} \rightarrow L ^ {2} (D) ^ {m},
$$
is injective if and only if for every  $v\in (\mathrm{span}\{\varphi_k\}_{k\leq N})^n$
$$
\left\{u \in L ^ {2} (D) ^ {n} \mid \vec {u} _ {N} \in \operatorname {K e r} \left(C _ {S, N}\right) \right\} \cap X (v, K _ {N} + b _ {N}) \cap (\operatorname {s p a n} \{\varphi_ {k} \} _ {k \leq N}) ^ {n} = \{0 \}.
$$
where  $S(v, K_N + b_N) \subset [m]$  and  $X(v, K_N + b_N)$  are defined in Definition 2, and
$$
\vec {u} _ {N} := \left(\left(u, \varphi_ {p}\right)\right) _ {p \leq N} \in \mathbb {R} ^ {N n}, C _ {S, N} := \left(C _ {k, q} \mid_ {S \left(v, K _ {N} + b _ {N}\right)}\right) _ {k, q \in [ N ]} \in \mathbb {R} ^ {N \left| S \left(v, K _ {N} + b _ {N}\right) \right| \times N n}. \tag {D.2}
$$
(ii) Let  $\sigma$  be injective. Then the operator
$$
\sigma \circ \left(K _ {N} + b _ {N}\right): \left(\operatorname {s p a n} \left\{\varphi_ {k} \right\} _ {k \leq N}\right) ^ {n} \rightarrow L ^ {2} (D) ^ {m},
$$
is injective if and only if  $C_N$  is injective, where
$$
C _ {N} := \left(C _ {k, q}\right) _ {k, q \in [ N ]} \in \mathbb {R} ^ {N m \times N n}. \tag {D.3}
$$
Proof. The above statements follow from Propositions 1 and 2 by observing that  $u \in \mathrm{Ker}(K_N)$  is equivalent to (cf. (D.2) and (D.3))
$$
\sum_ {k, p \leq N} C _ {k, p} (u, \varphi_ {k}) \varphi_ {p} = 0, \Longleftrightarrow C _ {N} \vec {u} _ {N} = 0.
$$
D.3 Global injectivity
We revisit global injectivity in the case of finite rank approximation. As an analogue of Lemma 1, we have the following
Lemma 2. Let  $N, N' \in \mathbb{N}$  and  $n, m, \ell \in \mathbb{N}$  with  $N'm > N'\ell \geq 2Nn + 1$ , and let  $T: L^2(D)^n \to L^2(D)^m$  be a finite rank operator with  $N'$  rank, that is,
$$
\operatorname {R a n} (T) \subset (\operatorname {s p a n} \left\{\varphi_ {k} \right\} _ {k \leq N ^ {\prime}}) ^ {m}, \tag {D.4}
$$
and Lipschitz continuous, and
$$
T: (\operatorname {s p a n} \left\{\varphi_ {k} \right\} _ {k \leq N}) ^ {n} \rightarrow L ^ {2} (D) ^ {m},
$$
is injective. Then, there exists a finite rank operator  $B \in \mathcal{L}(L^2(D)^m, L^2(D)^\ell)$  with rank  $N'$  such that
$$
B \circ T: (\operatorname {s p a n} \{\varphi_ {k} \} _ {k \leq N}) ^ {n} \rightarrow (\operatorname {s p a n} \{\varphi_ {k} \} _ {k \leq N ^ {\prime}}) ^ {\ell},
$$
Proof. From (D.4),  $T: L^2(D)^n \to L^2(D)^m$  has the form of
$$
T (a) = \sum_ {k \leq N ^ {\prime}} \left(T (a), \varphi_ {k}\right) \varphi_ {k},
$$
where  $(T(a),\varphi_k)\in \mathbb{R}^m$  . We define  $\mathbf{T}:\mathbb{R}^{Nn}\to \mathbb{R}^{N'm}$  by
$$
\mathbf {T} (\mathbf {a}) := \left(\left(T (\mathbf {a}), \varphi_ {k}\right)\right) _ {k \in [ N ^ {\prime} ]} \in \mathbb {R} ^ {N ^ {\prime} m}, \mathbf {a} \in \mathbb {R} ^ {N n},
$$
where  $T(\mathbf{a}) \in L^{2}(D)^{m}$  is defined by
$$
T (\mathbf {a}) := T \left(\sum_ {k \leq N} a _ {k} \varphi_ {k}\right) \in L ^ {2} (D) ^ {m},
$$
in which  $a_{k}\in \mathbb{R}^{n},\mathbf{a} = (a_{1},\dots,a_{N})\in \mathbb{R}^{Nn}$
Since  $T: L^2(D)^n \to L^2(D)^m$  is Lipschitz continuous,  $\mathbf{T}: \mathbb{R}^{Nn} \to \mathbb{R}^{N'm}$  is also Lipschitz continuous. As  $N'm > N'\ell \geq 2Nn + 1$ , we can apply Lemma 29 from Puthawala et al. [2022a] with  $D = N'm$ ,  $m = N'\ell$ ,  $n = Nn$ . According to this lemma, there exists a  $N'\ell$ -dimensional linear subspace  $\mathbf{V}^\perp$  in  $\mathbb{R}^{N'm}$  such that
$$
\left\| P _ {\mathbf {V} ^ {\perp}} - P _ {\mathbf {V} _ {0} ^ {\perp}} \right\| _ {\mathrm {o p}} <   1,
$$
$$
P _ {\mathbf {V} ^ {\perp}} \circ \mathbf {T}: \mathbb {R} ^ {N n} \rightarrow \mathbb {R} ^ {N ^ {\prime} m},
$$
is injective, where  $\mathbf{V}_{0}^{\perp} = \{0\}^{N'(m - \ell)} \times \mathbb{R}^{N'\ell}$ . Furthermore, in the proof of Theorem 15 of Puthawala et al. [2022a], denoting
$$
\mathbf {B} := \pi_ {N ^ {\prime} \ell} \circ \mathbf {Q} \circ P _ {\mathbf {V} ^ {\perp}} \in \mathbb {R} ^ {N ^ {\prime} \ell \times N ^ {\prime} m},
$$
we are able to show that
$$
\mathbf {B} \circ \mathbf {T}: \mathbb {R} ^ {N n} \to \mathbb {R} ^ {N ^ {\prime} \ell}
$$
is injective. Here,  $\pi_{N'\ell} : \mathbb{R}^{N'm} \to \mathbb{R}^{N'\ell}$
$$
\pi_ {N ^ {\prime} \ell} (a, b) := b, \quad (a, b) \in \mathbb {R} ^ {N ^ {\prime} (m - \ell)} \times \mathbb {R} ^ {N ^ {\prime} \ell},
$$
where  $\mathbf{Q}:\mathbb{R}^{N^{\prime}m}\to \mathbb{R}^{N^{\prime}m}$  is defined by
$$
\mathbf {Q} := \left(P _ {\mathbf {V} _ {0} ^ {\perp}} P _ {\mathbf {V} ^ {\perp}} + (I - P _ {\mathbf {V} _ {0} ^ {\perp}}) (I - P _ {\mathbf {V} ^ {\perp}})\right) \left(I - (P _ {\mathbf {V} _ {0} ^ {\perp}} - P _ {\mathbf {V} ^ {\perp}}) ^ {2}\right) ^ {- 1 / 2}.
$$
We define  $B:L^{2}(D)^{m}\to L^{2}(D)^{\ell}$  by
$$
B u = \sum_ {k, p \leq N ^ {\prime}} \mathbf {B} _ {k, p} (u, \varphi_ {k}) \varphi_ {p},
$$
where  $\mathbf{B}_{k,p} \in \mathbb{R}^{\ell \times m}$ ,  $\mathbf{B} = (\mathbf{B}_{k,p})_{k,p \in [N']}$ . Then  $B: L^{2}(D)^{m} \to L^{2}(D)^{\ell}$  is a linear finite rank operator with  $N'$  rank, and
$$
B \circ T: L ^ {2} (D) ^ {n} \to L ^ {2} (D) ^ {\ell}
$$
is injective because, by the construction, it is equivalent to
$$
\mathbf {B} \circ \mathbf {T}: \mathbb {R} ^ {N n} \to \mathbb {R} ^ {N ^ {\prime} \ell},
$$
Definition 4. We define the set of integral neural operators with  $N$  rank by
$$
\mathrm {N O} _ {L, N} (\sigma ; D, d _ {i n}, d _ {o u t}) := \left\{G _ {N}: L ^ {2} (D) ^ {d _ {i n}} \rightarrow L ^ {2} (D) ^ {d _ {o u t}}: \right.
$$
$$
G _ {N} = K _ {L + 1, N} \circ \left(K _ {L, N} + b _ {L, N}\right) \circ \sigma \dots \circ \left(K _ {2, N} + b _ {2, N}\right) \circ \sigma \circ \left(K _ {1, N} + b _ {1, N}\right) \circ \left(K _ {0, N} + b _ {0, N}\right),
$$
$$
K _ {\ell , N} \in \mathcal {L} \left(L ^ {2} (D) ^ {d _ {\ell}}, L ^ {2} (D) ^ {d _ {\ell + 1}}\right), K _ {\ell , N}: f \mapsto \sum_ {k, p \leq N} C _ {k, p} ^ {(\ell)} (f, \varphi_ {k}) \varphi_ {p},
$$
$$
b _ {\ell , N} \in L ^ {2} (D; \mathbb {R} ^ {d _ {\ell + 1}}), b _ {\ell , N} = \sum_ {p \leq N} b _ {p} ^ {(\ell)} \varphi_ {m},
$$
$$
C _ {k, p} ^ {(\ell)} \in \mathbb {R} ^ {d _ {\ell + 1} \times d _ {\ell}}, b _ {p} ^ {(\ell)} \in \mathbb {R} ^ {d _ {\ell + 1}}, k, p \leq N,
$$
$$
\left. d _ {\ell} \in \mathbb {N}, d _ {0} = d _ {i n}, d _ {L + 2} = d _ {o u t}, \ell = 0, \dots , L + 2 \right\}. \tag {D.5}
$$
where  $B_R(0) \coloneqq \{u \in L^2(D)^{d_{in}} \mid \| u \|_{L^2(D)^{d_{in}}} \leq R\}$ . As ReLU and Leaky ReLU function belongs to  $\mathrm{A}_0^L \cap \mathrm{BA}$ , by Theorem 11 of Kovachki et al. [2021b], there exists  $L \in \mathbb{N}$  and  $\widetilde{G} \in \mathrm{NO}_L(\sigma; D, d_{in}, d_{out})$  such that
$$
\sup  _ {a \in K} \left\| G ^ {+} (a) - \widetilde {G} (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \leq \frac {\epsilon}{3}. \tag {D.6}
$$
$$
\widetilde {k} _ {\ell} \in L ^ {2} (D \times D; \mathbb {R} ^ {d _ {\ell + 1} \times d _ {\ell}}), \widetilde {b} _ {\ell} \in L ^ {2} (D; \mathbb {R} ^ {d _ {\ell + 1}}),
$$
We set  $\widetilde{G}_{N'} \in \mathrm{NO}_{L,N'}(\sigma; D, d_{in}, d_{out})$  such that
$$
\widetilde {G} _ {N ^ {\prime}} = \widetilde {K} _ {L + 1, N ^ {\prime}} \circ (\widetilde {K} _ {L, N ^ {\prime}} + \widetilde {b} _ {L, N ^ {\prime}}) \circ \sigma \dots \circ (\widetilde {K} _ {2, N ^ {\prime}} + \widetilde {b} _ {2, N ^ {\prime}}) \circ \sigma \circ (\widetilde {K} _ {1, N ^ {\prime}} + \widetilde {b} _ {1, N ^ {\prime}}) \circ (\widetilde {K} _ {0, N ^ {\prime}} + \widetilde {b} _ {0, N ^ {\prime}}),
$$
where  $\widetilde{K}_{\ell, N'}: L^2(D)^{d_\ell} \to L^2(D)^{d_{\ell+1}}$  is defined by
$$
\widetilde {K} _ {\ell , N ^ {\prime}} u (x) = \sum_ {k, p \leq N ^ {\prime}} C _ {k, p} ^ {(\ell)} (u, \varphi_ {k}) \varphi_ {p} (x),
$$
$$
c _ {k, p, i j} ^ {(\ell)} = \left(\widetilde {k} _ {\ell , i j}, \varphi_ {k} \varphi_ {p}\right) _ {L ^ {2} (D \times D)}.
$$
$$
\left\| \widetilde {K} _ {\ell} - \widetilde {K} _ {\ell , N ^ {\prime}} \right\| _ {\mathrm {o p}} ^ {2} \leq \left\| \widetilde {K} _ {\ell} - \widetilde {K} _ {\ell , N ^ {\prime}} \right\| _ {\mathrm {H S}} ^ {2} = \sum_ {k, p \geq N ^ {\prime} + 1} \sum_ {i, j} | c _ {k, p, i j} ^ {(\ell)} | ^ {2} \to 0 \text {a s} N ^ {\prime} \to \infty ,
$$
there is a large  $N^{\prime}\in \mathbb{N}$  such that
$$
\sup  _ {a \in K} \left\| \widetilde {G} (a) - \widetilde {G} _ {N ^ {\prime}} (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \leq \frac {\epsilon}{3}. \tag {D.7}
$$
$$
\begin{array}{l} \sup  _ {a \in K} \left\| \widetilde {G} _ {N ^ {\prime}} (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \leq \sup  _ {a \in K} \left\| \widetilde {G} _ {N ^ {\prime}} (a) - \widetilde {G} (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} + \sup  _ {a \in K} \left\| \widetilde {G} (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \\ \leq 1 + 4 M. \\ \end{array}
$$
We define the operator  $H_{N'}: L^2(D)^{d_{in}} \to L^2(D)^{d_{in} + d_{out}}$  by
$$
H _ {N ^ {\prime}} (a) = \left( \begin{array}{c} H _ {N ^ {\prime}} (a) _ {1} \\ H _ {N ^ {\prime}} (a) _ {2} \end{array} \right) := \left( \begin{array}{c} K _ {i n j, N} \circ \dots \circ K _ {i n j, N} (a) \\ \widetilde {G} _ {N ^ {\prime}} (a) \end{array} \right),
$$
where  $K_{inj,N}:L^2 (D)^{d_{in}}\to L^2 (D)^{d_{in}}$  is defined by
$$
K _ {i n j, N} u = \sum_ {k \leq N} (u, \varphi_ {k}) \varphi_ {k}.
$$
As  $K_{inj,N}:(\mathrm{span}\{\varphi_k\}_{k\leq N})^{d_{in}}\to L^2 (D)^{d_{in}}$  is injective,
$$
H _ {N ^ {\prime}}: (\operatorname {s p a n} \{\varphi_ {k} \} _ {k \leq N}) ^ {d _ {i n}} \rightarrow (\operatorname {s p a n} \{\varphi_ {k} \} _ {k \leq N}) ^ {d _ {i n}} \times (\operatorname {s p a n} \{\varphi_ {k} \} _ {k \leq N ^ {\prime}}) ^ {d _ {o u t}},
$$
is injective. Furthermore, by the same argument (ii) (construction of  $H$ ) in the proof of Theorem 1,
$$
H _ {N ^ {\prime}} \in N O _ {L, N ^ {\prime}} (\sigma ; D, d _ {i n}, d _ {o u t}),
$$
because both of two-layer ReLU and Leaky ReLU neural networks can represent the identity map. Note that above  $K_{inj,N}$  is an orthogonal projection, so that  $K_{inj,N} \circ \dots \circ K_{inj,N} = K_{inj,N}$ . However, we write above  $H_{N'}(a)_1$  as  $K_{inj,N} \circ \dots \circ K_{inj,N}(a)$  so that it can be considered as a combination of  $(L + 2)$  layers of neural networks.
We estimate that for  $a \in L^2(D)^{d_{in}}$ ,  $\|a\|_{L^2(D)^{d_{in}}} \leq R$ ,
$$
\left\| H _ {N ^ {\prime}} (a) \right\| _ {L ^ {2} (D) ^ {d _ {i n} + d _ {o u t}}} \leq 1 + 4 M + \left\| K _ {i n j} \right\| _ {\mathrm {o p}} ^ {L + 2} R =: C _ {H}.
$$
Here, we repeat an argument similar to the one in the proof of Lemma 2:  $H_{N'}: L^2(D)^{d_{in}} \to L^2(D)^{d_{in} + d_{out}}$  has the form of
$$
H _ {N ^ {\prime}} (a) = \left(\sum_ {k \leq N} \left(H _ {N ^ {\prime}} (a) _ {1}, \varphi_ {k}\right) \varphi_ {k}, \sum_ {k \leq N ^ {\prime}} \left(H _ {N ^ {\prime}} (a) _ {2}, \varphi_ {k}\right) \varphi_ {k}\right).
$$
where  $(H_{N^{\prime}}(a)_1,\varphi_k)\in \mathbb{R}^{d_{in}}$ $(H_{N^{\prime}}(a)_2,\varphi_k)\in \mathbb{R}^{d_{out}}$ . We define  $\mathbf{H}_{N^{\prime}}:\mathbb{R}^{Nd_{in}}\to \mathbb{R}^{Nd_{in} + N^{\prime}d_{out}}$  by
$$
\mathbf {H} _ {N ^ {\prime}} (\mathbf {a}) := \left[ \big ((H _ {N ^ {\prime}} (\mathbf {a}) _ {1}, \varphi_ {k}) \big) _ {k \in [ N ]}, \big ((H _ {N ^ {\prime}} (\mathbf {a}) _ {2}, \varphi_ {k}) \big) _ {k \in [ N ^ {\prime} ]} \right] \in \mathbb {R} ^ {N d _ {i n} + N ^ {\prime} d _ {o u t}}, \mathbf {a} \in \mathbb {R} ^ {N d _ {i n}},
$$
where  $H_{N^{\prime}}(\mathbf{a}) = (H_{N^{\prime}}(\mathbf{a})_{1},H_{N^{\prime}}(\mathbf{a})_{2})\in L^{2}(D)^{d_{in} + d_{out}}$  is defined by
$$
H _ {N ^ {\prime}} (\mathbf {a}) _ {1} := H _ {N ^ {\prime}} \left(\sum_ {k \leq N} a _ {k} \varphi_ {k}\right) _ {1} \in L ^ {2} (D) ^ {d _ {i n}},
$$
$$
H _ {N ^ {\prime}} (\mathbf {a}) _ {2} := H _ {N ^ {\prime}} \left(\sum_ {k \leq N ^ {\prime}} a _ {k} \varphi_ {k}\right) _ {2} \in L ^ {2} (D) ^ {d _ {o u t}},
$$
where  $a_{k}\in \mathbb{R}^{d_{in}}$ ,  $\mathbf{a} = (a_1,\dots,a_N)\in \mathbb{R}^{Nd_{in}}$ . Since  $H_{N^{\prime}}:L^{2}(D)^{d_{in}}\to L^{2}(D)^{d_{in} + d_{out}}$  is Lipschitz continuous,  $\mathbf{H}_{N^{\prime}}:\mathbb{R}^{Nd_{in}}\to \mathbb{R}^{N^{\prime}d_{out}}$  is also Lipschitz continuous. As
$$
N d _ {i n} + N ^ {\prime} d _ {o u t} > N ^ {\prime} d _ {o u t} \geq 2 N d _ {i n} + 1,
$$
we can apply Lemma 29 of Puthawala et al. [2022a] with  $D = N d_{in} + N' d_{out}$ ,  $m = N' d_{out}$ ,  $n = N d_{in}$ . According to this lemma, there exists a  $N' d_{out}$ -dimensional linear subspace  $\mathbf{V}^{\perp}$  in  $\mathbb{R}^{N d_{in} + N' d_{out}}$  such that
$$
\left\| P _ {\mathbf {V} ^ {\perp}} - P _ {\mathbf {V} _ {0} ^ {\perp}} \right\| _ {o p} <   \min  \left(\frac {\epsilon}{1 5 C _ {H _ {N}}}, 1\right) =: \epsilon_ {0}
$$
$$
P _ {\mathbf {V} ^ {\perp}} \circ \mathbf {H} _ {N ^ {\prime}}: \mathbb {R} ^ {N d _ {i n}} \rightarrow \mathbb {R} ^ {N d _ {i n} + N ^ {\prime} d _ {o u t}},
$$
is injective, where  $\mathbf{V}_{0}^{\perp} = \{0\}^{Nd_{in}} \times \mathbb{R}^{N'd_{out}}$ . Furthermore, in the proof of Theorem 15 of Puthawala et al. [2022a], denoting by
$$
\mathbf {B} := \pi_ {N ^ {\prime} d _ {o u t}} \circ \mathbf {Q} \circ P _ {\mathbf {V} ^ {\perp}},
$$
we can show that
$$
\mathbf {B} \circ \mathbf {H} _ {N ^ {\prime}}: \mathbb {R} ^ {N d _ {i n}} \rightarrow \mathbb {R} ^ {N ^ {\prime} d _ {o u t}},
$$
is injective, where  $\pi_{N'd_{out}}: \mathbb{R}^{Nd_{in} + N'd_{out}} \to \mathbb{R}^{N'd_{out}}$
$$
\pi_ {N ^ {\prime} d _ {o u t}} (a, b) := b, \quad (a, b) \in \mathbb {R} ^ {N d _ {i n}} \times \mathbb {R} ^ {N ^ {\prime} d _ {o u t}},
$$
and  $\mathbf{Q}:\mathbb{R}^{Nd_{in} + N'd_{out}}\to \mathbb{R}^{Nd_{in} + N'd_{out}}$  is defined by
By the same argument in proof of Theorem 15 in Puthawala et al. [2022a], we can show that
$$
\| I - \mathbf {Q} \| _ {\mathrm {o p}} \leq 4 \epsilon_ {0}.
$$
We define  $B:L^{2}(D)^{d_{in} + d_{out}}\to L^{2}(D)^{d_{out}}$
$\mathbf{B}_{k,p} \in \mathbb{R}^{d_{out} \times (d_{in} + d_{out})}$ ,  $\mathbf{B} = (\mathbf{B}_{k,p})_{k,p \in [N']} \text{ then } B: L^2(D)^{d_{in} + d_{out}} \to L^2(D)^{d_{out}}$  is a linear finite rank operator with  $N'$  rank. Then,
$$
G _ {N ^ {\prime}} := B \circ H _ {N ^ {\prime}}: L ^ {2} (D) ^ {d _ {i n}} \to L ^ {2} (D) ^ {d _ {o u t}},
$$
is injective because by the construction, it is equivalent to
$$
\mathbf {B} \circ \mathbf {H} _ {N ^ {\prime}}: \mathbb {R} ^ {N d _ {i n}} \to \mathbb {R} ^ {N ^ {\prime} d _ {o u t}},
$$
is injective. Furthermore, we have
$$
G _ {N ^ {\prime}} \in N O _ {L, N ^ {\prime}} (\sigma ; D, d _ {i n}, d _ {o u t}).
$$