SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Giovanni S. Alberti, Matteo Santacesaria, and Silvia Scutto. Continuous generative neural networks. arXiv:2205.14627, 2022.
Jose Antonio Lara Benitez, Takashi Furuya, Florian Faucher, Xavier Tricoche, and Maarten V de Hoop. Fine-tuning neural-operator architectures for training and generalization. arXiv preprint arXiv:2301.11509, 2023.
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B Kovachki, and Andrew M Stuart. Model reduction and neural networks for parametric pdes. The SMAI journal of computational mathematics, 7: 121-157, 2021.
Edward B Burger and Michael Starbird. The heart of mathematics: An invitation to effective thinking. Springer Science & Business Media, 2004.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jorn-Henrik Jacobsen. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
Yeol Je Cho and Yu-Qing Chen. Topological degree theory and applications. CRC Press, 2006.
Maarten De Hoop, Daniel Zhengyu Huang, Elizabeth Qian, and Andrew M Stuart. The cost-accuracy trade-off in operator learning with neural networks. arXiv preprint arXiv:2203.13181, 2022.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.
David Gilbarg and Neil S. Trudinger. Elliptic partial differential equations of second order. Classics in Mathematics. Springer-Verlag, Berlin, 2001. ISBN 3-540-41160-7. Reprint of the 1998 edition.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. Advances in neural information processing systems, 30, 2017.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 646-661. Springer, 2016.
Isao Ishikawa, Takeshi Teshima, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Universal approximation property of invertible neural networks. arXiv preprint arXiv:2204.07415, 2022.
Aref Jeribi. Spectral theory and applications of linear operators and block operator matrices, volume 9. Springer, 2015.
Tosio Kato. *Perturbation theory for linear operators*, volume 132. Springer Science & Business Media, 2013.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improving variational inference with inverse autoregressive flow. arXiv preprint arXiv:1606.04934, 2016.
Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling-based invertible neural networks are universal diffeomorphism approximators. Advances in Neural Information Processing Systems, 33:3362-3373, 2020.
Tapas Tripura and Souvik Chakraborty. Wavelet neural operator for solving parametric partial differential equations in computational mechanics problems. Computer Methods in Applied Mechanics and Engineering, 404:115783, 2023.
A Motivation behind our injectivity & bijectivity
PDEs-based inverse problems: We consider the following partial differential equations (PDEs) of the form
$$
\mathcal {L} _ {a} u (x) = f (x), \quad x \in D,
$$
$$
\mathcal {B} u (x) = g (x), \quad x \in \partial D,
$$
where  $D \subset \mathbb{R}^d$  is a bounded domain,  $\mathcal{L}_a$  is partial differential operator with a coefficient  $a \in \mathcal{A}(D; \mathbb{R}^{d_a}) \coloneqq \{a : D \to \mathbb{R}^{d_a}\}$ , and  $\mathcal{B}$  is some boundary operator, e.g. Dirichlet or Neumann, and function  $f \& g$  are fixed. When a solution  $u \in \mathcal{U}(D; \mathbb{R}^{d_u}) \coloneqq \{u : D \to \mathbb{R}^{d_a}\}$  is uniquely determined, we may define the solution operator  $G: \mathcal{A}(D; \mathbb{R}^{d_a}) \to \mathcal{U}(D; \mathbb{R}^{d_u})$  by putting  $G(a) \coloneqq u$ . Note that the operator  $G$  is, in general, non-linear even if partial differential Operator  $\mathcal{L}_a$  is linear (e.g.,  $\mathcal{L}_a = \Delta + a$ ). The aim of the inverse problem is to evaluate  $G^{-1}$ , the inverse of the solution operator. When  $G$  is non-injective, the problem is termed ill-posed.
The key link between PDE and our injective neural operators is as follows. Can  $G$  be approximated by injective neural operators  $N_{inj}$ ? If so, then  $N_{inj}$  can be a surrogate model to solve an ill-posed inverse problem. In general, this is possible, even if  $G$  is non-injective, per the results in Section 3.
Moreover, by the results of Section 4, we may conduct the inverse  $N_{inj}^{-1}$  of surrogate model  $N_{inj}$  with another neural operator.
Approximation of pushes forward measure: Let  $\mathcal{M}$  be the submanifold of  $X = L^{2}([0,1]^{2})$  or  $X = L^{2}([0,1]^{3})$  corresponding to natural images or 3D medical model. Let  $K\subset \mathbb{R}^D$  be a manifold with the same topology as  $\mathcal{M}$ ,  $\iota :\mathbb{R}^D\to X$  be an embedding, and let  $K_{1} = \iota (K)\subset \mathcal{M}$ . Given  $\mu$ , a measure supported on  $M$ , the task is to find a neural operator  $f_{\theta}\colon X\to X$  that maps (pushes forward) the uniform distribution on the model space  $K_{1}$  to  $\mu$  and so thus maps  $K_{1}$  to  $\mathcal{M}$ . If  $f_{\theta}\colon X\to X$  is bijective, computing likelihood functions in statistical analysis is made easier via the change of variables formula. Further, we may interpret  $f_{\theta}^{-1}$  as an encoder and  $f_{\theta}$  as the corresponding decoder, which parameterized elements of  $\mathcal{M}$ . As everything is formulated in infinite dimension function space  $X$ , we obtain discretization invariant methods.
B Proof of Proposition 1 in Section 2
Proof. We use the notation  $T|_{S(v,T + b)}(v) = (T_i v)_{i \in S(v,T + b)}$ . Assume that  $T + b$  has a DSS with respect to every  $v \in L^2(D)^n$  in the sense of Definition 2, and that
$$
\operatorname {R e L U} \left(T v ^ {(1)} + b\right) = \operatorname {R e L U} \left(T v ^ {(2)} + b\right) \quad \text {i n} D, \tag {B.1}
$$
where  $v^{(1)}, v^{(2)} \in L^2(D)^n$ . Since  $T + b$  has a DSS with respect to  $v^{(1)}$ , we have for  $i \in S(v^{(1)}, T + b)$
$$
0 <   \operatorname {R e L U} \left(T _ {i} v ^ {(1)} + b _ {i}\right) = \operatorname {R e L U} \left(T _ {i} v ^ {(2)} + b _ {i}\right) \text {i n} D,
$$
$$
T _ {i} v ^ {(1)} + b _ {i} = T _ {i} v ^ {(2)} + b _ {i} \text {i n} D.
$$
$$
v ^ {(1)} - v ^ {(2)} \in \operatorname {K e r} \left(T \mid_ {S (v ^ {(1)}, T + b)}\right). \tag {B.2}
$$
By assuming (B.1), we have for  $i \notin S(v^{(1)}, T)$
$$
\{x \in D \mid T _ {i} v ^ {(1)} (x) + b _ {i} (x) \leq 0 \} = \{x \in D \mid T _ {i} v ^ {(2)} (x) + b _ {i} (x) \leq 0 \}.
$$
$$
T _ {i} (v ^ {(1)} - (v ^ {(1)} - v ^ {(2)})) (x) + b _ {i} (x) = T _ {i} v ^ {(2)} (x) + b _ {i} (x) \leq 0 \text {i f} T _ {i} v ^ {(1)} (x) + b _ {i} (x) \leq 0,
$$
$$
T _ {i} v ^ {(1)} (x) + b _ {i} (x) \leq T _ {i} \left(v ^ {(1)} - v ^ {(2)}\right) (x) \text {i f} T _ {i} v ^ {(1)} (x) + b _ {i} (x) \leq 0.
$$
$$
T _ {i} (v ^ {(1)} - v ^ {(2)}) (x) = T _ {i} v ^ {(1)} (x) + b _ {i} (x) - \Big (T v ^ {(2)} (x) + b _ {i} (x) \Big) = 0 \mathrm {i f} T _ {i} v ^ {(1)} (x) + b _ {i} (x) > 0.
$$
$$
v ^ {(1)} - v ^ {(2)} \in X (v, T + b). \tag {B.3}
$$
Combining (B.2) and (B.3), and (2.1) as  $v = v^{(1)}$ , we conclude that
$$
v ^ {(1)} - v ^ {(2)} = 0.
$$
Conversely, assume that there exists a  $v \in L^{2}(D)^{n}$  such that
$$
\operatorname {K e r} \left(T \big | _ {S (v, T + b)}\right) \cap X (v, T + b) \neq \{0 \}.
$$
Then there is  $u \neq 0$  such that
$$
u \in \operatorname {K e r} \left. \left(T \right| _ {S (v, T + b)}\right) \cap X (v, T + b).
$$
For  $i\in S(v,T + b)$  , we have by  $u\in \mathrm{Ker}(T_i)$
$$
\operatorname {R e L U} \left(T _ {i} (v - u) + b _ {i} (x)\right) = \operatorname {R e L U} \left(T _ {i} v + b _ {i} (x)\right).
$$
For  $i \notin S(v, T + b)$ , we have by  $u \in X(v, T + b)$
$$
\begin{array}{l} \operatorname {R e L U} \left(T _ {i} (v - u) (x) + b _ {i} (x)\right) = \left\{ \begin{array}{l l} 0 & \text {i f} T _ {i} v (x) + b _ {i} (x) \leq 0 \\ T _ {i} v (x) + b _ {i} (x) & \text {i f} T _ {i} v (x) + b _ {i} (x) > 0 \end{array} \right. \\ = \operatorname {R e L U} \left(T _ {i} v (x) + b _ {i} (x)\right). \\ \end{array}
$$
$$
\operatorname {R e L U} (T (v - u) + b) = \operatorname {R e L U} (T v + b),
$$
where  $u\neq 0$  , that is,  $\mathrm{ReLU}\circ (T + b)$  is not injective.
C Details in Sections 3.1 and 3.2
Proof. The restriction operator,  $\pi_{\ell}: L^{2}(D)^{m} \to L^{2}(D)^{\ell} (\ell < m)$ , acts as follows,
$$
\pi_ {\ell} (a, b) := b, \quad (a, b) \in L ^ {2} (D) ^ {m - \ell} \times L ^ {2} (D) ^ {\ell}. \tag {C.1}
$$
Since  $L^2 (D)$  is a separable Hilbert space, there exists an orthonormal basis  $\{\varphi_k\}_{k\in \mathbb{N}}$  in  $L^2 (D)$ . We denote by
$$
\varphi_ {k, j} ^ {0} := \left(0, \ldots , 0, \underbrace {\varphi_ {k}} _ {j - t h}, 0, \ldots , 0\right) \in L ^ {2} (D) ^ {m},
$$
for  $k \in \mathbb{N}$  and  $j \in [m - \ell]$ . Then,  $\{\varphi_{k,j}^{0}\}_{k \in \mathbb{N}, j \in [m - \ell]}$  is an orthonormal sequence in  $L^{2}(D)^{m}$ , and
$$
\begin{array}{l} V _ {0} := L ^ {2} (D) ^ {m - \ell} \times \{0 \} ^ {\ell} \\ = \operatorname {s p a n} \left\{\varphi_ {k, j} ^ {0} \mid k \in \mathbb {N}, j \in [ m - \ell ] \right\}. \\ \end{array}
$$
We define, for  $\alpha \in (0,1)$
$$
\varphi_ {k, j} ^ {\alpha} := \left(0, \dots , 0, \underbrace {\sqrt {(1 - \alpha)} \varphi_ {k}} _ {j - t h}, 0, \dots , 0, \sqrt {\alpha} \xi_ {(k - 1) (m - \ell) + j}\right) \in L ^ {2} (D) ^ {m}, \tag {C.2}
$$
with  $k \in \mathbb{N}$  and  $j \in [m - \ell]$ . We note that  $\{\varphi_{k,j}^{\alpha}\}_{k \in \mathbb{N}, j \in [m - \ell]}$  is an orthonormal sequence in  $L^{2}(D)^{m}$ . We set
$$
V _ {\alpha} := \operatorname {s p a n} \left\{\varphi_ {k, j} ^ {\alpha} \mid k \in \mathbb {N}, j \in [ m - \ell ] \right\}. \tag {C.3}
$$
It holds for  $0 < \alpha < 1/2$  that
$$
\left\| P _ {V _ {\alpha} ^ {\perp}} - P _ {V _ {0} ^ {\perp}} \right\| _ {\mathrm {o p}} <   1.
$$
Indeed, for  $u\in L^{2}(D)^{m}$  and  $0 < \alpha < 1 / 2$
$$
\begin{array}{l} \left\| P _ {V _ {\alpha} ^ {\perp}} u - P _ {V _ {0} ^ {\perp}} u \right\| _ {L ^ {2} (D) ^ {m}} ^ {2} = \| P _ {V _ {\alpha}} u - P _ {V _ {0}} u \| _ {L ^ {2} (D) ^ {m}} ^ {2} \\ = \left\| \sum_ {k \in \mathbb {N}, j \in [ m - \ell ]} (u, \varphi_ {k, j} ^ {\alpha}) \varphi_ {k, j} ^ {\alpha} - (u, \varphi_ {k, j} ^ {0}) \varphi_ {k, j} ^ {0} \right\| _ {L ^ {2} (D) ^ {m}} ^ {2} \\ = \left\| \sum_ {k \in \mathbb {N}, j \in [ m - \ell ]} (1 - \alpha) \left(u _ {j}, \varphi_ {k}\right) \varphi_ {k} - \left(u _ {j}, \varphi_ {k}\right) \varphi_ {k} \right\| _ {L ^ {2} (D)} ^ {2} \\ + \left\| \sum_ {k \in \mathbb {N}, j \in [ m - \ell ]} \alpha (u _ {m}, \xi_ {(k - 1) (m - \ell) + j}) \xi_ {(k - 1) (m - \ell) + j} \right\| _ {L ^ {2} (D)} ^ {2} \\ \leq \alpha^ {2} \sum_ {j \in [ m - \ell ]} \sum_ {k \in \mathbb {N}} | (u _ {j}, \varphi_ {k}) | ^ {2} + \alpha^ {2} \sum_ {k \in \mathbb {N}} | (u _ {m}, \xi_ {k}) | ^ {2} \leq 4 \alpha^ {2} \| u \| _ {L ^ {2} (D) ^ {m}} ^ {2}, \\ \end{array}
$$
which implies that  $\left\| P_{V_{\alpha}^{\perp}} - P_{V_0^{\perp}}\right\|_{\mathrm{op}}\leq 2\alpha .$
We will show that the operator
$$
P _ {V _ {\alpha} ^ {\perp}} \circ T: L ^ {2} (D) ^ {n} \to L ^ {2} (D) ^ {m},
$$
is injective. Assuming that for  $a, b \in L^2(D)^n$ ,
$$
P _ {V _ {\alpha} ^ {\perp}} \circ T (a) = P _ {V _ {\alpha} ^ {\perp}} \circ T (b),
$$
$$
T (a) - T (b) = P _ {V _ {\alpha}} (T (a) - T (b)).
$$
Denoting by  $P_{V_{\alpha}}(T(a) - T(b)) = \sum_{k\in \mathbb{N},j\in [m - \ell ]}c_{k,j}\varphi_{k,j}^{\alpha}$
$$
\pi_ {1} (T (a) - T (b)) = \sum_ {k \in \mathbb {N}, j \in [ m - \ell ]} c _ {k, j} \xi_ {(k - 1) (m - \ell) + j}.
$$
From (3.1), we obtain that  $c_{kj} = 0$  for all  $k,j$ . By injectivity of  $T$ , we finally get  $a = b$ .
We define  $Q_{\alpha}:L^{2}(D)^{m}\to L^{2}(D)^{m}$  by
$$
Q _ {\alpha} := \left(P _ {V _ {0} ^ {\perp}} P _ {V _ {\alpha} ^ {\perp}} + (I - P _ {V _ {0} ^ {\perp}}) (I - P _ {V _ {\alpha} ^ {\perp}})\right) \left(I - (P _ {V _ {0} ^ {\perp}} - P _ {V _ {\alpha} ^ {\perp}}) ^ {2}\right) ^ {- 1 / 2}.
$$
By the same argument as in Section I.4.6 Kato [2013], we can show that  $Q_{\alpha}$  is injective and
$$
Q _ {\alpha} P _ {V _ {\alpha} ^ {\perp}} = P _ {V _ {0} ^ {\perp}} Q _ {\alpha},
$$
that is,  $Q_{\alpha}$  maps from  $\mathrm{Ran}(P_{V_{\alpha}^{\perp}})$  to
$$
\operatorname {R a n} \left(P _ {V _ {0} ^ {\perp}}\right) \subset \{0 \} ^ {m - \ell} \times L ^ {2} (D) ^ {\ell}.
$$
$$
\pi_ {\ell} \circ Q _ {\alpha} \circ P _ {V _ {\alpha} ^ {\perp}} \circ T: L ^ {2} (D) ^ {n} \to L ^ {2} (D) ^ {\ell}
$$
is injective.
C.2 Remarks following Lemma 1
Remark 3. Lemma 1 and Eqn. (3.1) may be interpreted as saying that if some orthonormal sequence  $\{\xi_k\}_{k\in \mathbb{N}}$  exists that doesn't overlap the range of  $T$ , then  $T$  may be embedded in a small space without losing injectivity. An example that satisfies (3.1) is the neural operator whose  $L$ -th layer operator  $\mathcal{L}_L$  consists of the integral operator  $K_{L}$  with continuous kernel function  $k_{L}$ , and with continuous activation function  $\sigma$ . Indeed, in this case, we may choose the orthogonal sequence  $\{\xi_k\}_{k\in \mathbb{N}}$  in  $L^2(D)$  as a discontinuous function sequence  $^1$  so that  $\operatorname{span}\{\xi_k\}_{k\in \mathbb{N}} \cap C(D) = \{0\}$ . Then, by  $\mathrm{Ran}(\mathcal{L}_L) \subset C(D)^{d_L}$ , the assumption (3.1) holds.
Remark 4. In the proof of Lemma 1, an operator  $B \in \mathcal{L}(L^2(D)^m, L^2(D)^\ell)$ ,
$$
B = \pi_ {\ell} \circ Q _ {\alpha} \circ P _ {V _ {\alpha} ^ {- 1}},
$$
appears, where  $P_{V_\alpha^\perp}$  is the orthogonal projection onto orthogonal complement  $V_\alpha^\perp$  of  $V_\alpha$  with
$$
V _ {\alpha} := \operatorname {s p a n} \left\{\varphi_ {k, j} ^ {\alpha} \Big | k \in \mathbb {N}, j \in [ m - \ell ] \right\} \subset L ^ {2} (D) ^ {m},
$$
in which  $\varphi_{k,j}^{\alpha}$  is defined for  $\alpha \in (0,1)$ ,  $k\in \mathbb{N}$  and  $j\in [\ell ]$
$$
\varphi_ {k, j} ^ {\alpha} := \left(0, \dots , 0, \underbrace {\sqrt {(1 - \alpha)} \varphi_ {k}} _ {j - t h}, 0, \dots , 0, \sqrt {\alpha} \xi_ {(k - 1) (m - \ell) + j}\right).
$$
Here,  $\{\varphi_k\}_{k\in \mathbb{N}}$  is an orthonormal basis in  $L^2 (D)$ . Furthermore,  $Q_{\alpha}:L^{2}(D)^{m}\to L^{2}(D)^{m}$  is defined by
$$
Q _ {\alpha} := \left(P _ {V _ {0} ^ {\perp}} P _ {V _ {\alpha} ^ {\perp}} + (I - P _ {V _ {0} ^ {\perp}}) (I - P _ {V _ {\alpha} ^ {\perp}})\right) \left(I - (P _ {V _ {0} ^ {\perp}} - P _ {V _ {\alpha} ^ {\perp}}) ^ {2}\right) ^ {- 1 / 2},
$$
where  $P_{V_0^\perp}$  is the orthogonal projection onto orthogonal complement  $V_0^\perp$  of  $V_0$  with
$$
V _ {0} := L ^ {2} (D) ^ {m - \ell} \times \{0 \} ^ {\ell}.
$$
The operator  $Q_{\alpha}$  is well-defined for  $0 < \alpha < 1/2$  because it holds that
$$
\left\| P _ {V _ {\alpha} ^ {\perp}} - P _ {V _ {0} ^ {\perp}} \right\| _ {\mathrm {o p}} <   2 \alpha .
$$
This construction is given by the combination of "Pairs of projections" discussed in Kato [2013, Section I.4.6] with the idea presented in [Puthawala et al., 2022b, Lemma 29].
We begin with
Definition 3. The set of  $L$ -layer neural networks mapping from  $\mathbb{R}^d$  to  $\mathbb{R}^{d'}$  is
$$
\mathrm {N} _ {L} (\sigma ; \mathbb {R} ^ {d}, \mathbb {R} ^ {d ^ {\prime}}) := \left\{f: \mathbb {R} ^ {d} \rightarrow \mathbb {R} ^ {d ^ {\prime}} \middle | f (x) = W _ {L} \sigma \left(\dots W _ {1} \sigma \left(W _ {0} x + b _ {0}\right) + b _ {1} \dots\right) + b _ {L}, \right.
$$
$$
\left. W _ {\ell} \in \mathbb {R} ^ {d _ {\ell + 1} \times d _ {\ell}}, b _ {\ell} \in \mathbb {R} ^ {d _ {\ell + 1}}, d _ {\ell} \in \mathbb {N} _ {0} \left(d _ {0} = d, d _ {L + 1} = d ^ {\prime}\right), \ell = 0, \dots , L \right\},
$$
where  $\sigma : \mathbb{R} \to \mathbb{R}$  is an element-wise nonlinear activation function. For the class of nonlinear activation functions,
$$
\mathrm {A} _ {0} := \left\{\sigma \in C (\mathbb {R}) \middle | \exists n \in \mathbb {N} _ {0} s. t. \mathrm {N} _ {n} (\sigma ; \mathbb {R} ^ {d}, \mathbb {R}) i s d e n s e i n C (K) f o r \forall K \subset \mathbb {R} ^ {d} c o m p a c t \right\}
$$
$$
\mathrm {A} _ {0} ^ {L} := \left\{\sigma \in A _ {0} \mid \sigma \text {i s B o r e l m e a s u r a b l e s . t .} \sup  _ {x \in \mathbb {R}} \frac {| \sigma (x) |}{1 + | x |} <   \infty \right\}
$$
e.g., step functions whose supports are disjoint for each sequence.
$\operatorname{BA} := \left\{\sigma \in A_0 \middle| \forall K \subset \mathbb{R}^d \text{ compact}, \forall \epsilon > 0, \text{ and } \forall C \geq \operatorname{diam}(K), \exists n \in \mathbb{N}_0, \right.$
$$
\left. \exists f \in \mathrm {N} _ {n} \left(\sigma ; \mathbb {R} ^ {d}, \mathbb {R} ^ {d}\right) \text {s . t .} | f (x) - x | \leq \epsilon , \forall x \in K, a n d, | f (x) | \leq C, \forall x \in \mathbb {R} ^ {d} \right\}.
$$
The set of integral neural operators with  $L^2$ -integral kernels is
$$
\mathrm {N O} _ {L} (\sigma ; D, d _ {i n}, d _ {o u t}) := \left\{ \right.G: L ^ {2} (D) ^ {d _ {i n}} \rightarrow L ^ {2} (D) ^ {d _ {o u t}} \left. \right|
$$
$$
G = K _ {L + 1} \circ (K _ {L} + b _ {L}) \circ \sigma \dots \circ (K _ {2} + b _ {2}) \circ \sigma \circ (K _ {1} + b _ {1}) \circ (K _ {0} + b _ {0}),
$$
$$
K _ {\ell} \in \mathcal {L} \left(L ^ {2} (D) ^ {d _ {\ell}}, L ^ {2} (D) ^ {d _ {\ell + 1}}\right), K _ {\ell}: f \mapsto \int_ {D} k _ {\ell} (\cdot , y) f (y) d y \Bigg | _ {D}, \tag {C.4}
$$
$$
k _ {\ell} \in L ^ {2} (D \times D; \mathbb {R} ^ {d _ {\ell + 1} \times d _ {\ell}}), b _ {\ell} \in L ^ {2} (D; \mathbb {R} ^ {d _ {\ell + 1}}),
$$
$$
\left. d _ {\ell} \in \mathbb {N}, d _ {0} = d _ {i n}, d _ {L + 2} = d _ {o u t}, \ell = 0, \dots , L + 2 \right\}.
$$
Proof. Let  $R > 0$  such that
$$
K \subsetneq B _ {R} (0),
$$
where  $B_R(0) \coloneqq \{u \in L^2(D)^{d_{in}} \mid \| u \|_{L^2(D)^{d_{in}}} \leq R\}$ . By Theorem 11 of Kovachki et al. [2021b], there exists  $L \in \mathbb{N}$  and  $\widetilde{G} \in \mathrm{NO}_L(\sigma; D, d_{in}, d_{out})$  such that
$$
\sup  _ {a \in K} \left\| G ^ {+} (a) - \widetilde {G} (a) \right\| _ {L ^ {2} (D) ^ {d o u t}} \leq \frac {\epsilon}{2}, \tag {C.5}
$$
$$
\left\| \widetilde {G} (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \leq 4 M, \quad \text {f o r} a \in L ^ {2} (D) ^ {d _ {i n}}, \quad \| a \| _ {L ^ {2} (D) ^ {d _ {i n}}} \leq R.
$$
We write operator  $\widetilde{G}$  by
$$
\widetilde {G} = \widetilde {K} _ {L + 1} \circ (\widetilde {K} _ {L} + \widetilde {b} _ {L}) \circ \sigma \dots \circ (\widetilde {K} _ {2} + \widetilde {b} _ {2}) \circ \sigma \circ (\widetilde {K} _ {1} + \widetilde {b} _ {1}) \circ (\widetilde {K} _ {0} + \widetilde {b} _ {0}),
$$
$$
\widetilde {K} _ {\ell} \in \mathcal {L} \left(L ^ {2} (D) ^ {d _ {\ell}}, L ^ {2} (D) ^ {d _ {\ell + 1}}\right), \widetilde {K} _ {\ell}: f \mapsto \int_ {D} \widetilde {k} _ {\ell} (\cdot , y) f (y) d y,
$$
$$
\widetilde {k} _ {\ell} \in C (D \times D; \mathbb {R} ^ {d _ {\ell + 1} \times d _ {\ell}}), \widetilde {b} _ {\ell} \in L ^ {2} (D; \mathbb {R} ^ {d _ {\ell + 1}}),
$$
$$
d _ {\ell} \in \mathbb {N}, d _ {0} = d _ {i n}, d _ {L + 2} = d _ {o u t}, \ell = 0, \dots , L + 2.
$$
We remark that kernel functions  $\widetilde{k}_{\ell}$  are continuous because neural operators defined in Kovachki et al. [2021b] parameterize the integral kernel function by neural networks, thus,
$$
\operatorname {R a n} (\widetilde {G}) \subset C (D) ^ {d _ {\text {o u t}}}. \tag {C.6}
$$
We define the neural operator  $H: L^2(D)^{d_{in}} \to L^2(D)^{d_{in} + d_{out}}$  by
$$
H = K _ {L + 1} \circ \left(K _ {L} + b _ {L}\right) \circ \sigma \dots \circ \left(K _ {2} + b _ {2}\right) \circ \sigma \circ \left(K _ {1} + b _ {1}\right) \circ \left(K _ {0} + b _ {0}\right),
$$
where  $K_{\ell}$  and  $b_{\ell}$  are defined as follows. First, we choose  $K_{inj} \in \mathcal{L}(L^{2}(D)^{d_{in}}, L^{2}(D)^{d_{in}})$  as a linear injective integral operator<sup>2</sup>.
(i) When  $\sigma_{1}\in \mathrm{A}_{0}^{L}\cap \mathrm{BA}$  is injective,
$$
K _ {0} = \left( \begin{array}{c} K _ {i n j} \\ \tilde {K} _ {0} \end{array} \right) \in \mathcal {L} (L ^ {2} (D) ^ {d _ {i n}}, L ^ {2} (D) ^ {d _ {i n} + d _ {1}}), \quad b _ {0} = \left( \begin{array}{c} O \\ \tilde {b} _ {0} \end{array} \right) \in L ^ {2} (D) ^ {d _ {i n} + d _ {1}},
$$
2For example, if we choose the integral kernel  $k_{inj}$  as  $k_{inj}(x,y) = \sum_{k = 1}^{\infty}\vec{\varphi}_k(x)\vec{\varphi}_k(y)$ , then the integral operator  $K_{\mathrm{inj}}$  with the kernel  $k_{\mathrm{inj}}$  is injective where  $\{\vec{\varphi}\}_{k}$  is the orthonormal basis in  $L^2 (D)^{d_{in}}$ .
$$
\begin{array}{l} K _ {\ell} = \left( \begin{array}{c c} K _ {i n j} & O \\ O & \widetilde {K} _ {\ell} \end{array} \right) \in \mathcal {L} (L ^ {2} (D) ^ {d _ {i n} + d _ {\ell}}, L ^ {2} (D) ^ {d _ {i n} + d _ {\ell + 1}}), \quad b _ {\ell} = \left( \begin{array}{c} O \\ \widetilde {b} _ {\ell} \end{array} \right) \in L ^ {2} (D) ^ {d _ {i n} + d _ {\ell + 1}}, \\ (1 \leq \ell \leq L), \\ \end{array}
$$
$$
K _ {L + 1} = \left( \begin{array}{c c} K _ {i n j} & O \\ O & \widetilde {K} _ {L + 1} \end{array} \right) \in \mathcal {L} (L ^ {2} (D) ^ {d _ {i n} + d _ {L + 1}}, L ^ {2} (D) ^ {d _ {i n} + d _ {o u t}}), \quad b _ {\ell} = \left( \begin{array}{c} O \\ O \end{array} \right) \in L ^ {2} (D) ^ {d _ {i n} + d _ {o u t}}.
$$
(ii) When  $\sigma_{1} = \mathrm{ReLU}$
$$
\begin{array}{l} K _ {0} = \left( \begin{array}{c} K _ {i n j} \\ \widetilde {K} _ {0} \end{array} \right) \in \mathcal {L} (L ^ {2} (D) ^ {d _ {i n}}, L ^ {2} (D) ^ {d _ {i n} + d _ {1}}), \quad b _ {0} = \left( \begin{array}{c} O \\ \widetilde {b} _ {0} \end{array} \right) \in L ^ {2} (D) ^ {d _ {i n} + d _ {1}}, \\ K _ {1} = \left( \begin{array}{c c} K _ {i n j} & O \\ - K _ {i n j} & O \\ O & \widetilde {K} _ {1} \end{array} \right) \in \mathcal {L} (L ^ {2} (D) ^ {d _ {i n} + d _ {1}}, L ^ {2} (D) ^ {2 d _ {i n} + d _ {2}}), b _ {0} = \left( \begin{array}{c} O \\ O \\ \widetilde {b} _ {1} \end{array} \right) \in L ^ {2} (D) ^ {2 d _ {i n} + d _ {1}}, \\ K _ {\ell} = \left( \begin{array}{c c c c} K _ {i n j} & - K _ {i n j} & O \\ - K _ {i n j} & - K _ {i n j} & - \\ \hline O & - K _ {\ell} \end{array} \right) \in \mathcal {L} (L ^ {2} (D) ^ {2 d _ {i n} + d _ {\ell}}, L ^ {2} (D) ^ {2 d _ {i n} + d _ {\ell + 1}}), \\ b _ {\ell} = \left( \begin{array}{c} O \\ O \\ \widetilde {b} _ {\ell} \end{array} \right) \in L ^ {2} (D) ^ {2 d _ {i n} + d _ {\ell + 1}}, \quad (2 \leq \ell \leq L), \\ \begin{array}{c}\vdots \\ K_{L} = \left( \begin{array}{cc} - \frac{K_{inj}}{-} & -\frac{K_{inj}}{-}\\ \widetilde{O} \end{array} \right)^{\prime}\left( \begin{array}{c}O\\ \widetilde{K}_{L} \end{array} \right)\\ \in \mathcal{L}(L^{2}(D)^{2d_{in} + d_{L}},L^{2}(D)^{d_{in} + d_{L + 1}}), \end{array} \\ b _ {L} = \left( \begin{array}{c} O \\ \widetilde {b} _ {L} \end{array} \right) \in L ^ {2} (D) ^ {d _ {i n} + d _ {L + 1}}, \\ K _ {L + 1} = \left( \begin{array}{c c} K _ {i n j} & O \\ O & \widetilde {K} _ {L + 1} \end{array} \right) \in \mathcal {L} (L ^ {2} (D) ^ {d _ {i n} + d _ {L + 1}}, L ^ {2} (D) ^ {d _ {i n} + d _ {o u t}}), \\ b _ {L + 1} = \left( \begin{array}{c} O \\ O \end{array} \right) \in L ^ {2} (D) ^ {d _ {i n} + d _ {o u t}}. \\ \end{array}
$$
Then, the operator  $H: L^2(D)^{d_{in}} \to L^2(D)^{d_{in} + d_{out}}$  has the form of
$$
H := \left\{ \begin{array}{l l} \left( \begin{array}{c} K _ {i n j} \circ K _ {i n j} \circ \sigma \circ K _ {i n j} \circ \dots \circ \sigma \circ K _ {i n j} \circ K _ {i n j} \\ \widetilde {G} \end{array} \right) & \text {i n t h e c a s e o f (i)}. \\ \left( \begin{array}{c} K _ {i n j} \circ \dots \circ K _ {i n j} \\ \widetilde {G} \end{array} \right) & \text {i n t h e c a s e o f (i i)}. \end{array} \right.
$$
For the case of (ii), we have used the fact