SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This approach resembles the approach in the finite-rank space Puthawala et al. [2022a, Theorem 15], but unlike that theorem we don't have any dimensionality restrictions. More specifically, in the case of Euclidean spaces  $\mathbb{R}^d$ , Puthawala et al. [2022a, Theorem 15] requires that  $2d_{in} + 1 \leq d_{out}$  before all continuous functions  $G^{+}: \mathbb{R}^{d_{in}} \to \mathbb{R}^{d_{out}}$  can be uniformly approximated in compact sets by injective neural networks. When  $d_{in} = d_{out} = 1$  this result is not true, as is shown in Remark 5 (iii) in Appendix C using topological degree theory [Cho and Chen, 2006]. In contrast, Theorem 1 does not assume any conditions on  $d_{in}$  and  $d_{out}$ . Therefore, we can conclude that infinite dimensional case yields better approximation results than the finite dimensional case.
This surprising improvement in restrictions in infinite-dimensions can be elucidated by an analogy to Hilbert's hotel paradox, see [Burger and Starbird, 2004, Sec 3.2]. In this analogy, the orthonormal bases  $\{\varphi_k\}_{k\in \mathbb{N}}$  and  $\Psi_{j,k}(x) = (\delta_{ij}\varphi_k(x))_{i = 1}^d$  play the part of guests in the hotel with  $\mathbb{N}$  floor, each of which as  $d$  rooms. A key step in the proof of Theorem 1 is that there is a linear isomorphism  $S:L^{2}(D)^{d}\to L^{2}(D)$  (i.e., a rearrangement of guests) which maps  $\Psi_{j,k}$  to  $\varphi_{b(j,k)}$ , where  $b:$ $[d]\times \mathbb{N}\rightarrow \mathbb{N}$  is a bijection.
3.3 Injectivity-preserving transfer to Euclidean spaces via finite-rank approximation
In the previous section, we have discussed injective integral neural operators. The conditions are given in terms of integral kernels, but such kernels may not actually be implementable with finite width and depth networks, which have a finite representational power. A natural question to ask, therefore, is how should these formal integral kernels be implemented on actual finite rank networks, the so-called implementable neural operators? In this section we discuss this question.
We consider linear integral operators  $K_{\ell}$  with  $L^2$ -integral kernels  $k_{\ell}(x,y)$ . Let  $\{\varphi_k\}_{k\in \mathbb{N}}$  be an orthonormal basis in  $L^2 (D)$ . Since  $\{\varphi_k(y)\varphi_p(x)\}_{k,p\in \mathbb{N}}$  is an orthonormal basis of  $L^2 (D\times D)$ , integral kernels  $k_{\ell}\in L^{2}(D\times D;\mathbb{R}^{d_{\ell +1}\times d_{\ell}})$  in integral operators  $K_{\ell}\in \mathcal{L}(L^{2}(D)^{d_{\ell}},L^{2}(D)^{d_{\ell +1}})$  has the expansion
$$
k _ {\ell} (x, y) = \sum_ {k, p \in \mathbb {N}} C _ {k, p} ^ {(\ell)} \varphi_ {k} (y) \varphi_ {p} (x),
$$
where  $C_{k,p}^{(\ell)} \in \mathbb{R}^{d_{\ell + 1} \times d_{\ell}}$  whose  $(i, j)$ -th component  $c_{k,p,ij}^{(\ell)}$  is given by
$$
c _ {k, p, i j} ^ {(\ell)} = (k _ {\ell , i j}, \varphi_ {k} \varphi_ {p}) _ {L ^ {2} (D \times D)},
$$
Here, we denote  $(u,\varphi_k)\in \mathbb{R}^{d_\ell}$  by  $(u,\varphi_{k}) = \bigl ((u_{1},\varphi_{k})_{L^{2}(D)},\dots,(u_{d_{\ell}},\varphi_{k})_{L^{2}(D)}\bigr)$ . By truncating by  $N$  finite sums, we approximate  $L^2$ -integral operators  $K_{\ell}\in \mathcal{L}(L^{2}(D)^{d_{\ell}},L^{2}(D)^{d_{\ell +1}})$  by finite rank operator  $K_{\ell ,N}$  with rank  $N$ , having the form
$$
K _ {\ell , N} u (x) = \sum_ {k, p \in [ N ]} C _ {k, p} ^ {(\ell)} (u, \varphi_ {k}) \varphi_ {p} (x), u \in L ^ {2} (D) ^ {d _ {\ell}}.
$$
The choice of orthonormal basis  $\{\varphi_k\}_{k\in \mathbb{N}}$  is a hyperparameter. If we choose  $\{\varphi_k\}_k$  as Fourier basis and wavelet basis, then network architectures correspond to Fourier Neural Operators (FNOs) [Li et al., 2020b] and Wavelet Neural Operators (WNOs) [Tripura and Chakraborty, 2023], respectively. We show that Propositions 1, 2 (characterization of layerwise injectivity), and Lemma 1 (global injectivity) all have natural analogues for finite rank operator  $K_{\ell ,N}$  in Proposition 5 and Lemma 2 in Appendix D. These conditions apply out-of-the-box to both FNOs and WNOs.
Lemma 2 and Remark 3 in the appendix D give a 'recipe' to construct the projection  $B$  such that the composition  $B \circ T$  (interpreted as augmenting finite rank neural operator  $T$  with one layer  $B$ ) is injective. The projection  $B$  is constructed by using an orthogonal sequence  $\{\xi_k\}_{k \in \mathbb{N}}$  subject to the condition (3.1), which does not over-leap the range of  $T$ . This condition is automatically satisfied for any orthogonal base  $\{\varphi_k\}_{k \in \mathbb{N}}$ . This could yield practical implications in guiding the choice of the orthogonal basis  $\{\varphi_k\}_{k \in \mathbb{N}}$  for the neural operator's design.
We also show the universal approximation in the case of finite rank approximation. We denote  $\mathrm{NO}_{L,N}(\sigma;D,d_{in},d_{out})$  by the set of integral neural operators with  $N$  rank, that is the set (3.2) replacing  $L^2$ -integral kernel operators  $K_{\ell}$  with finite rank operators  $K_{\ell,N}$  with rank  $N$  (see Definition 4). We define by
$$
\mathrm {N O} _ {L, N} ^ {i n j} (\sigma ; D, d _ {i n}, d _ {o u t}) := \left\{G _ {N} \in \mathrm {N O} _ {L, N} (\sigma ; D, d _ {i n}, d _ {o u t}): G _ {N} \text {i s i n j e c t i v e} \right\}.
$$
Theorem 2. Let  $D \subset \mathbb{R}^d$  be a Lipschitz bounded domain, and  $N \in \mathbb{N}$ , and  $G^{+}: L^{2}(D)^{d_{in}} \to L^{2}(D)^{d_{out}}$  be continuous with boundedness as in (3.3). Assume that the non-linear activation function  $\sigma$  is either ReLU or Leaky ReLU. Then, for any compact set  $K \subset L^{2}(D)^{d_{in}} \cap (\mathrm{span}\{\varphi_{k}\}_{k \leq N})^{d_{in}}$ ,  $\epsilon \in (0,1)$ , there exists  $L \in \mathbb{N}$ ,  $N' \in \mathbb{N}$  with
$$
N ^ {\prime} d _ {o u t} \geq 2 N d _ {i n} + 1, \tag {3.4}
$$
and  $G_{N^{\prime}}\in \mathrm{NO}_{L,N^{\prime}}^{inj}(\sigma ;D,d_{in},d_{out})$  such that
$$
\sup  _ {a \in K} \left\| G ^ {+} (a) - G _ {N ^ {\prime}} (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \leq \epsilon .
$$
See Section D.4 in Appendix D for the proof. In the proof, we make use of Puthawala et al. [2022a, Lemma 29], which gives rise to the assumption (3.4). We do not require any condition on  $d_{in}$  and  $d_{out}$  as well as Theorem 1.
Remark 1. Observe that in our finite-rank approximation result, we only require that the target function  $G+$  is continuous and bounded, but not smooth. This differs from prior work that requires smoothness of the function to be approximated.
We assume square matrices for the bijectivity and construction of inversions. Weight matrices in actual neural operators are not necessarily square. Lifting and projection are local operators which map the inputs into a higher-dimensional feature space and project back the feature output to the output space. We also haven't addressed possible aliasing effects of our injective operators. We will relax the square assumption and investigate the aliasing of injective operators in future work.
4 Subnetworks & nonlinear integral operators: bijectivity and inversion
So far our analysis of injectivity has been restricted to the case where the only source of nonlinearity are the activation functions. In this section we consider a weaker and more abstract problem where nonlinearities can also arise from the integral kernel with surjective activation function, such as leaky ReLU. Specifically, we consider layers of the form
$$
F _ {1} (u) = W u + K (u), \tag {4.1}
$$
where  $W \in \mathcal{L}(L^2(D)^n, L^2(D)^n)$  is a linear bounded bijective operator, and  $K: L^2(D)^n \to L^2(D)^n$  is a non-linear operator. This arises in the non-linear neural operator construction by Kovachki et al. [2021b] or in Ong et al. [2022] to improve performance of integral autoencoders. In this construction, each layer  $\mathcal{L}_{\ell}$  is written as
$$
x \in D, \quad (\mathcal {L} _ {\ell} v) (x) = \sigma (W _ {\ell} v (x) + K _ {\ell} (v) (x)), \quad K _ {\ell} (u) (x) = \int_ {D} k _ {\ell} (x, y, u (x), u (y)) u (y) d y,
$$
where  $W_{\ell} \in \mathbb{R}^{d_{\ell +1} \times d_{\ell}}$  independent of  $x$ , and  $K_{\ell}: L^{2}(D)^{d_{\ell}} \to L^{2}(D)^{d_{\ell +1}}$  is the non-linear integral operator.
This relaxing of assumptions is motivated by a desire to obtain theoretical results for both subnetworks and operator transformers. By subnetworks, we mean compositions of layers within a network. This includes, for example, the encoder or decoder block of a traditional VAE. By neural operator we mean operator generalizations of finite-rank transformers, which can be modeled by letting  $K$  be an integral transformation with nonlinear kernel  $k$  of the form
$$
k (x, y, v (x), v (y)) \equiv \operatorname {s o f t m a x} \circ \left\langle A v (x), B v (y) \right\rangle ,
$$
where  $A$  and  $B$  are matrices of free parameters, and the (integral) softmax is taken over  $x$ . This specific choice of  $k$  can be understood as a natural generalization of the attention mechanism in transformers, see [Kovachki et al., 2021b, Sec. 5.2] for further details.
4.1 Surjectivity and bijectivity
Critical to our analysis is the notion of coercivity. Apart from being a useful theoretical tool, layerwise coercivity of neural networks is a useful property in imaging applications, see e.g.Li et al. [2020a] Recall from Showalter [2010, Sec 2, Chap VII] that a non-linear operator  $K: L^2(D)^n \to L^2(D)^n$  is coercive if
$$
\lim  _ {\| u \| _ {L ^ {2} (D) ^ {n}} \rightarrow \infty} \langle K (u), \frac {u}{\| u \| _ {L ^ {2} (D) ^ {n}}} \rangle_ {L ^ {2} (D) ^ {n}} = \infty . \tag {4.2}
$$
Proposition 3. Let  $\sigma : \mathbb{R} \to \mathbb{R}$  be surjective and  $W: L^2(D)^n \to L^2(D)^n$  be linear bounded bijective (then the inverse  $W^{-1}$  is bounded linear), and let  $K: L^2(D)^n \to L^2(D)^n$  be a continuous and compact mapping. Moreover, assume that the map  $u \mapsto \alpha u + W^{-1}K(u)$  is coercive with some  $0 < \alpha < 1$ . Then, the operator  $\sigma \circ F_1$  is surjective.
See Section E.1 in Appendix E for the proof. An example  $K$  satisfying the coercivity (4.2) condition is given here.
Example 1. We simply consider the case of  $n = 1$ , and  $D \subset \mathbb{R}^d$  is a bounded interval. We consider the non-linear integral operator
$$
K (u) (x) := \int_ {D} k (x, y, u (x)) u (y) d y, x \in D.
$$
The operator  $u \mapsto \alpha u + W^{-1}K(u)$  with some  $0 < \alpha < 1$  is coercive when the non-linear integral kernel  $k(x,y,t)$  satisfies certain boundedness conditions. In Examples 2 and 3 in Appendix E, we show that these conditions are met by kernels  $k(x,y,t)$  of the form
$$
k (x, y, t) = \sum_ {j = 1} ^ {J} c _ {j} (x, y) \sigma \left(a _ {j} (x, y) t + b _ {j} (x, y)\right),
$$
where  $\sigma : \mathbb{R} \to \mathbb{R}$  is the sigmoid function  $\sigma_s : \mathbb{R} \to \mathbb{R}$ , and  $a, b, c \in C(\overline{D} \times \overline{D})$  or by a wavelet activation function  $\sigma_{wire} : \mathbb{R} \to \mathbb{R}$ , see Saragadam et al. [2023], where  $\sigma_{wire}(t) = \operatorname{Im}(e^{i\omega t} e^{-t^2})$  and  $a, b, c \in C(\overline{D} \times \overline{D})$ , and  $a_j(x, y) \neq 0$ .
In the proof of Proposition 3, we utilize the Leray-Schauder fix point theorem. By employing the Banach fix point theorem under a contraction mapping condition (4.3), we can obtain bijectivity as follows:
Proposition 4. Let  $\sigma : \mathbb{R} \to \mathbb{R}$  be bijective. Let  $W: L^2(D)^n \to L^2(D)^n$  be bounded linear bijective, and let  $K: L^2(D)^n \to L^2(D)^n$ . If  $W^{-1}K: L^2(D)^n \to L^2(D)^n$  is a contraction mapping, that is, there exists  $\rho \in (0,1)$  such that
$$
\left\| W ^ {- 1} K (u) - W ^ {- 1} K (v) \right\| \leq \rho \| u - v \|, u, v \in L ^ {2} (D) ^ {n}, \tag {4.3}
$$
then, the operator  $\sigma \circ F_1$  is bijective.
See Section E.3 in Appendix E for the proof. We note that if  $K$  is compact linear, then assumption (4.3) implies that  $W + K$  is an injective Fredholm operator with index zero, which is equivalent to
$\sigma \circ (W + K)$  being bijective as observed in Proposition 2. That is, Proposition 4 requires stronger assumptions when applied to the linear case.
Assumption (4.3) implies the injectivity of  $K$ . An interesting example of injective operators arises when  $K$  are Volterra operators. When  $D \subset \mathbb{R}^d$  is bounded and  $K(u) = \int_{D} k(x, y, u(y)) u(y) dy$ , where we denote  $x = (x_1, \ldots, x_d)$  and  $y = (y_1, \ldots, y_d)$ , we recall that  $K$  is a Volterra operator if  $k(x, y, t) \neq 0$  implies  $y_j \leq x_j$  for all  $j = 1, 2, \ldots, d$ . A well known fact, as discussed in Example 4 in Appendix E, is that if  $K(u)$  is a Volterra operator whose kernel  $k(x, y, t) \in C(\overline{D} \times \overline{D} \times \mathbb{R}^n)$  is bounded and uniformly Lipschitz in  $t$ -variable then  $F: u \mapsto u + K(u)$  is injective.
Remark 2. A similar analysis (illustrating coercivity) shows that operators of the following two forms are bijective.
1. Operators of the form  $F(U) \coloneqq \alpha u + K(u)$  where  $K(u) \coloneqq \int_{D} a(x, y, u(x), u(y)) dy$  where  $a(x, y, s_1, s_2)$  is continuous and is such that  $\exists R > 0$ ,  $c_1 < \alpha$  so that for all  $|(s_1, s_2)| > R$ ,  $\operatorname{sign}(s_1) \operatorname{sign}(s_2) a(x, y, s_1, s_2) \geq -c_1$ .
2. Layers of the form  $(\mathcal{L}_{\ell}v)(x) := \sigma_{1}(Wu + \sigma_{2}(K(u)))$  where  $\sigma_{1}$  is bijective and  $\sigma_{2}$  bounded.
Finally we remark that coercivity is bounded preserved by perturbations in a bounded domain. This makes it possible to study non-linear and non-positive perturbations of physical models. For example, in quantum mechanics when a non-negative energy potential  $|\phi|^4$  is replaced by a Mexican hat potential  $-C|\phi|^2 + |\phi|^4$ , as occurs in the study of magnetization, superconductors and the Higgs field.
4.2 Construction of the inverse of a non-linear integral neural operator
The preceding sections clarified sufficient conditions for surjectivity and bijectivity of the non-linear operator  $F_{1}$ . We now consider how to construct the inverse of  $F_{1}$  in a compact set  $\mathcal{V}$ . We find that constructing inverses is possible in a wide variety of settings and, moreover, that the inverses themselves can be given by neural operators. The proof that neural operators may be inverted with other neural operators provides a theoretical justification for Integral Auto Encoder networks [Ong et al., 2022] where an infinite encoder/decoder pair plays a role parallel to those of encoder/decoders in finite-dimensional VAEs [Kingma and Welling, 2013]. This section proves that the decoder half of a IAE-net is provably able to inverse the encoder half. Our analysis also shows that injective differential operators (as arise in PDEs) and integral operator encoders form a formal algebra under operator composition. We prove this in the rest of this section, but first we summarize the main three steps of the proof.
First, by using the Banach fixed point theorem and invertibility of derivatives of  $F_{1}$  we show that, locally,  $F_{1}$  may be inverted by an iteration of a contractive operator near  $g_{j} = F_{1}(v_{j})$ . This makes local inversion simple in balls which cover the set  $\mathcal{V}$ . Second, we construct partition of unity functions  $\Phi_{j}$  that masks the support of each covering ball and allows us to construct one global inverse that simply passes through to the local inverse on the appropriate ball. Third and finally, we show that each function used in both of the above steps are representable using neural operators with distributional kernels.
As a simple case, let us first consider the case when  $n = 1$ , and  $D \subset \mathbb{R}$  is a bounded interval, and the operator  $F_{1}$  of the form
$$
F _ {1} (u) (x) = W (x) u (x) + \int_ {D} k (x, y, u (y)) u (y) d y,
$$
where  $W \in C^{1}(\overline{D})$  satisfies  $0 < c_{1} \leq W(x) \leq c_{2}$  and the function  $(x,y,s) \mapsto k(x,y,s)$  is in  $C^3 (\overline{D}\times \overline{D}\times \mathbb{R})$  and in  $\overline{D} \times \overline{D} \times \mathbb{R}$  its three derivatives and the derivatives of  $W$  are uniformly bounded by  $c_{0}$ , that is,
$$
\left\| k \right\| _ {C ^ {3} (\bar {D} \times \bar {D} \times \mathbb {R})} \leq c _ {0}, \quad \left\| W \right\| _ {C ^ {1} (\bar {D})} \leq c _ {0}. \tag {4.4}
$$
The condition (4.4) implies that  $F_{1}: H^{1}(D) \to H^{1}(D)$ , contains locally Lipschitz smooth functions. Furthermore,  $F_{1}: H^{1}(D) \to H^{1}(D)$  is Fréchet differentiable at  $u_{0} \in C(\overline{D})$ , and we denote Fréchet
derivative of  $F_{1}$  at  $u_{0}$  by  $A_{u_0}$ , which can be written as the integral operator (F.2). We will assume that for all  $u_{0}\in C(\overline{D})$ , the integral operator
$$
A _ {u _ {0}}: H ^ {1} (D) \rightarrow H ^ {1} (D) \text {i s a n i n j e c t i v e o p e r a t o r .} \tag {4.5}
$$
This happens for example when  $K(u)$  is a Volterra operator, see Examples 4 and 5. As the integral operators  $A_{u_0}$  are Fredholm operators having index zero, this implies that the operators (4.5) are bijective. The inverse operator  $A_{u_0}^{-1}: H^1(D) \to H^1(D)$  can be written by the integral operator (F.5).
We will consider the inverse function of the map  $F_{1}$  in  $\mathcal{Y} \subset \sigma_{a}(\overline{B}_{C^{1,\alpha}(\overline{D})}(0,R)) = \{\sigma_{a} \circ g \in C(\overline{D}) : \|g\|_{C^{1,\alpha}(\overline{D})} \leq R\}$ , which is a set of the image of Hölder spaces  $C^{n,\alpha}(\overline{D})$  through (leaky) ReLU-type functions  $\sigma_{a}(s) = \mathrm{ReLU}(s) - a\mathrm{ReLU}(-s)$  with  $a \geq 0$ . We note that  $\mathcal{Y}$  is a compact subset of the Sobolev space  $H^{1}(D)$ , and we use the notations  $B_{\mathcal{X}}(g,R)$  and  $\overline{B}_{\mathcal{X}}(g,R)$  as open and closed balls with radius  $R > 0$  at center  $g \in \mathcal{X}$  in Banach space  $\mathcal{X}$ .
To this end, we will cover the set  $\mathcal{V}$  with small balls  $B_{H^1 (D)}(g_j,\varepsilon_0)$ ,  $j = 1,2,\ldots ,J$  of  $H^{1}(D)$ , centered at  $g_{j} = F_{1}(v_{j})$ , where  $v_{j}\in H^{1}(D)$ . As considered in detail in Appendix F, when  $g$  is sufficiently near to the function  $g_{j}$  in  $H^{1}(D)$ , the inverse map of  $F_{1}$  can be written as a limit  $(F_{1}^{-1}(g),g) = \lim_{m\to \infty}\mathcal{H}_{j}^{\circ m}(v_{j},g)$  in  $H^{1}(D)^{2}$ , where
$$
\mathcal {H} _ {j} \left( \begin{array}{c} u \\ g \end{array} \right) := \left( \begin{array}{c} u - A _ {v _ {j}} ^ {- 1} (F _ {1} (u) - F _ {1} (v _ {j})) + A _ {v _ {j}} ^ {- 1} (g - g _ {j}) \\ g \end{array} \right),
$$
that is, near  $g_{j}$  we can approximate  $F_{1}^{-1}$  as a composition  $\mathcal{H}_j^{\circ m}$  of  $2m$  layers of neural operators.
To glue the local inverse maps together, we use a partition of unity  $\Phi_{\vec{i}}$ ,  $\vec{i} \in \mathcal{I}$  in the function space  $\mathcal{V}$ , where  $\mathcal{I} \subset \mathbb{Z}^{\ell_0}$  is a finite index set. The function  $\Phi_{\vec{i}}$  are given by neural operators
$$
\Phi_ {\vec {i}} (v, w) = \pi_ {1} \circ \phi_ {\vec {i}, 1} \circ \phi_ {\vec {i}, 2} \circ \dots \circ \phi_ {\vec {i}, \ell_ {0}} (v, w), \quad \text {w h e r e} \quad \phi_ {\vec {i}, \ell} (v, w) = (F _ {y _ {\ell}, s (\vec {i}, \ell), \epsilon_ {1}} (v, w), w),
$$
where some  $\epsilon_1 > 0$ ,  $s(\vec{i},\ell)\in \mathbb{R}$  are some suitable values near  $g_{j(\vec{i})}(y_{\ell})$ , some  $y_{\ell}\in D$  ( $\ell = 1,\dots,\ell_0$ ), and  $\pi_1$  is the map  $\pi_1(v,w) = v$  that maps a pair  $(v,w)$  to the first function  $v$ . Here,  $F_{z,s,h}(v,w)$  are integral neural operators with distributional kernels  $F_{z,s,h}(v,w)(x) = \int_{D}k_{z,s,h}(x,y,v(x),w(y))dy$ , where  $k_{z,s,h}(x,y,v(x),w(y)) = v(x)\mathbf{1}_{[s - \frac{1}{2} h,s + \frac{1}{2} h)}(w(y))\delta (y - z)$ , and  $\mathbf{1}_A$  is a indicator function of a set  $A$  and  $y\mapsto \delta (y - z)$  is the Dirac delta distribution at the point  $z\in D$ . Using these, we can write the inverse of  $F_{1}$  at  $g\in \mathcal{V}$  as
$$
F _ {1} ^ {- 1} (g) = \lim  _ {m \rightarrow \infty} \sum_ {\vec {i} \in \mathcal {I}} \Phi_ {\vec {i}} \mathcal {H} _ {j (\vec {i})} ^ {\circ m} \left(\begin{array}{c}v _ {j (\vec {i})}\\g\end{array}\right) \quad \text {i n} H ^ {1} (D) \tag {4.6}
$$
where  $j(\vec{i})\in \{1,2,\dots ,J\}$
This result is summarized in following theorem which is proven in Appendix F.
Theorem 3. Assume that  $F_{1}$  satisfies the above assumptions (4.4) and (4.5) and that  $F_{1}: H^{1}(D) \to H^{1}(D)$  is a bijection. Let  $\mathcal{Y} \subset \sigma_{a}(\overline{B}_{C^{1,\alpha}(\overline{D})}(0,R))$  be a compact subset of the Sobolev space  $H^{1}(D)$ , where  $\alpha > 0$  and  $a \geq 0$ . Then the inverse of  $F_{1}: H^{1}(D) \to H^{1}(D)$  in  $\mathcal{Y}$  can written as a limit (4.6) that is, as a limit of integral neural operators.
In this paper, we provided a theoretical analysis of injectivity and bijectivity for neural operators. In the future, we will further develop applications of our theory, particularly in the areas of generative models and inverse problems and integral autoencoders. We gave a rigorous framework for the analysis of the injectivity and bijectivity of neural operators including when either the ReLU activation or bijective activation functions are used. We further proved that injective neural operators are universal approximators and their finite-rank implementation are still injective. Finally, we ended by considering the 'coarser' problem of non-linear integral operators, as arises in subnetworks, operator transformers and integral autoencoders.
ML was partially supported by Academy of Finland, grants 273979, 284715, 312110. M.V. de H. was supported by the Simons Foundation under the  $\mathrm{MATH} + \mathrm{X}$  program, the National Science Foundation under grant DMS-2108175, and the corporate members of the Geo-Mathematical Imaging Group at Rice University.