SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Takashi Furuya<sup>1</sup> Michael Puthawala<sup>2</sup> Matti Lassas<sup>3</sup> Maarten V. de Hoop<sup>4</sup>
<sup>1</sup>Shimane University, takashi.furuya0101@gmail.com  
<sup>2</sup>South Dakota State University, Michael.Puthawala@sdstate.edu  
<sup>3</sup>University of Helsinki, matti.lassas@helsinki.fi  
<sup>4</sup>Rice University, mdehoop@rice.edu
Recently there has been great interest in operator learning, where networks learn operators between function spaces from an essentially infinite-dimensional perspective. In this work we present results for when the operators learned by these networks are injective and surjective. As a warmup, we combine prior work in both the finite-dimensional ReLU and operator learning setting by giving sharp conditions under which ReLU layers with linear neural operators are injective. We then consider the case when the activation function is pointwise bijective and obtain sufficient conditions for the layer to be injective. We remark that this question, while trivial in the finite-rank setting, is subtler in the infinite-rank setting and is proven using tools from Fredholm theory. Next, we prove that our supplied injective neural operators are universal approximators and that their implementation, with finite-rank neural networks, are still injective. This ensures that injectivity is not 'lost' in the transcription from analytical operators to their finite-rank implementation with networks. Finally, we conclude with an increase in abstraction and consider general conditions when subnetworks, which may have many layers, are injective and surjective and provide an exact inversion from a 'linearization.' This section uses general arguments from Fredholm theory and Leray-Schauder degree theory for non-linear integral equations to analyze the mapping properties of neural operators in function spaces. These results apply to subnetworks formed from the layers considered in this work, under natural conditions. We believe that our work has applications in Bayesian uncertainty quantification where injectivity enables likelihood estimation and in inverse problems where surjectivity and injectivity corresponds to existence and uniqueness of the solutions, respectively.
In this work, we produce results at the intersection of two fields: neural operators (NO), and injective and bijective networks. Neural operators [Kovachki et al., 2021a,b] are neural networks that take a infinite dimensional perspective on approximation by directly learning an operator between Sobolev spaces. Injectivity and bijectivity on the other hand are fundamental properties of networks that enable likelihood estimation by the change of variables formula, are critical in applications to inverse problems, and are useful properties for downstream applications.
The key contribution of our work is the translation of fundamental notions from the finite-rank setting to the infinite-rank setting. By the 'infinite-dimension setting' we refer to the case when the object of approximation is a mapping between Sobolev spaces. This task, although straightforward on first inspection, often requires dramatically different arguments and proofs as the topology, analysis and notion of noise are much simpler in the finite-rank case as compared to the infinite-rank case. We see our work as laying the groundwork for the application of neural operators to generative models
in function spaces. In the context of operator extensions of traditional VAEs [Kingma and Welling, 2013], injectivity of a decoder forces distinct latent codes to correspond to distinct outputs.
Our work draws parallels between neural operators and pseudodifferential operators Taylor [1981], a class that contains many inverses of linear partial differential operators and integral operators. The connection to pseudodifferential operators provided an algebraic perspective to linear PDE Kohn and Nirenberg [1965]. An important fact in the analysis of pseudodifferential operators, is that the inverses of certain operators, e.g. elliptic pseudodifferential operators, are themselves pseudodifferential operators. By proving an analogous result in section 4.2, that the inverse of invertible NO are themselves given by NO, we draw an important and profound connection between (non)linear partial differential equations and NO.
We also believe that our methods have applications to the solution of inverse problems with neural networks. The desire to use injective neural networks is one of the primary motivations for this work. These infinite dimensional models can then be approximated by a finite dimensional model without losing discretization invariance, see Stuart [2010]. Crucially, discretization must be done 'at the last possible moment,' or else performance degrades as the discretization becomes finer, see Lassas and Siltanen [2004] and also Saksman et al. [2009]. By formulating machine learning problems in infinite dimensional function spaces and then approximating these methods using finite dimensional subspaces, we avoid bespoke ad-hoc methods and instead obtain methods that apply to any discretization.
More details on our motivations for and applications of injectivity & bijectivity of neural operators are given in Appendix A.
1.1 Our Contribution
In this paper, we give a rigorous framework for the analysis of the injectivity and bijectivity of neural operators. Our contributions are as follows:
(A) We show an equivalent condition for the layerwise injectivity and bijectivity for linear neural operators in the case of ReLU and bijective activation functions (Section 2). In the particular ReLU case, the equivalent condition is characterized by a directed spanning set (Definition 2).
(B) We prove that injective linear neural operators are universal approximators, and that their implementation by finite rank approximation is still injective (Section 3). We note that universal approximation theorem (Theorem 1) in the infinite dimensional case does not require an increase in dimension, which deviate from the finite dimensional case Puthawala et al. [2022a, Thm. 15].
(C) We zoom out and perform a more abstract global analysis in the case when the input and output dimensions are the same. In this section we 'coarsen' the notion of layer, and provide a sufficient condition for the surjectivity and bijectivity of nonlinear integral neural operators with nonlinear kernels. This application arises naturally in the context of subnetworks and transformers. We construct their inverses in the bijective case (Section 4).
In the finite-rank setting, injective networks have been well-studied, and shown to be of theoretical and practical interest. See Gomez et al. [2017], Kratsios and Bilokopytov [2020], Teshima et al. [2020], Ishikawa et al. [2022], Puthawala et al. [2022a] for general references establishing the usefulness of injectivity or any of the works on flow networks for the utility of injectivity and bijectivity for downstream applications, [Dinh et al., 2016, Siahkoohi et al., 2020, Chen et al., 2019, Dinh et al., 2014, Kingma et al., 2016], but their study in the infinite-rank setting is comparatively underdeveloped. These works, and others, establish injectivity in the finite-rank setting as a property of theoretical and practical interest. Our work extends Puthawala et al. [2022a] to the infinite-dimensional setting as applied to neural operators, which themselves are a generalization of multilayer perceptrons (MLPs) to function spaces. Moreover, our work includes not only injectivity, but also surjectivity in the non-ReLU activation case, which Puthawala et al. [2022a] has not focused on.
Examples of works in these setting include neural operators Kovachki et al. [2021b,a], DeepONet Lu et al. [2019], Lanthaler et al. [2022], and PCA-Net Bhattacharya et al. [2021], De Hoop et al. [2022]. The authors of Alberti et al. [2022] recently proposed continuous generative neural networks (CGNNs), which are convolution-type architectures for generating  $L^2 (\mathbb{R})$ -functions, and provided the sufficient condition for the global injectivity of their network. Their approach is the wavelet basis expansion, whereas our work relies on an independent choice of basis expansion.
1.3 Networks considered and notation
Let  $D \subset \mathbb{R}^d$  be an open and connected domain, and  $L^2(D; \mathbb{R}^h)$  be the  $L^2$  space of  $\mathbb{R}^h$ -value function on  $D$  given by  $L^2(D; \mathbb{R}^h) \coloneqq \underbrace{L^2(D; \mathbb{R}) \times \cdots \times L^2(D; \mathbb{R})}_{h} = L^2(D)^h$ .
Definition 1 (Integral and pointwise neural operators). We define an integral neural operator  $G: L^2(D)^{d_{in}} \to L^2(D)^{d_{out}}$  and layers  $\mathcal{L}_{\ell}: L^2(D)^{d_{\ell}} \to L^2(D)^{d_{\ell+1}}$  by
$$
\begin{array}{l} G := T _ {L + 1} \circ \mathcal {L} _ {L} \circ \dots \mathcal {L} _ {1} \circ T _ {0}, \quad (\mathcal {L} _ {\ell} v) (x) := \sigma (T _ {\ell} (v) (x) + b _ {\ell} (x)), \\ T _ {\ell} (v) (x) = W _ {\ell} (x) u (x) + \int_ {D} k _ {\ell} (x, y) u (y), \quad x \in D, \\ \end{array}
$$
where  $\sigma : \mathbb{R} \to \mathbb{R}$  is a non-linear activation operating element-wise, and  $k_{\ell}(x,y) \in L^{2}(D \times D; \mathbb{R}^{d_{\ell+1} \times d_{\ell}})$  are integral kernels, and  $W_{\ell} \in C(\overline{D}; \mathbb{R}^{d_{\ell+1} \times d_{\ell}})$  are pointwise multiplications with matrices, and  $b_{\ell} \in L^{2}(D)^{d_{\ell+1}}$  are bias functions ( $\ell = 1, \dots, L$ ). Here,  $T_{0}: L^{2}(D)^{d_{in}} \to L^{2}(D)^{d_{1}}$  and  $T_{L+1}: L^{2}(D)^{d_{L+1}} \to L^{2}(D)^{d_{out}}$  are mappings (lifting operator) from the input space to the feature space and mappings (projection operator) from the feature spaces to the output space, respectively.
The layers  $T_{0}$  and  $T_{L + 1}$  play a special role in the neural operators. They are local linear operators and serve to lift and project the input data from and to finite-dimensional space respectively. These layers may be 'absorbed' into the layers  $\mathcal{L}_1$  and  $\mathcal{L}_L$  without loss of generality (under some technical conditions), but are not in this text to maintain consistency with prior work. Prior work assumes that  $d_{in} < d_{1}$ , we only assume that  $d_{in} \leq d_{1}$  for lifting operator  $T_0: L^2(D)^{d_{in}} \to L^2(D)^{d_1}$ . This would seemingly play an important role in the context of injectivity or universality, but we find that our analysis does not require that  $d_{in} < d_{1}$  at all. In fact, as elaborated in Section 3.2, we may take  $d_{in} = d_{\ell} = d_{out}$  for  $\ell = 1, \ldots, L$  and our analysis is the same.
2 Injective linear neural operator layers with ReLU and bijective activations
In this section we present sharp conditions under which a layer of a neural operator with ReLU activation is injective. The Directed Spanning Set (DSS) condition, described by Def. 2 is a generalization of the finite-dimensional DSS [Puthawala et al., 2022a] which guarantees layerwise injectivity of ReLU layers. Extending this condition from finite to infinite dimensions is not automatic. The finite-dimensional DSS will hold with high probability for random weight matrices if they are expansive enough ([Puthawala et al., 2022a, Theorem7]). However, the infinite-dimensional DSS is much more restrictive than the finite dimensional setting. We then present a less restrictive condition that is met when the activation function is bijective, e.g. a leaky-ReLU activation is used.
Although it may appear that the end-to-end result is strictly stronger than the layerwise result, this is not the case. The layerwise result is an exact characterization, whereas the end-to-end result is sufficient for injectivity, but not necessary. The layerwise analysis is also constructive, and so gives a rough guide for the construction of injective networks, whereas the global analysis is less so. Finally, the layerwise condition has different applications, such as network of stochastic depth, see e.g. Huang et al. [2016], Benitez et al. [2023]. End-to-end injectivity by enforcing layerwise injectivity is straightforward, whereas deriving a sufficient condition for networks of any depth is more daunting.
We denote by
$$
\sigma (T v + b) (x) := \left( \begin{array}{c} \sigma (T _ {1} v (x) + b _ {1} (x)) \\ \vdots \\ \sigma (T _ {m} v (x) + b _ {m} (x)) \end{array} \right), \quad x \in D,   v \in L ^ {2} (D) ^ {n}
$$
where  $\sigma : \mathbb{R} \to \mathbb{R}$  is a non-linear activation function,  $T \in \mathcal{L}(L^2(D)^n, L^2(D)^m)$ , and  $b \in L^2(D)^m$ , where  $\mathcal{L}(L^2(D)^n, L^2(D)^m)$  is the space of linear bounded operators from  $L^2(D)^n$  to  $L^2(D)^m$ . The aim of this section is to characterize the injectivity condition for the operator  $v \mapsto \sigma(Tv + b)$  mapping from  $L^2(D)^n$  to  $L^2(D)^m$ , which corresponds to layer operators  $\mathcal{L}_{\ell}$ . Here,  $T : L^2(D)^n \to L^2(D)^m$  is linear.
2.1 ReLU activation
Let  $\mathrm{ReLU}:\mathbb{R}\to \mathbb{R}$  be ReLU activation, defined by  $\mathrm{ReLU}(s) = \max \{0,s\}$ . With this activation function, we introduce a definition which we will find sharply characterizes layerwise injectivity.
Definition 2 (Directed Spanning Set). We say that the operator  $T + b$  has a directed spanning set (DSS) with respect to  $v \in L^2(D)^n$  if
$$
\left. \operatorname {K e r} \left(T \mid_ {S (v, T + b)}\right) \cap X (v, T + b) = \{0 \}, \right. \tag {2.1}
$$
where  $T|_{S(v,T + b)}(v) = (T_i v)_{i\in S(v,T + b)}$  and
$$
S (v, T + b) := \left\{i \in [ m ] \mid T _ {i} v + b _ {i} > 0 \text {i n} D \right\}, \tag {2.2}
$$
$$
X (v, T + b) := \left\{u \in L ^ {2} (D) ^ {n} \left| \begin{array}{l} f o r i \notin S (v, T + b) a n d x \in D, \\ (i) T _ {i} v (x) + b _ {i} (x) \leq T _ {i} u (x) i f T _ {i} v (x) + b _ {i} (x) \leq 0, \\ (i i) T _ {i} u (x) = 0 i f T _ {i} v (x) + b _ {i} (x) > 0 \end{array} \right. \right\}. \tag {2.3}
$$
The name directed spanning set arises from the  $\ker (T|_{S(v,T + b)})$  term of (2.1). The indices of  $S(v,T + b)$  are those that are directed (positive) in the direction of  $v$ . If  $T$  restricted to these indices together span  $L^2 (D)^n$ , then  $\ker (T|_{S(v,T + b)})$  is  $\{0\}$ , and the condition is automatically satisfied. Hence, the DSS condition measures the extent to which the set of indices, which are directed w.r.t.  $v$ , form a span of the input space.
Proposition 1. Let  $T \in \mathcal{L}(L^2(D)^n, L^2(D)^m)$  and  $b \in L^2(D)^m$ . Then, the operator  $\operatorname{ReLU} \circ (T + b): L^2(D)^n \to L^2(D)^m$  is injective if and only if  $T + b$  has a DSS with respect to every  $v \in L^2(D)^n$  in the sense of Definition 2.
See Section B for the proof. Puthawala et al. [2022a] has provided the equivalent condition for the injectivity of ReLU operator in the case of the Euclidean space. However, proving analogous results for operators in function spaces require different techniques. Note that because Def. 2 is a sharp characterization of injectivity, it can not be simplified in any significant way. The condition restrictive Def. 2 is, therefore, difficult to relax while maintaining generality. This is because for each function  $v$ , multiple components of the function  $Tv + b$  are strictly positive in the entire domain  $D$ , and cardinality  $|S(v; T + b)|$  of  $S(v; T + b)$  is larger than  $n$ . This observation prompts us to consider the use of bijective activation functions instead of ReLU, such as leaky ReLU function, defined by  $\sigma_{a}(s) \coloneqq \mathrm{ReLU}(s) - a\mathrm{ReLU}(-s)$  where  $a > 0$ .
2.2 Bijective activation
If  $\sigma$  is injective, then injectivity of  $\sigma \circ (T + b): L^2(D)^n \to L^2(D)^m$  is equivalent to the injectivity of  $T$ . Therefore, we consider the bijectivity in the case of  $n = m$ . As mentioned in Section 1.3, an significant example is  $T = W + K$ , where  $W \in \mathbb{R}^{n \times n}$  is injective and  $K: L^2(D)^n \to L^2(D)^n$  is a linear integral operator with a smooth kernel. This can be generalized to Fredholm operators (see e.g., Jeribi [2015, Section 2.1.4]), which encompasses the property for identity plus a compact operator. It is well known that a Fredholm operator is bijective if and only if it is injective and its Fredholm index is zero. We summarize the above observation as follows:
Proposition 2. Let  $\sigma : \mathbb{R} \to \mathbb{R}$  be bijective, and let  $T: L^2(D)^n \to L^2(D)^m$  and  $b \in L^2(D)^m$ . Then,  $\sigma \circ (T + b): L^2(D)^n \to L^2(D)^m$  is injective if and only if  $T: L^2(D)^n \to L^2(D)^m$  is injective. Furthermore, if  $n = m$  and  $T \in \mathcal{L}(L^2(D)^n, L^2(D)^n)$  is the linear Fredholm operator, then,  $\sigma \circ (T + b): L^2(D)^n \to L^2(D)^n$  is bijective if and only if  $T: L^2(D)^n \to L^2(D)^n$  is injective with index zero.
We believe that this characterization of layerwise injectivity is considerably less restrictive than Def. 2, and the characterization of bijectivity in terms of Fredholm theory will be particularly useful in establishing operator generalization of flow networks.
3 Global analysis of injectivity and finite-rank implementation
In this section we consider global properties of the injective and bijective networks that constructed in Section 2. First we construct end-to-end injective networks that are not layerwise injective. By doing this, we may avoid the dimension increasing requirement that would be necessary from a layerwise analysis. Next we show that injective neural operators are universal approximators of continuous functions. Although the punchline resembles that of [Puthawala et al., 2022a, Theorem 15], which relied on Whitney's embedding theorem, the arguments are quite different. The finite-rank case has dimensionality restrictions, as required by degree theory, whereas our infinite-rank result does not. Finally, because all implementations of neural operators are ultimately finite-dimensional, we present a theorem that gives conditions under which finite-rank approximations to injective neural operators are also injective.
3.1 Global analysis
By using the characterization of layerwise injectivity discussed in Section 2, we can compose injective layers to form  $\mathcal{L}_L\circ \dots \mathcal{L}_1\circ T_0$ , a injective network. Layerwise injectivity, however, prevents us from getting injectivity of  $T_{L + 1}\circ \mathcal{L}_L\circ \dots \mathcal{L}_1\circ T_0$  by a layerwise analysis if  $d_{L + 1} > d_{out}$ , as is common in application [Kovachki et al., 2021b, Pg. 9]. In this section, we consider global analysis and show that  $T_{L + 1}\circ \mathcal{L}_L\circ \dots \mathcal{L}_1\circ T_0$ , nevertheless, remains injective. This is summarized in the following lemma.
Lemma 1. Let  $\ell \in \mathbb{N}$  with  $\ell < m$ , and let the operator  $T:L^{2}(D)^{n}\to L^{2}(D)^{m}$  be injective. Assume that there exists an orthogonal sequence  $\{\xi_k\}_{k\in \mathbb{N}}$  in  $L^2 (D)$  and a subspace  $S$  in  $L^2 (D)$  such that  $\mathrm{Ran}(\pi_1T)\subset S$  and
$$
\operatorname {s p a n} \left\{\xi_ {k} \right\} _ {k \in \mathbb {N}} \cap S = \{0 \}. \tag {3.1}
$$
where  $\pi_1: L^2(D)^m \to L^2(D)$  is the restriction operator defined in (C.1). Then, there exists a linear bounded operator  $B \in \mathcal{L}(L^2(D)^m, L^2(D)^\ell)$  such that  $B \circ T: L^2(D)^n \to L^2(D)^\ell$  is injective.
See Section C.1 in Appendix C for the proof.  $T$  and  $B$  correspond to  $\mathcal{L}_L\circ \dots \mathcal{L}_1\circ T_0$  (from lifting to  $L$ -th layer) and  $T_{L + 1}$  (projection), respectively. The assumption (3.1) on the span of  $\xi_{k}$  encodes a subspace distinct from the range of  $T$ . In Remark 3 of Appendix C, we provide an example that satisfies the assumption (3.1). Moreover, in Remark 4 of Appendix C, we show the exact construction of the operator  $B$  by employing projections onto the closed subspace, using the orthogonal sequence  $\{\xi_k\}_{k\in \mathbb{N}}$ . This construction is given by the combination of "Pairs of projections" discussed in Kato [2013, Section I.4.6] with the idea presented in [Puthawala et al., 2022b, Lemma 29].
3.2 Universal approximation
We now show that the injective networks that we consider in this work are universal approximators. We define the set of integral neural operators with  $L^2$ -integral kernels by
$$
\mathrm {N O} _ {L} (\sigma ; D, d _ {i n}, d _ {o u t}) := \left\{G: L ^ {2} (D) ^ {d _ {i n}} \rightarrow L ^ {2} (D) ^ {d _ {o u t}}: \right.
$$
$$
G = K _ {L + 1} \circ \left(K _ {L} + b _ {L}\right) \circ \sigma \dots \circ \left(K _ {2} + b _ {2}\right) \circ \sigma \circ \left(K _ {1} + b _ {1}\right) \circ \left(K _ {0} + b _ {0}\right),
$$
$$
K _ {\ell} \in \mathcal {L} (L ^ {2} (D) ^ {d _ {\ell}}, L ^ {2} (D) ^ {d _ {\ell + 1}}), K _ {\ell}: f \mapsto \int_ {D} k _ {\ell} (\cdot , y) f (y) d y \Bigg | _ {D}, k _ {\ell} \in L ^ {2} (D \times D; \mathbb {R} ^ {d _ {\ell + 1} \times d _ {\ell}}),
$$
$$
\left. b _ {\ell} \in L ^ {2} \left(D; \mathbb {R} ^ {d _ {\ell + 1}}\right), d _ {\ell} \in \mathbb {N}, d _ {0} = d _ {i n}, d _ {L + 2} = d _ {o u t}, \ell = 0, \dots , L + 2 \right\}, \tag {3.2}
$$
$$
\mathrm {N O} _ {L} ^ {i n j} (\sigma ; D, d _ {i n}, d _ {o u t}) := \{G \in \mathrm {N O} _ {L} (\sigma ; D, d _ {i n}, d _ {o u t}): G \text {i s i n j e c t i v e} \}.
$$
The following theorem shows that  $L^2$ -injective neural operators are universal approximators of continuous operators.
Theorem 1. Let  $D \subset \mathbb{R}^d$  be a Lipschitz bounded domain, and  $G^{+}: L^{2}(D)^{d_{in}} \to L^{2}(D)^{d_{out}}$  be continuous such that for all  $R > 0$  there is  $M > 0$  so that
$$
\left\| G ^ {+} (a) \right\| _ {L ^ {2} (D) ^ {d _ {\text {o u t}}}} \leq M, \forall a \in L ^ {2} (D) ^ {d _ {i n}}, \| a \| _ {L ^ {2} (D) ^ {d _ {i n}}} \leq R, \tag {3.3}
$$
We assume that either (i)  $\sigma \in \mathrm{A}_0^L\cap \mathrm{BA}$  is injective, or (ii)  $\sigma = \mathrm{ReLU}$ . Then, for any compact set  $K\subset L^{2}(D)^{d_{in}}$ $\epsilon \in (0,1)$ , there exists  $L\in \mathbb{N}$  and  $G\in \mathrm{NO}_L^{inj}(\sigma ;D,d_{in},d_{out})$  such that
$$
\sup  _ {a \in K} \left\| G ^ {+} (a) - G (a) \right\| _ {L ^ {2} (D) ^ {d _ {o u t}}} \leq \epsilon .
$$
See Section C.3 in Appendix C for the proof. For the definitions of  $\mathrm{A}_0^L$  and BA, see Definition 3 in Appendix C. For example, ReLU and Leaky ReLU functions belong to  $\mathrm{A}_0^L\cap \mathrm{BA}$  (see Remark 5 (i)).
We briefly remark on the proof of Theorem 1 emphasizing how its proof differs from a straightforward extension of the finite-rank case. In the proof we first employ the standard universal approximation theorem for neural operators ([Kovachki et al., 2021b, Theorem 11]). We denote the approximation of  $G^{+}$  by  $\widetilde{G}$ , and define the graph of  $\widetilde{G}$  as  $H: L^2(D)^{d_{in}} \to L^2(D)^{d_{in}} \times L^2(D)^{d_{out}}$ . That is  $H(u) = (u, \widetilde{G}(u))$ . Next, utilizing Lemma 1, we construct the projection  $Q$  such that  $Q \circ H$  becomes an injective approximator of  $G^{+}$  and belongs to  $\mathrm{NO}_L(\sigma; D, d_{in}, d_{out})$ . The proof for universal approximation theorem is constructive. If, in the future, efficient approximation bounds for neural operators are given, such bounds can likely be used directly in our universality proof to generate corresponding efficient approximation bounds for injective neural operators.