Globally injective and bijective neural operators
Authors: Takashi Furuya
Michael Anthony Puthawala
Matti Lassas
Maarten V. de Hoop
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Learning
Operator Learning
Functional Analysis
Injectivity
Bijectivity
Universal approximation
TL;DR: We ask and answer some fundamental questions related to injectivity and bijectivity in operator learning.

Abstract: Recently there has been great interest in operator learning, where networks learn operators between function spaces from an essentially infinite-dimensional perspective. In this work we present results for when the operators learned by these networks are injective and surjective. As a warmup, we combine prior work in both the finite-dimensional ReLU and operator learning setting by giving sharp conditions under which ReLU layers with linear neural operators are injective. We then consider the case when the activation function is pointwise bijective and obtain sufficient conditions for the layer to be injective. We remark that this question, while trivial in the finite-rank setting, is subtler in the infinite-rank setting and is proven using tools from Fredholm theory. Next, we prove that our supplied injective neural operators are universal approximators and that their implementation, with finite-rank neural networks, are still injective. This ensures that injectivity is not 'lost' in the transcription from analytical operators to their finite-rank implementation with networks. Finally, we conclude with an increase in abstraction and consider general conditions when subnetworks, which may have many layers, are injective and surjective and provide an exact inversion from a 'linearization.’ This section uses general arguments from Fredholm theory and Leray-Schauder degree theory for non-linear integral equations to analyze the mapping properties of neural operators in function spaces. These results apply to subnetworks formed from the layers considered in this work, under natural conditions. We believe that our work has applications in Bayesian uncertainty quantification where injectivity enables likelihood estimation and in inverse problems where surjectivity and injectivity corresponds to existence and uniqueness of the solutions, respectively.

Globally injective and bijective neural operators
Takashi Furuya!
Michael Puthawala
Matti Lassas
Maarten V. de Hoop
Shimane University, takashi takashh..furuya001110 www.uaaa0010@mmaal..cmmmmm
South Dakota State University, University of Helsinki, matti lassss@helllliiiiii
edu
Ricee University, met.iioyy.mdhoooricee.eoe
Abstract
Recently there has been great interest in operator learning, where networks learn operators between function spaces from an essentially infiite-dimensional per- spective. In this work we present results for when the operators learned by these networks are injective and surjective. As a warmup, we combine prior work in both the finit-diimensional ReLU and operator learning setting by giving sharp conditions under which ReLU layers with linear neural operators are injective. We then consider the case when the activation function pointwise bijective and obtain sufficient conditions for the layer be injective. We remark that this question, while trivial in the finite-rank setting, is subtler the infinite-rank setting and is proven using tools from Fredholm theory. Next, prove that supplied injec- tive neural operators are universal approximators and that their implementation. with finite-rank neural networks, are still injective. This ensures that injectivity is not 'lost' in the transcription from analytical operators to their finite-rank imple- mentation with networks. Finally, we conclude with increase in abstraction and consider general conditions when subnetworks, which may have many layers, are injective and surjective and provide exact inversion from a 'linearization.' This section uses general arguments from Fredholm theory and Leray- -enhudS degree theory for non-linear integral equations analyze the mapping properties of neural operators in function spaces. These results apply to subnetworks formed from the layers considered in this work, under natural conditions. We believe that our work has applications in Bayesian uncertainty quantification where injectivity enables likelihood estimation and in inverse problems where surjectivity and injectivity corresponds to existence and uniqueness of the solutions, respectively.
Introduction
In this work, we produce results at the intersection of two fields: neural operators (NO), and injective and bijective networks. Neural operators [Kovachki et al. 2021a]b) are neural networks that take infinite dimensional perspective on approximation by directly learning an operator between Sobolev spaces. Injectivity and bijectivity on the other hand are fundamental properties of networks that enable likelihood estimation by the change of variables formula, are critical in applications inverse problems, and are useful properties for downstream applications.
The key contribution of our work is the translation of fundamental notions from the finite-rank setting to the infinite-rank setting. By the 'infinite-dimension setting' we refer to the case when the object of approximation a mapping between Sobolev spaces. This task, although straight-forward on first inspection, often requires dramatically different arguments and proofs as the topology, analysis and notion of noise are much simpler in the finite-rank case as compared to the infinite-rank case. We work as laying the groundwork for the application of neural operators to generative models
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

in function spaces. In the context of operator extensions of traditional VAEs [Kingma and Welling 2013]. injectivity of decoder forces distinct latent codes to correspond to distinct outputs.
Our work draws parallels between neural operators and pseudodifferential operators Taylor 198 class that contains many inverses of linear partial differential operators and integral operators. The connection to pseudodifferential operators provided an algebraic perspective to linear PDE Kohn and Nirenberg [166].. An important fact in the analysis of pseudodifferential operators, that the inverses of certain operators, e.g. elliptic pseudodifferential operators, are themselves pseudodifferential operators. By proving an analogous result in section 4.2, that the inverse of invertible NO are themselves given by NO, we draw an important and profound connection between (non)linear partial differential equations and NO.
We also believe that our methods have applications to the solution of inverse problems with neural networks. The desire to use injective neural networks is one of the primary motivations for this work. These infinite dimensional models can then be approximated by finite dimensional model without losing discretization invariance, see [tuart [210]] Crucially, discretization must be done "at the last possible moment," else performance degrades as the discretization becomes finer, see Lassas and Siltanen (2004) and also Saksman et al. (2009]. By formulating machine learning problems in infinite dimensional function spaces and then approximating these methods using finite dimensional subspaces, we avoid bespoke ad-hoc methods and instead obtain methods that apply to any discretization.
More details on our motivations for and applications of injectivity & bijectivity of neural operators given in Appendix
1.1 Our Contribution
In this paper, we give a rigorous framework for the analysis of the injectivity and bijectivity of neural operators. Our contributions are as follows:
(A) We show an equivalent condition for the layerwise injectivity and bijectivity for linear neural operators in the case of ReLU and bijective activation functions (Section[2). In the particular ReLUU case, the equivalent condition is characterized by directed spanning set (Definition
(B) We prove that injective linear neural operators are universal approximators, and that their implementation by finite rank approximation is still injective (Section 33. We note that universal approximation theorem (Theorem IT in the infinite dimensional case does not require an increase in dimension, which deviate from the finite dimensional case Puthawala
et al. 2022a Thm. 15].
(C) We zoom out and perform a more abstract global analysis in the case when the input and output dimensions are the same. In this section we 'coarsen' the notion of layer, and provide sufficient condition for the surjectivity and bijectivity of nonlinear integral neural operators with nonlinear kernels. This application arises naturally in the context of subnetworks and transformers. We construct their inverses in the bijective case (Section
1.2 Related Works
In the finite-rank setting, injective networks have been well-studied, and shown to be of theoretical and practical interest. See Gomez al. [2017] Kratsios and Bilokopytov (2020). Teshima al.) (2020). Ishikawa et al. [2022). Puthawala et al. 2022al for general references establishing the usefulness of injectivity any of the works on Hlww networks for the utility of injectivity and bijectivity for
downstream applications, [Dinh et al. 2016 Siahkoohi et al. 2020,Chhen Chen et al.) 2019 Dinh et al. 2014 Kingmaa et al., (0166), but their study in the inñnite- rank setting is comparatively underdeveloped. These works, and others, establish injectivity in the finite-rank, setting
property of theoretical and practical interest. Our work extends Puthawala et al. (022aa to the infinie-diimeniinall setting as applied to neural operators, which themselves are a generalization of multilayer perceptrons (MLPs) to function spaces. Moreover, our work includes not only injectivity, but also surjectivity in the
non-ReLU activation case, which Puthawala et al. 2022a has not focused on.

---

22221b a DeepONet Lu et al. (2019), Lanthaler al. [2022]. and PCA-Net (Bhattacharya et al. [021]. De Hoop al.)
Examples of works in these setting include neural operators Kovachki et al.
[2022] The authors of Albberti et al. [022] recently proposed continuous generative neural networks
(CGNNs), which are convolution-yye ttppe architectures for generating (2("R)-uncttin and provided
the sufficient condition for the global injectivity of their network. Their approach is the wavelet basis
expansion, whereas our work relies on an independent choice of basis expansion.
1.3 Networks considered and notation
Let D C R2 be an open and connected domain, and (22 I be the 1,2 space of -R-uvalue function on D given by (D; = (D; x .... x (D; = (2))..
Definition 1 (Integral and pointwise neural operators). We define an integral neural operator G (²(D) 7 ( and layers Le (2( + (D) by
G := :++1+ L1 .... L1 To. (Crv)(z) = (z)
(z(+)(x)= We(z)u(z)+ J, (elaz,))(o))
E D.
where a R + R is non-linear activation operating element-wise, and Ke(z, y) E L2(D x D; D: xdr) are integral kernels, and We E C(D; ((D:Rd+ix00 xdx) are pointwise multiplications with matrices, and by E (D) de are bias functions (E = =.....,,) L.) Here, To (2) dmn + (2) and TL+1 L2 + L'(D)dou dDdd are mappings (lifting operator) from the input space to the feature space and mappings (projection operator) from the feature spaces to the output space, respectively.
The layers To and Ti-1 play a special role in the neural operators. They are local linear operators and serve to lift and project the input data from and to finite-dimensional space respectively. These layers may be 'absorbed' into the layers &1 and L. without loss of generality (under some technical conditions), but are not in this text to maintain consistency with prior work. Prior work assumes that din < d1. we only assume that d S d1 for lifting operator To (D))d.. + [D))) This would seemingly play an important role in the context of injectivity or universality, but we find that
our analysis does not require that din. < di at all. In fact, as elaborated in Section 3.2 we may take din = dg dout for l I..... L and our analysis the same.
2 Injective linear neural operator layers with ReLU and bijective activations
In this section we present sharp conditions under which layer of neural operator with ReLU activation injective. The Directed Spanning Set (DSS) condition, described by Def. 2 a generalization of the finite-dimensional DSS (Puthawala et al. 2022a] which guarantees layerwise injectivity of ReLU layers. Extending this condition from finite to infinite dimensions not automatic. The miite-dimeniionll DSS will hold with high probability for random weight matrices if they are expansive enough ((Puthawala et al. 2022al Theorem7]). However, the infiniteddimensional DSS is much more restrictive than the finite dimensional setting. We then present a less restrictive condition that is met when the activation function is bijective, e.g. a leaky-Rell Re activation used.
Although it may appear that the end-to-end result is strictly stronger than the layerwise result, this is not the case. The layerwise result is an exact characterization, whereas the end-to-end result is sufficient for injectivity, but not necessary. The layerwise analysis is also constructive, and so gives a rough guide for the construction of injective networks, whereas the global analysis is less so. Finally, the layerwise condition has different applications, such as network of stochastic depth, see e.g. Huang et al. (2016), [enitez et al.) 2023). End-to-end injectivity by enforcing layerwise injectivity is straightforward, whereas deriving sufficient condition for networks of any depth is more daunting.
We denote by
(TTi(()) + bz()))
(T)+
x DED, v (D)"
(T(T)(()) + (x))
3

---

where a. IR + R is a non-linear activation function, T (D),
and b E DD) where (D)" (I" is the space of linear bounded operators from (D)) to DD) The aim of this section is to characterize the injectivity condition for the operator ->> a (Tu+T mapping from (D)) to T.2 (D)",, which corresponds to layer operators Lz. Here, T: (D)) - (D)
is linear.
2.1 ReLU activation
Let ReLU R - R. be ReLU activation, defined by ReI.U(s) max s). With this activation function, we introduce a definition which we will find sharply characterizes layerwise injectivity. Definition 2 (Directed Spanning Set). We say that the operator has a directed spanning ser (DSS) with respect to E (D)" if
Ker (Tlso)44)) (x(u,T++b) =0},
(2.1)
where T ((( ((::)))) and
T = {iE [m]] T/ + b, > 0 D).
(2.2)
for & Swu, b) and x E D. (1) (T + b,(x) S TT((x) ifT, + bx))) s 0, }. (2.3)
u E (D))"
(1) T(((z) = 0 [[ + bx))) >
The name directed spanning set arises from the ker(TT) ((v term of S(v,, T + b) are those that are directed (positive) in the direction of v. If T restricted to these indices together span (D)",, then ker(T) (().T+))) is {0), and the condition is automatically satisfied. Hence, the DSS condition measures the extent to which the set of indices, which are directed w.r.t. form a span of the input space.
The indices of
Proposition 1. Let T E (DL"(D)", (") and E L²(D)" Then, the operator ReLU (T)
(D)) + (D)m is injective if and only ifT + has a DSS with respect to every o E (D)) in the sense of Definition 2
See Section B for the proof. Puthawala al. injectivity of ReLU operator in the case of the Euclidean space. However, proving analogous results for operators in function spaces require different techniques. Note that because Def. 2 sharp characterization of injectivity, it can not be simplified in any significant way. The condition restrictive Def. is, therefore, difficult relax while maintaining generality. This is because for each function multiple components of the function To + b are strictly positive in the entire domain D, and cardinality [S(e; T + b)] of S(v; T + b) is larger than n. This observation prompts us to consider the use of bijective activation functions instead of ReLU, such as leaky ReLU function, defined by (*) == (eLU(s) - a ReLU(-s) where O.
2022a has provided the equivalent condition for the
2.2 Bijective activation
If a is injective, then injectivity of a (T bb) (D)" (D) I2(D)" is equivalent to the injectivity
T. Therefore, we consider the bijectivity in the case of m. As mentioned in Section 1.3 an significant example is T = W + K, where W E Rnxa is injective and K L²(D)" + (D) is linear integral operator with a smooth kernel. This can be generalized to Fredholm operators (see e.g.. Jeribi (2015) Section 2.1. which encompasses the property for identity plus compact operator. It is well known that a Fredholm operator is bijective if and only if it is injective and its Fredholm index is zero. We summarize the above observation as follows:
Proposition 2. Let a IR + R be bijective, and lei T LD(D)" + L2(D)m and E L²(D) Then, (T+ + b)= =²(D)" + L²(D) is injective if and only isT (D)" + (²(D)) is injective. Furthermore, if n = m and T E ,
is the linear Fredholm operator,
- (D)" is bijective if and only ifT LD) + (D)" is injective then, a (T bb) :) (D)) with index zero.
We believe that this characterization of layerwise injectivity is considerably less restrictive than Def. 2, and the characterization of bijectivity in terms of Fredholm theory will be particularly useful in establishing operator generalization of flow networks.

---

3 Global analysis of injectivity and finite-rank implementation
In this section we consider global properties of the injective and bijective networks that constructed in Section[] First construct end-to-end injective networks that not layerwise injective. By doing this, we may avoid the dimension increasing requirement that would be necessary from layerwise analysis. Next we show that injective neural operators are universal approximators of continuous functions. Although the punchline resembles that of [Puthawala et al. 2022a] Theorem 15), which relied on Whitney's s embedding theorem, the arguments are quite different. The finite-rank case has dimensionality restrictions. as required by degree theory, whereas our infinite-rank result does not, Finally, because all implementations of neural operators are ultimately umite-dimensional we present a theorem that gives conditions under which tinite-rank approximations to injective neural operators are also injective.
3.1 Global analysis
By using the characterization of layerwise injectivity discussed in Section 2 we can compose injective layers to form L2 o ... L1 a To, injective network. Layerwise injectivity, however, prevents us from getting injectivity of ++ LL .... L, a To by layerwise analysis if d dout, as is common in application [Kovachki et al., 2021b] Pg. 9]. In this section, we consider global analysis and show that 7 1++1 o or. Li o To. nevertheless, remains injective. This is summarized in the following lemma.
Lemma 1. Lett E N with é < m, and let the operator T (D) - (D)) be injective. Assume that there exists an orthogonal sequence {Ex keN I.) and a subspace ID) such that Ran(TIT) C and
span n ===(0}
(3.1)
where (2 + (D) is the restriction operator defined in C.
Then, there exists a linear
bounded operator BE ( such that B o T LA² (D) + LD( is injective.
See Section C.I in Appendix for the proof. T and B correspond to Lg o.... L1 o To (from lifting to L-th layer) and TL+1 (projection), respectively. The assumption (3.1) the span of & encodes a subspace distinct from the range of T. In Remark 3 of Appendix CJ we provide an example that satisfies the assumption
(3.] Moreover, in Remark 4 of Appendix C we show the exact construction of the operator B by employing projections onto the closed subspace, using the orthogonal sequence (Ex ken- This construction is given by the combination of "Pairs of projections" discussed in Kato 2013 Section 1.4.6] with the idea presented in [Puthawala et al. 2022b Lemma 29].
3.2 Universal approximation
We now show that the injective networks that we consider in this work are universal approximators. We define the of integral neural operators with L -ininteraal kernels by
NO D, dowt) = { LD( + L2(D)dowe:
G = KLT1 (KL + bL) o........ ..... (K2 b2) (Ki + b1) (Ko + bo),
K2 (D),, (D) Ke:ft + / kel Rel:v)
ke E I.2 x D: Dd
be E 1. (D; de E N, do din, drt2 dont, E = 0 I 0=0,..,++2 2
(3.2)
and
NO (o; D,din, dow) = (G E NOr,oo; D, din, dow)) G is injective).
The following theorem shows that 1.2 injeecteeee neural operators are universal approximators of continuous operators.
Theorem 1. Let D C Rdd be Lipschitz bounded domain, and G+ 12(DD) - T2(()) be continuous such that for all R > there is M > so that
(G) (a)) < M, Va E (D)dd,, | $ R.
(3.3)
5

---

We assume that either (i) E A n BA is injective. or (ii) = ReLU, Then, for any compact set K C. (2))), (0, 1), there exists L N and G E NO (o; D, dins dout) such that
sup acK
|| (a) G(a)
S..
See Section C.3 in Appendix for the proof. For the definitions of A and BA, see Definition 3 in Appendix For example, ReLU and Leaky ReLU functions belong to A n BA (see Remark 5
(1).
We briefly remark the proof of Theorem I lemphasizing how its proof differs from straighttfrward extension of the finite-rank case. In the proof we first employ the standard universal approximation
theorem for neural operators (Kovachkii al., 2021b Theorem 11 We denote the approximation of G+ by G. and define the graph of G H (D(D)á + (DDdd X L L²(D)doue That is H(u) = (u, ((u)). Next, utilizing Lemma we construct the projection Q such that Q H becomes an injective approximator of G+ and belongs to NO D, din dout). The proof for universal approximation theorem is constructive. If, in the future, efficient approximation bounds for neural operators given, such bounds can likely be used directly in universality proof to generate corresponding efficient approximation bounds for injective neural operators.
This approach resembles the approach in the finite-rank space Puthawala et al. 2022a Theorem 15], but unlike that theorem we don't 't have any dimensionality restrictions. More specifically, in the case of Euclidean spaces Rdd Puthawala al. 2002a Theorem 15] requires that 2dl., + dout before all continuous functions G+ 1 + IR Rdau can be uniformly approximated in compact sets by injective neural networks. When din dout 1 this result is not true, as is shown in Remark 5 (iii) in Appendix C using topological degree theory [Cho and Chen 2006]. In contrast, Theorem does not assume any conditions on din and dout- Therefore, can conclude that infinite dimensional case yields better approximation results than the finite dimensional case.
This surprising improvement in restrictions in infinite-dimensions can be elucidated by an analogy Hilbert's hotel paradox, see [Burger and Starbird [2004] Sec 3.2]. In this analogy, the orthonormal bases { JKEN and J, (z) (dsjsP >==) play the part of guests in the hotel with N floor, each of which as d rooms, A key step in the proof of Theorem [ is that there is a linear isomorphism
(D) + (D( (i.e., a rearrangement of guests) which maps j.k to (P((.A), where b [d] N N a bijection.
3.3 dnfettvvity-presevvin transfer to Euclidean spaces via finite-rank approximation
In the previous section, we have discussed injective integral neural operators. The conditions are given in terms of integral kernels, but such kernels may not actually be implementable with finite width and depth networks, which have finite representational power. A natural question ask, therefore, is how should these formal integral kernels be implemented on actual finite rank networks, the so-called implementable neural operators? In this section we discuss this question.
We consider linear integral operators K₁ with L Li-interral kernels ke y). Let {spk tREN be an orthonormal basis in L LDD).. Since (1){))); [x)] is an orthonormal basis of L2(D x D). integral kernels kg L2 x D; B.dee1xxde xde) in integral operators Ke L(D"(D)ds 1 (D)) has the expansion
ke(x, v)
Σ E.PEN
(0 ((()))))))))
where clO) E Redeiidee whose j)th) component (0) R.p.ij
is given by
(0) k.n.j = (kz,1), (PNPP)
Here, we denote (u, (px) ) E de by (u, (p)) = ((u1, (>)))>)))>)>>>>>>>>>>>>>>>>>)))) (ud,, 9px (D))) By truncating by N finite sums, we approximate inttnelraal operators K₂ LD) (D) by finite rank operator KE,N with rank N, having the form
KeNu((x)= = Σ [.PEN]]
(u w (2)(ddt

---

The choice of orthonormal basis {yk KeN is a hyperparameter. If we choose {yp as Fourier basis and wavelet basis, then network architectures correspond Fourier Neural Operators (FNOs) [Li
et al. 2020b] and Wavelet Neural Operators (WNOs) (Tripura and Chakraborty, 2023). respectively.
show that Propositions 2
(characterization layerwise injectivity), and Lemma (global
injectivity) all have natural analogues for finite rank operator KN in Proposition 5 and Lemma in
Appendix D These conditions applies out-of-the-box to both FNOs and WNOs.
'recipe' to construct the projection B such that the composition B o (interpreted as augmenting finite rank neural operator T with one layer B) is injective. The projection B is constructed by using an orthogonal sequence {Ex JEEN subject to the condition (3.1). which does not over-leap the range of T'. This condition is automatically satisfied for any orthogonal base (pphkess This could yield practical implications in guiding the choice of the orthogonal basis {sP for the neural operator's design.
Lemma 2 and Remark 3 in the appendix givve
We also show the universal approximation in the case of finite rank approximation. We denote NOL.N D, din, dout) by the set of integral neural operators with N rank, that is the set (3.2) replacing I,2 i.2-integral kernel operators K2 with finite rank operators Ke with rank (see Defini- tion D. We define by
NO (a; D, din: dowr) = {GN E NOL,N(O; din dout) GN is injective).
Theorem 2. Let D C Rd be a Lipschitz bounded domain, and N E N, and G+ L22 -
12 (D)youu be continuous with boundedness as in (3.3). Assume that the non-linear activation function a is either ReL.U or Leaky ReLU. Then, for any compact set C I2 n (span (1ppn((p)) (px)) kgN))" 6 (0,1), there exists L E N, N' E N with
oecivattinfuunction
N'dout 2 2Nd:. +1,
(3.4)
ing and Gxv E L.v (a; D, dan, dout) such that
sup oEK
(a) Gy
S.
See Section D.4 in A Lemma 29]. which gives rise the assumption (3.4) We do not require any condition on din and dout as well as Theorem [
for for the proof. In the proof, we make use of Puthawala et al. 2022a
Remark 1. Observe that in our finite-rank approximation result, we only require that the target function G+ is continuous and bounded, but not smooth. This differs from prior work that requires smoothness of the function to be approximated.
3.4 Limitations
We assume square matrices for the bijectivity and construction of inversions. Weight matrices in actual neural operators are not necessarily square. Lifting and projection are local operators which map the inputs into a higher-dimensional feature space and project back the feature output to the output space. We also haven't 't addressed possible aliasing effects of our injective operators. We will relax the square assumption and investigate the aliasing of injective operators in future work.
4 Subnetworks & nonlinear integral operators: bijectivity and inversion
So far our analysis of injectivity has been restricted to the case where the only source of nonlinearity are the activation functions. In this section we consider weaker and more abstract problem where nonlinearities can also arise from the integral kernel with surjective activation function, such leaky ReLU. Specifically, we consider layers of the form
Fu( = Wu + K(u),
(4.1)
where W E L(TP(DD)) ID)(D))) is a linear bounded bijective operator, and K I.2(D))
(D) is non-linear operator. This arises in the non-linear neural operator construction by Kovachki et al. [221b) or in Ong et al. [2022] to improve performance of integral autocncoders. In this construction, each layer L is written as
I E D, (Lev)(x) = (Weo(v) + Kx(u(=z)), Ke(u)(x) = - ke(z, y, u(x), u(v))u(y)dy
7

---

where Wi E Rd independent of x, and Ke (2)4 - (D)dde is the non non-lineat integral operator.
This relaxing of assumptions is motivated by desire to obtain theoretical results for both subnetworks and operator transformers. By subnetworks, we mean compositions of layers within a network. This includes, for example, the encoder or decoder block of traditional VAE. By neural operator we mean operator generalizations of finite-rank transformers, which can be modeled by letting K be an integral transformation with nonlinear kernel & of the form
k( u(r), ((v) = softmax o (Ar(æ), Be(y))
where and B are matrices free parameters, and the (integral) softmax taken over æ. This specific choice of k can be understood as natural generalization of the attention mechanism in transformers, see [Kovachki et al. 2021b Sec. 5.2] for further details.
4.1
Surjectivity and bijectivity
Critical to our analysis the notion of coercivity, Apart from being a useful theoretical tool, layerwise coercivity of neural networks a useful property in imaging applications, see e.g Li et al. 2020
Recall from Showalter 2010, Sec 2, Chap VII] that a non-linear operator K (D) L (D) is
coercive if
lim
(K(u),
LP(D)" = 00.
(4.2)
|| 12
| LAADY
Proposition 3. Let a : R + R be surjective and W (D2( + (D) be linear bounded bijective (then the inverse W-1 bounded linear), and let K (D) (D)" be a continuous and compact mapping. Moreover, assume that the map u + ou + W-1K(u) is coercive with some O < 1. Then, the operator F1 is surjective.
See Section E.1 in Appendix E for the proof. An example K satisfying the coercivity
4.2 condition
given here.
Example We simply consider the case of n and D Rd is a bounded interval. We consider the non-linear integral operator
(())x))== = K(,, u. k, u()))u(y)d x D.
The operator u + au + W-1 W- K(u) kernel k(I, y,t) satisfies certain boundedness conditions. In Examples and in Appendix E show that these conditions are met by kernels k(z, y, t) of the form
with some < a < 1 is coercive when the non-linear integral
k(x, v, = Σ (2) g)t bx,,,))),,
3=1
where a : R. + - is the sigmoid function o, + R. and b, E C(D x D) or by wavelet activation function Twire R RR, see Saragadam et al. 2023)) where owire Im et and a, E C(D x D), and (2 v) 0.
In the proof of Proposition 3 we utilize the Leray- Sray--Shauder fix point theorem. By employing the Banach fix point theorem under a contraction mapping condition (4.3). we can obtain bijectivity as follows:
Proposition 4. Let : + be bijective. Let W (D) + (D)")" be bounded linear bijective, and let K LL2 + (²(")". WW--KK (D) D2(D)" a contraction mapping, that is, there exists E (0, such that
K(u) -ow s spllu - u|1, u. e u. 12(D)"
(4.3)
then, the operator a o F1 is bijective.
See Section E.3 in Appendix For the proof. We note that if K is compact linear, then assumption (4.3 implics that W + K is injective Fredholm operator with index zero, which is equivalent to

---

a (W + K) being bijective as observed in Proposition 2 That is, Proposition 4 requires stronger assumptions when applied the linear case.
Assumption 4.3 implies the the injectivity of K. An interesting example of injective operators arises
when K are Volterra operators. When C IR is bounded and ((u) = J k( y, u(y))u(y)d where we denote x (Z1 Ia) and y (yl,.... Dd), we recall that K is Volterra operator if k(I, y: 1) j 0 iimplis My S. 23 for all 1, 2,..,,d. A well known fact, as discussed in Example in Appendix E] is that if K (u) is Volterra operator whose kernel k(æ, y, t) E C(D x D x R") is bounded and uniformly Lipschitz f-variable then F u +> + ((u) is injective.
Remark 2. A similar analysis (illustrating coercivity) shows that operators of the following two forms are bijective.
the 1. Operators the form F(U) == acu. acu+K(u) where (()) = y. u(z), uly))dy where
a(x, y. 81, 82) is continuous and such that 3R > e1 < so that for all [($1, 82)l R, sign( ($1 (22gg(s2)a(2) y, $1, $2) 2 -1.
2. Layers of the form (Z) = σn (Wu (W (K(()))) where 0n is bijective and 02 bounded.
Finally we remark that coercivity is bounded preserved by perturbations in a bounded domain. This makes it possible to study non-linear and non-positive perturbations of physical models. For example, in quantum mechanics when a non-negative energy potential |Ø| replaced by a Mexican hat potential 2 + 1Ø] as occurs in the study of magnetization, superconductors and the Higgs field.
4.2 Construction of the inverse of a non-linear integral neural operator
The preceding sections clarified sufficient conditions for surjectivity and bijectivity of the non-linear operator Fj. We now consider how to construct the inverse of F1 in a compact set y. We find that constructing inverses is possible in wide variety of settings and, moreover, that the inverses themselves can be given by neural operators. The proof that neural operators may be inverted with other neural operators provides theoretical justification for Integral Auto Encoder networks [Ong) jet al. 2022) where an infinite encoder/decoder pair play a role parallel to those of encoder/decoders finite-dimensional VAEs Kingmaa and Welling 2013). This section proves that the decoder half of a IAE-net is provably able to inverse the encoder half. Our analysis also shows that injective differential operators (as arise in PDEs) and integral operator encoders form a formal algebra under operator composition. We prove this in the rest of this section, but first we summarize the main three steps of the proof.
First, by using the Banach fixed point theorem and invertibility of derivatives of F1 we show that, locally, Fi may be inverted by an iteration of contractive operator near 9; (v,).. This makes local inversion simple in balls which cover the set y. Second, we construct partition of unity functions Φ that masks the support of each covering ball and allows us to construct one global inverse that simply passes through to the local inverse on the appropriate ball. Third and finally. we show that each function used in both of the above steps are representable using neural operators with distributional kernels.
As a simple case, let us first consider the case when n and C R a bounded interval, and the operator Fi of the form
(z) = ((u)())=((<u((e)) + / k(x, y, u(y))
where W E CDD satisfies 0 < < W(x) S 02 and the function (x, + k(r,, y, 8) is in C3(D D x R) and in D x x IR its three derivatives and the derivatives of W are uniformly bounded by Co. that is,
|| | co | co
(4.4)
The condition implies that F1 HD + (D),, contains locally Lipschitz smooth functions. Furthermore, F (D) + H (D) is Fréchet differentiable at uo C(D), and we denote Fréchet

---

derivative of Fi at dg by Aug which can be written as the integral operator (..2) We will assume that for all uo E CD), the integral operator
Au H (D) H'' H'' (D) is an injective operator.
(4.5)
This happens for example when ((u) is Volterra operator, see Examples and 5 As the integral operators Aug are Fredholm operators having index zero, this implies that the operators (4.5) are bijective. The inverse operator A H (D) + H (D) can be written by the integral operator (.5)
We will consider the inverse function of the map F1 in Y Ba (D) (0, R)) (0a C(D) (99l)c.-( - R). which is a set of the image of Hölder spaces C (D) through (leaky) ReLU-type functions (s) RcLU(s) - ReLU(-s) with 2 O. We note that y is compact subset the the notations (g R) and (x R) open and closed balls with radius R > 0 at center E X in Banach space X.
Sobolev space H' (D), and
To this end, we will cover the set y with small balls (1)r j = 1,2...... . of (D), centered at gj Fi (wz), where V/ E (D).). As considered in detail in Appendix F when is sufficiently near to the function gj in H (D), the inverse map of Fi can be written limit
(FT (FF'(9).9) g) limm- H3 (0z.9) in H' (D)?, where
Hi 393 - u A (Fi(u) - Fi(2,)) + Any (g 95) )
(0)-((AA))
9
that is, near Øj we can approximate FT- composition Hmm of 2m layers of neural operators.
To glue the local inverse maps together, we use a partition of unity Φ3 E I in the function space } where I c 2o is finite index set. The function D are given by neural operators
Lz w) = m o Φ o 27,2 o ..... &re,, (v, w), where (r,(U, w) = (F (..... (v, w) w),
where some > 0, a(,, E) E IR are some suitable values near 03(3) (we), some y E D (E = 1,..., to). and mi is the map ग (v, w) v that maps pair (u, w) to the first function v. Here, Fz,s,1 (v, w) integral neural operators with distributional kernels (2, (x) = SD K2,0, (I,, y, u(r), w(y) ddy, where kiz,s. (I, y, v(x), w(y)) = u(z) (e- + (w(y)))(y - =). and L is a indicator function (w)++))
of set A and y 8(y= z) is the Dirac delta distribution at the point E D. Using these, we can write the inverse of Fi at g E as
FT (g)
lim
Σ
Hom com
in (D)
(4.6)
50
m 700000
9
TET
where j(0) E (I,
This result is summarized following theorem which is proven in Appendix
Theorem Assume that F1 satisfies the above assumptions (4.4) and (45) and that Fi : (D) + (D) is a bijection. Let y C oa (a(Ber.>(D) (0, R)) be a compact subset of the Sobolev space (D),, where a > and a 2 0. Then the inverse of F1 HD(D)) HHHH()) in y can written as limit 4.6p that is, as a limit of integral neural operators.
Discussion and Conclusion
In this paper, we provided a theoretical analysis of injectivity and bijectivity for neural operators. In the future, we will further develop applications of our theory, particularly in the areas of generative models and inverse problems and integral autoencoders. We gave a rigorous framework for the analysis of the injectivity and bijectivity of neural operators including when either the ReLU activation or bijective activation functions are used. We further proved that injective neural operators are universal approximators and their finite-rank implementation are still injective. Finally, we ended by considering the 'coarser' problem of non-linear integral operators, as arises in subnetworks, operator transformers and integral autocncoders.
10

---

Acknowledgments
ML was partially supported by Academy of Finland, grants 273979, 284715, 312110. M.V. de H. was supported by the Simons Foundation under the MATH X program, the National Science Foundation under grant DMS-2108175, and the corporate members of the Geo-Mathematical Imaging Group at Rice University.
References
Giovanni S. Alberti, Matteo Santacesaria, and Silvia Sciutto. Continuous generative neural networks. arXiv:2205. 14627, 2022.
Jose Antonio Lara Benitez, Takashi Furuya, Florian Faucher, Xavier Tricoche, and Maarten V de Hoop. Fine-tuning neural-operator architectures for training and generalization. arXiv preprint arXiv:2301. 11509, 2023.
Kaushik Bhattacharya, Bamdad Hosseini, Nikola Kovachki, and Andrew Stuart. Model reduction and neural networks for parametric pdes. The SMAI journal of computational mathematics, 7: 121-157, 2021.
Edward B Burger and Michael Starbird. The heart of mathematics: An invilation to effective thinking. Springer Science & Business Media, 2004.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
Yeol Cho and Yu-Qing Chen. Topological degree theory and applications. CRC Press, 2006.
Maarten De Hoop, Daniel Zhengyu Huang, Elizabeth Qian, and Andrew M Stuart. The cost-accuracy trade-off in operator learning with neural networks. arXiv preprint 131 2022.
Laurent Dinh, David Krueger, and Yoshua Bengio.
Nice: Non-linear independent components
estimation. arXiv preprint 1rXiv:1410.8516 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv
preprint arXiv: 088003, 2016.
David Gilbarg and Neil S. Trudinger. Elliptic partial differential equations of second order. Classics in Mathematics. Springer- Verlag, Berlin, 2001, ISBN 3-540-41160-7, Reprint of the 1998 edition.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. Advances in neural information processing systems, 30, 2017.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 646-661. Springer, 2016
Isao Ishikawa, Takeshi Teshima, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Universal approximation property of invertible neural networks. arXiv preprint arXiv:2204.07415. 2022.
Aref Jeribi. Spectral theory and applications of linear operators and block operator matrices, volume 9. Springer, 2015.
Tosio Kato. Perturbation theory for linear operators, volume 132. Springer Science & Business Media, 2013.
Diederik Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes. arXiv preprint
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling. Improving variational inference with inverse autoregressive flow, arXiv preprint arXiv: I 1606 04934, 2016.
11

---

Joseph Kohn and Louis Nirenberg, An algebra of -iffeeeeeiffr operators. Communications on Pure and Applied Mathematics, (8(1-2)22993355 1965.
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22:Art-No, 2021a.
Nikola Kovachki, Zongyi Li. Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv: (0:::2208.0888)) 2021
Anastasis Kratsios and levgen Bilokopytov. Non-euclidean universal approximation. Advances in Neural Information Processing Systems, 53:10635-10644 2020.
Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis. Error estimates for deeponets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6(1):tnac001,200
Matti Lassas and Samuli Siltanen. Can one use total variation prior for adge-preservingg bayesian inversion? Inverse problems, 20(5):1537, 2004.
Housen Li, Johannes Schwab, Stephan Antholzer, and Markus Haltmeier. NETT: solving inverse problems with deep neural networks. Inverse Problems, 36(6):065005, 23, 2020a. ISSN 0266-5611. ab6d57
doi: 10. (088/1161-6420/abbb)
URL https://///: 1001://00/00011111311---------1-
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprini arXiv:2010.0889) 2020b.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint 19t0::19900011110 2019.
Yong Zheng Ong, Zuowei Shen, and Haizhao Yang. lae-net: Integral autoencoders for discretization- invariant learning. arXiv preprint MxXiv:2003001444 2022.
Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143-195, 1999.
Michael Puthawala, Konik Kothari, Matti Lassas, Ivan Dokmanić, and Maarten de Hoop. Globally injective relu networks. Journal of Machine Learning Research, 23(105):1-55, 2022a.
Michael Puthawala, Matti Lassas, Ivan Dokmanic, and Maarten De Hoop. Universal joint approxima- tion of manifolds and densities by simple injective flows. International Conference on Machine Learning, pages 7759--17983 PMLR, 2022b.
Matti Lassas Saksman, Samuli Siltanen, et al. cntretization-invariaan bayesian inversion and besov space priors. arXiv preprint arXiv:0901.48 42200, 2009.
Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Gua Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. arXiv preprint arXiv:2301.05188, 2023.
Ralph E Showalter. Hilbert space methods in partial differential equations. Courier Corporation, 2010.
Ali Siahkoohi, Gabrio Rizzuti, Philipp Witte, and Felix Herrmann. Faster uncertainty quantifica- tion for inverse problems with conditional normalizing flows, arXiv preprint arXiv:2007.07985, 2020.
Andrew M Stuart. Inverse problems: bayesian perspective. Acta numerica, 19:451-559, 2010.
M.E. Taylor. Pseudodifferentaal Operators. Princeton mathematical series. Princeton University Press, 1981.
12

---

Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling based invertible neural networks are universal diffeomorphism approximators. Advances in Neural Information Processing Systems, 33:3362-3373, 2020.
Tapas Tripura and Souvik Chakraborty. Wavelet neural operator for solving parametric partial differential equations computational mechanics problems. Computer Methods in Applied Mechanics and Engineering, 404: 2023.
13

---

Appendix
A Motivation behind our injectivity & bijectivity
We consider the following partial differential equations (PDEs) of PDEs-based inverse problems: the form
Lau(x)= = f(x), x EE x aD,
Bu(x)= 9 g(z),
where D C RR is bounded domain, La is partial differential operator with coefficient E A(D == {a and function f & y are fixed. When a solution u E UU(D; R4)) = {u : + RR is uniquely determined, we may define the solution operator G : A(D; + (D; Rdx) by putting G(a) u. Note that the operator G is, in general, non-linear even if partial differential Operator La is linear (e.g., L. A a). The aim of the inverse problem is to evaluate G-1, the inverse of the solution operator. When G is non-injective, the problem is termed ill-posed.
Rd,, and is some boundary operator, e.g. Dirichlet or Neumann,
The key link between PDE and our injective neural operators is as follows. Can G be approximated by injective neural operators Nenz If so, then Nany can be surrogate model to solve an ill ill-posed inverse problem. In general, this is possible, even if G is non-injective, per the results in Section
Moreover, by the results of Section 4 we may conduct the inverse Nin of surrogate model Nanj with another neural operator.
Approximation of pushes forward measure: Let .M be the submanifold of X [,2 1J2) or X = ("( 1]3) corresponding to natural images or 3D medical model. Let K C RD be manifold with the same topology as M. :: IR + X be an embedding, and let K₁ ((K) C M. Given μ., measure supported on M, the task is to find a neural operator fa: X - X that maps (pushes forward) the uniform distribution on the model space K₁ fa and so thus maps K₁ M. If fo: X X is bijective, computing likelihood functions in statistical analysis is made easier via the change of variables formula. Further, we may interpret fo as an encoder and so as the corresponding decoder, which parameterized elements of M. As everything is formulated in infinite dimension function space X, we obtain discretization invariant methods.
B Proof of Proposition 1 in Section 2
Proof. We use the notation T (s) = (Es Assume that + has a DSS with respect to every u E (D)))" in the sense of Definition 2 and that
ReLU(TT()) + 6) = (eLU(TT(2) + +6) in D,
(B.1)
where v()) x12) (D)". Since T+b b has DSS with respect to y(),, we have for E S(b(),T+b))
0 < (eLU(T, + b,) ReLU(T;x(2) + 6;) in D,
which implies that
(:o()) + b, = 2?) + by in D.
Thus,
( - ((² E Ker (I) (T(s(oC))44))
(B.2)
By assuming B..l.. we have for ((D(), ,T),
tx E D (x) (x(2) 0} = (xED (z) < 0}.
Then, we have
T((p!1) - (6(1) - (())) +x(x) = T, (2) + (.(2) 20 (2) + ++(x) 0,
that is,
Tx) + (x()) S ST, (9 -(2) (0) (T (xx=))+))) (.(x) 50.
14

---

In addition,
(()) (x) (z(z) (Tu (x) + b((()) Tf b,(2))
Thus,
(11) (2)) E ( f, b).
(B.3)
Combining (B.2) and B.3). and 2.1 as v = v(), we conclude that
x(1) (2)
Conversely, assume that there exists (D)" such that
Ker (T) (Tlster44) (.(( (+ {0}.
Then there is u + such that
Ker (T) (T(s(,,T4)) NX(u,T+0) b)).
For i E Sv,, T + b), we have by u E Ker(T:),
ReLU (T,(u u) + bx())) (T
For i $ S b), we have by u X(u, T b).
{ 0 b(()) <0
+
ReLU (Ti(v ([(u--)(z)++00))
Tu((z) b,(x) if if + (,)x) 0
ReLU (Tiv(x) + +x))..
Therefore, conclude that
ReLU (T(v - u) + b) = ReLU (Tv + b),
where u * 0, that is, ReLU o b) is not injective.
Details in Sections 3.1 and 3.2
C.1
Proof of Lemma
Proof. The restriction operator, कe (D) + [D) (L < m), acts as follows,
Te(a, b) := b, (a,b) E (D x LD()
(C.1)
Since LDD) is separable Hilbert space, there exists orthonormal basis (sx in LD))). We denote by
ph. (o.... 9R 0o,,0,.... 0,.,, ..... LD)))
for & E N and E [m f). Then, (4k (ea.))kNN.jEmm- is an orthonormal sequence in T.2(D) and
Vo T.2(D) m- {0)
= span {sl e j [m -m]-
define, for a (0,
Ph (0..,,,, (11 a O,..,, VoE((k (0m((-1)(m-0+) (D)"))
(C.2)
j-th
with A E N and E [m f.. We note that {5l (ENJE(m-e) is an orthonormal sequence in mD
We set
k KEN,jE [m o).
Va == span pk
(C.3)
15

---

It holds for 0 < a < 1/2 that
Pv2
Pvo op
<1
Indeed, for u E ()) and < < 1/2,
Pvi u- Pvo u
= 1V, Pvo
L?(D)"
Σ KEN.jE(m-A)
(k. (u, (u.
Σ
(1- (u,, pk)
KENJE(m-O)
L?(D)
Σ
axlums ((4-1)(m 10-$
KEN.Elm
122
<a²
Σ
(ug, + a² Σ
(um, Ex)1 4a |ull
je(m
([mm-]] KEN
KeN
which implies that Pvv
Pvo
20x
op
We will show that the operator
Pv. TT: :T?() - (2""))"
injective. Assuming that for a, b (D())" equivalent
Pv- o T(a) = Pv1 o T(b),
T(a) - T(b) = Pv. (T T(b)).
Denoting by Pv, (T(a))((()) - = EkEN,JE(m- Ck.jPk...
T (T(a) - T(())
Σ
kEN
From 3.1 we obtain that ckj 0 for all k,j. By injectivity of T, we finally get b.
We define Qa (D + DD) by
Qn = (Pv. (P Pv. + (I- (( Pv.)) (1
(1- (Pv: - )P -
11/2
By the same argument in Section I.4.6 Kato 2013) we can show that Qo is injective and
Q. Pv. Pvo Qa.
that is, Qa maps from Ran(Pv.) to
Ran(Py.) C [0]m- x x L²(D)..
It follows that
me Qa Pyr of (D)" L?)D)'
is injective.
16

---

C.2 Remarks following Lemmal
Remark Lemmal and Eqn. (3.1) may be interpreted as saying that if some orthonormal sequence {5x JeEN exists that doesn't I overlap the range of then T may be embedded in small space without losing injectivity. An example that satisfies (3.1) is the neural operator whose L-th layer operator LL consists of the integral operator KL with continuous kernel function kL, and with continuous activation function a. Indeed, in this case, we may choose the orthogonal sequence {Ek]ken in L (D) discontinuous functions sequence I so that span (5x) C(D) {0). Then, by Ran(Cr,) C C(D)dz, the assumption (.1) holds.
Remark 4. In the proof of Lemma ! operator B CDL"
== nI Qa Pvt.
appears, where Pv. is the orthogonal projection onto orthogonal complement V of Va with
:= span {
a Pkk
k KEN,j ( -1 c (D)"
in which Pk.j is defined E (0, k E (
Phe = 0, V. 0, ..., O, v vat 1)(m /a6(k-1)(0m-))) 11mm-m)ss)))
8-th
Here, {sk]KEn is an orthonormal basis (D). Futhermore, Qa (2((D) (2) defined by
(1- Pv 11/2
Qa =
where Pv, is the orthogonal projection onto orthogonal complement Vi- of Vo with
Vo L3 {0)s.
The operator Qa is well well-defined for 0 < a < 1/2 because it holds that
Pv- Pv llop
2a.
This construction is given by the combination of "Pairs of projections" discussed in Kato 2013
Section 1.4.6) with the idea presented in (Puthawala el al. 2022b Lemma 29].
C.3 Proof of Theorem 1
We begin with
Definition 3. The set of L-layer neural networks mapping from I2 to Rdd is
Rd) = {s Rd + Rdd 1 [(x) = Wzo(. .. Wio by) + by ...) + by,
We E xde be E Ri de E No(do = d, = -d), 0, O,... }
where a R R is an element-wise nonlinear activation function. For the class of nonlinear activation functions,
Ao {o E C(R) En E No s.t. (o R) is dense in C(K)for VK Rd compact
[x(x) 00
A 0 Aulo is Borel measurable s.t. sup xER + [x]
l.gg. step functions whose supports disjoint for each sequence.
17

---

BA == for E AoltK CR compact, Vc > O, and AC 2 diam(K). En Nu, 3 No; (o; Rd) s. (f(x) - e, Vo E K, and, [f(x) S S, Vo e B
The set of integral neural operators with L'-iinteral kernels is
[o D, dins doat) (D) (D)
G = KL+1 (Kr + bb) o (K2 + by) oa (Ki bi)
(Ko + bo).
K2 E (D) ( (D) Kr f H /, ke(:, ka(:v))(y)dy
(C.4)
kt E LD D; D: Rd be E
de E N, do din, dLt2 doat: ..., +++2
Proof. Let > such that
& Bn( where BR(0)= == {u E (D())= ||l)) S RJ. By Theorem 11 of Kovachki et al. [021bb]
there exists L E N and Õ E NO,(a; D, p,din, done) such that
sup
G+(()) G(a)
(C.5)
L2(D))ou 2
a K
and
((a))
S 4M, for (L( ||l|и s R.
L2(D)
We write operator G by
G o (K1 + +b) ... (Ke + b2) oo (Ki + b,) o (Ko + bo).
where
Ke E (DI' (((())( feif / (el(.)))(()ddd
Te C(D D; ( be E L3((D
de N, do = din, dL+2 dout, l O...., +
We remark that kernel functions bú are continuous because neural operators defined in Kovachki et al. 2021b parameterize the integral kernel function by neural networks, thus,
(G) C CCD)
(C.6)
We define the neural operator H L2 + L (D) by
H KI+1 (Kr. + br.) ..... (K2 + b2) oa (Ki + bi) (Ko + bo),
where K2 and by are defined as follows. First, choose Kinj (D) din I (D))d)) as linear injective integral operator 2
(i) When σn E A 0 BA injective,
Kn = (
Kinj Ko
)
(D) d((Dddi bo = (0 bp E (T2(DD) tha
For example, if choose the integral kernel keng as (x, = Ei_ (x)xx())) then the integral operator Kijj with the kernel kinj is injective where (7) is the orthonormal basis in L (D)din..
18

---

Kinj O
(DL +de (2((Ddmntdeii) b== = ( bt
Kt
L (D) di
Ke
(LSESL),
KL+1 (Ko
Kins
O Kit1 ) (DL L2 (D)dintdoou)) be = (0)
O
(D) (inddindoon
(ii) When 0n = ReLU,
Kn
(
Kinj
(D( (D())mntt))) [ to
(10
0
E (D) dnn
Ko
bo
King Kinj
0
K1 =
E (D(L((D)antd, +dl (2(())anntd))) (11)))))tttt11211))atttttt) b0
0
E
)
2din 2dnnd +d;
K1
b1
Kinj Kinj
- Kinj
0
K₁- =
K Kinj
E C(T. 2dm +de (T), 2dhn +de+1).
Ke
0
b
0
E 1.2 (D((D)2dntde))
(25851) I.),
be
Kinj Kinj
0 (D) 2dent 1 (,)( (d)dttntttt))
Kr.
-----
0 Kg
( 11
by =
E L2((D) den +de+1
(Kong
King
O
KL+1=
E ((T. den +db+1 I.2(D) din tdout) )
K1+1
(o
0
bL+1
(L3(D)dnntin)
Then, the operator H : (2()) + L3(D)dintdaan has the form of
Kinj o Kinj o Kinj o.. o... 0o Kinj o Kinj
in the case of (i).
H ==
King o... Kinj ) in the case of (ii).
G
For the case of (ii), we have used the fact
- 1.
( I -I) ReLUo
Thus, in both cases, H is injective.
In the case (i), as a E Af. we obtain the estimate
[|a(f)ll 22 II L I (D)
where
[a(x)
C. sup zER 1 + x
00.
19

---

Then we evaluate for a E K(C (R())),
H [a(a LA(D) tdout
$ ((a) (a(D)doue
+ +IlKing o King o 0000 Kinj ..... 0 King o Kanj(a)ll
(C.7)
4M + 22Didnn Co
Σ Kim lop + | Hp
L42 R =: CH.
t=1
In the case of (ii), we find the estimate, for a E K,
[H I 4M + +| [ Sompllop lop R < Cи-
1
(C.8)
From C.6 (especially, Ran(m) H) C(D)) and Remark 3
we can choose an orthogonal sequence [5x JEeN in (D) such that (3.1) holds. By applying Lemma as 7 = H, n = din- m. = din + dout.
donts we find that
G: dous Qu o Pv oH (D + (D)
injective. Here, Pv. and Qa are defined as Remark we choose 0 < a << such that
Pv. Pvt ll min
10CH (10C 6D.
where Pv-- is the orthogonal projection onto
Vo = (0) x (D
By the same argument as in the proof of Theorem 15 in Puthawala et al. [2022a], we can show that
| Qallop I 4eo-
Furthermore, since is linear operator, KL+1 is also a linear operator with integral kernel (Bk441(53)) (z), where k2+1(z, v) is the kernel of KL+1. This implies that
G E NOr D,dins dout).
We get, for a E K,
- G(a))lra(D)uuu $ G+(a) - G(a)
[(((a)- (G(a)-G(a)
(C.9)
L2(D)
Using
C.7 and C.8 we then obtain
((a) G(a))
L2(D)oou =|| I| H(aa) Tdout QQ Pv. HH@)ll
Tduuue (Pv.' -Pv PVv)) H(a) Tdune o Qa o Pv. H(a))
Adone o (Pvo - Pv3)) H(a) H(w) L.(D)/one + Adoue o (l - - o Pv. o H(a) LA(Dy
< 5eo (H loot S
(C.10)
Combining C.. and C.10). we conclude that
sup aek
|| (G)((a) -G(a)lla
6 + = 6.
é
2
2
20

---

C.4 Remark following Theorem 1
Remark 5. We make the following observations using Theorem 7
(i) ReLU and Leaky ReLU functions belong to Ag n BA due to the fact that {a E C(RR) a is not polynomial) S Ao (see Pinkus 1999)), and both the ReLU and Leaky ReLU functions belong to BA (see Lemma C.2 in Lanthaler et al. (2022]). We note that Lemma C.2 in Lanthuler et al.) [2022] solely established the case for ReLU.. However, it holds true for Leaky ReLU as well since the proof relies on the fact that the function x + min(max(x, R), R) can be exactly represented by a two-layer ReLU neural network, and two-layer Leaky ReLU neural network can also represent this function. Consequently, Leaky ReLU is one of example that satisfies (ii) in Theorem 7
(ii) We emphasize that our infinite- dirrniieddimessoondl result, Theorem 4 deviates from the finite- dimensional result. Puthhaala al. (2022a] Theorem 15] assumes that 2d + 1 S dout due to the use of Whitney's 's theorem. In contrast, Theorem does not assume any conditions din and dout, that is, we are able avoid invoking Whimey's 's theorem by employing Lemma 7
(iii) We provide examples that injective universality does not hold when (D) den and (D) Doou are replaced by RRden and Rdout Consider the case where din = dout = and G+ R R is defined G+()) (x) sin(x). We not approximate G+ IR R by injective function G IR + R in the set K = 00 2m in the I -norm. According to the topological degree theory (see (ho and Chen [2006] Theorem 1.2.6(iii]],, any continuous function G: IRR + IR which satisfies ||G G ((())20) < E satisfies the equation on both intervals
T = [0, m). 12 = [,, 2n] deg G, J, deg 8) = 1 for all E [-1 + E, - e).
j j= 1,2. This implies that G 1 + R obtains the value E [-1+ + E, 1 E] at least once.
Hence, G obtains the values E [-1+ E, e at least two times on the interval [0, 2 and is it thus not injective. It is worth noting that the degree theory exhibits significant differences between the infinite-dime dimensional and Unite-dimensione cases [Cho and Chen 20066).
D Details in Section 3.3
D.1 Finite rank approximation
We consider linear integral operators K2 with L kernels ky (,, v). Let { be an orthonormal basis in (D). Since (x)( is an orthonormal basis of L,²(D D), integral kernels ke T.2 x D; Rdd41xde) in integral operators Ke E ((T2( T?2 (D)d+++) has the expansion
ke(z,w) v)
Σ (hpoa(l))n()) K.PEN
then integral operators Ke E ( take the form
(D))ee))))
Kyu(z)= = Σ ( (x),, u (D(
k.PEN
where Chp E &Rdetixd whose j-th component (
Rg3i is given by kpp.i = (ke,,), (uPp))
Here, we write (u, (k)) E Rd,,
(u, (u, (Hd,
We define Ke,N E (D)))dt)) as the truncated expansion of K₃ by N finite sum, that is,
Ke,u(xz) = Σ
(u.
KP
21

---

Then KE,N E (D),,,) (2 is a finite rank operator with rank N. Furthermore, we have
1/2
[ Ke,Nllop I [|Ke Ke,N
Σ ΣΣ
(6) Ckpis
(D.1)
A.P2N
which implies that as N + 00,
|K+ KeNNlop + 0.
D.2 Layerwise injectivity
We first revisit layerwise injectivity and bijectivity in the case of the finite rank approximation. Let KN: (D)) T ( be finite rank operator defined by
KNu(o)=
Σ
Ckp(u, u E (D)",
A.PEN
where Ck.p E Rmxx and (u, pp) E IR" is given by
(u, pp) (p))) (uns Pp) (p)) L2(D))
Let by (D)" be defined by
bx(x)
Σ (x))
PEN
which hp E mRm As analogues of Propositions
and
we obtain the following characterization.
Proposition (i) The operator
ReLU o(KN + bx): (span(ra)esN)" - L²(D)"M
is injective if and only for every (span(p*)AgN)",
{uE (D))" UN E (cr(Cs.N)) n X(U, KN + bx) n
(0}.
where Svv, KN bx) C [m] and ((v, KN bx) defined in Definition 2 and
UN = ((u,4p)) PSN E N", Cs.N =
E txxx
k.E[[N]
(D.2)
(ii) Let a be injective. Then the operator
a o (KN + bb)
LDD)
is injective if and only if CN is injective, where
ON == (Ck,a) E NmxNN.
(D.3)
Proof. The above statements follow from Propositions and by observing that u E Ker (Kx) is
equivalent to (cf. (D22) and (D.3)
Σ K.PSN
Cu,((u =0, CNăN 0.
D.3 Global injectivity
We revisit global injectivity in the case of finite rank approximation. As an analogue of Lemma we have the following
22

---

Lemma 2. Let N, E N and n. m. E N with N'm > 2 2Nn + 1, and TerT (2) 7 (D) be a finite rank operator with N' rank, that
RTn(T) (span(
(D.4)
and Lipschitz continuous, and
T: (span(pa)KsN) + L2(
injective. Then, there exists a finite rank operator B E (D)" (I(())) with rank N' such that
BoT: (span(px))sN) + (span((puasNN))
is injective.
Proof. From D.A.T :T.TT + (D) has the form
T(a)= Σ
(T(a),
pp)
A
where (T(a), E R". We define T R' R N' by
T(a) ==
E R'
E R."
where T(a) (D) is defined by
T(a) = T
Σ OKSK AEN
E LL
in which ag E CR, a = (a1, ..., aN) E E.N.
Since T (2( + (2) is Lipschitz continuous, T Nv + NR'm is also Lipschitz continuous. As N'm > N' 2Nn 1, we can apply Lemma 29 from Pothawwaaa et al. [2022a] with D = N'm, m = N'C, n Nn. According to this lemma, there exists a Eid-mnnnni linear subspace V in N such that
Pv Pv- < 1,
and
Pvr o T R + R'm. injective, where V = (0) N e) R N' Furthermore, in the proof of Theorem 15 of Puthawala et al. [2022a], denoting
TNY Qo Pva NEXN'
we are able to show that
B T Nv + N'
injective. Here, :eE: RN' N'E
TN b) b, (m x XR
where Q R'N' + N'm is defined by
Q= Pvg Pvi + (1 (T -( (1-(Pv - -Pv -1/2
We define B: LD + L2(D) by
Bu
Σ
BR (u,
where BR E f B = (BRo)kPEINT) Then B L2(D)" - LZ(D)é is a linear finite rank operator with rank, and
BoT: (D( : +
(D)
is injective because, by the construction, it is equivalent to
BoT: Nn R N'L
is injective.
23

---

D.4 Proof of Theorem 2
Definition 4. We define the set of integral neural operators with rank by
NO didi dout) = { Gx (D) + (D)dout:
GN = KLTIN o (KL,N + bI,N) ..... ... (K2,N + b2,N) 0 (Ki,N + x) (KoN - bo,N),
KE,N E ( KEN f+ Σ
A.PSN
DEN E (I,(DR bEN = Σ
p pm
PSN
Chp E Eddtiide, (1 E BR k. S N,
de E N, do din, dLt2 dout, = 0, ..., + 2
(D.5)
Proof. Let R 0 such that
K& BR(O);
where (R) {u E (D))) belongs to Ab BA, by Theorem of Kovachki et (2021b). there exists L N and G NO D. din, doue) such that
|| I R). As ReLU and Leaky ReLU function
sup aeK
G (a)-a)((a) 1A(D)ooa 3
(D.6)
and
(((a)
I 4M, for (D()),m
| s R.
LP(D)dout
We write operator by
G = Kr+1 o (K, + by) oo. bb) + +b,) o (Ko + bo).
where
Ke E (D) , (D())e1),, Ke:f + / hel:,v)
Th E T22 x D; DeRderixde). dy E (D; (e) de E N, do = din, d1.42 = douts ..., +
We set GN E NOL.N' (o: D, din, dout) such that GN-- = KLTIN' (K + )o0 ... -o (Ro
where KeN' (3(D)dt + (D) is defined by kex)u(x)=
Σ K.SN
where k.p E Rdeiixde whose (i. (d.j)-th component k.p.s is given by
(0) = (Ka,aj (20x))) kp.
Since
Kg Ke, 2
<
Kr KeN'
Σ
kapsy 2 - aas N' - oc,
op
HS
2.p2N'11
is
there is large N' E N such that
sup aeK
Ō(a)) Gx((a)
Le(p))mee $
(D.7)
24

---

Then, we have
sup aEK
(x
I sup aeK
(a)(a) C(a)
sup
G(a)
La(D)dou
ask
11+4M.
We define the operator Hx (D) - L2 (D(Dddintdou by
H.. )
King.N o... ... N ) Gvr(a)
(a)
HN )=
where King, (D()d (D(ddm is defined by
Kinj.Nu (u,
ASN
As King,N (span(()))<sw) dm 122 is injective,
Hx' (span(F*)esN) tn - (ppan( (span(fp) )ou
injective. Furthermore, by the same argument (ii) (construction of H) in the proof of Theorem
Hv E NOL, D, din dows),
because both of two-layer ReLU and Leaky ReLU neural networks can represent the identity map. Note that above KinjN is an orthogonal projection, so that N o King.N. However, we write above (x)(a)) (a) as KingNN
Kinj (a) that it can be considered as combination of
(L + 2) layers of neural networks.
We estimate that for a L2(D)A [a]l S R.
H (D)) I 11+ + 4M + || op
L+2 R =: CH-
Here, repeat argument similar to the one in the proof of Lemma 2 H [2 (D) has the form of
(a) = Σ
(Hxx pk)s
Σ (Hx
ASN
ASN
where (HNv (a),(()),, Rdd, (Hxr sPx) E Rdout, We define H NN - BR NdnntN' dous by
(x) = (HN-() (a) pP) uE(NN)
Nddn a E R Ndin,
where (a) (II,, (N) E L2(D) din tdous defined by
(v)(a): = HN.
aKpK ASN
Σ
E (D)
(x) = HN
Σ ON
(DD
KSN
where ag E Rdd., a (d)....,ax) E Ndin, Since H (DD L2()) tromt is Lipschitz continuous, Hy: Ndin + R dows is also Lipschitz continuous. As
Nd. + N'dous > N'dout > 2N + 1,
apply Lemma 29 of Puthawala al. [2022a] with D Ndi. N'duut m N' n Nd According to this lemma, there exists a dddou-ddimension linear subspace V- in Bg Nd,, +N dont such that
|| |v - -Pv op
min E =: eo
15C 1)
25

---

and
Pv H :R + R.Ndn+t'ddow
is injective, where V 10)-Ndan x IR d'our, Furthermore, in the proof of Theorem 15 of Puthawala al. [2022a], denoting by
B Nouu o Q Pv+.
we can show that
B HN RN Nouu
is injective, where dout : R Ndd +N dout 4 R. dout
Nout (a, =
E Nd... x R Vouut
and Q : RNd +N' N'du
- R Ndi daue is defined by
----P ( (Pv; - PV-))²) 1/2
Q (Pvg Pv- (v Pv ( By the same argument in proof of Theorem 15 Puthawala al. 2022a], we can show that
||1 Q I 4eg-
We define B: (D)))dannt
L2(()d
Bu
Bu,p(u, pk)ppp
APPNN'
BR.p E Rodut (di, B- (B (Br.p) then B (D) din doue - 1? is linear finite rank operator with N' rank. Then,
G == B Hx (D) + (.)
is injective because by the construction, it is equivalent to
B HN : Nd - IR N'duu
injective. Furthermore, we have
GN. NOL,N D, d,d: dout).
Indeed, Hs E NOL.N (a: D, din, dout), B the linear finite rank operator with N' rank, and multiplication of two linear finite rank operators with N' rank is also linear finite rank operator with N' rank.
Finally, we estimate for E K, ||G (a) GN((a) L2(D)dout
G (a) G(a)
((a))-Gx(aa) Gxr(a) La(D) + (a) Gar(a))
La(D)
L2(D)
D.6
(D.8)
Using notation (a, pk)) E Rdi,, and a = ((a, sk (x)) kE E Ndi,, we further estimate for a E K,
- (a) (a)
=|| (a) Bo Hx (o)
(o(
T'nd (v
Bo BN
i Niduut
(NN -Ndd Qo PvL HNr(a)llz
(Pv) Pyi + Pvi) HN.(a) -N''du Q Pv Hv((a)
dout
(D.9)
(Pv) -Pv Har(a) + +l oT Q) Pvi (Nr
< 5Eg
(a)ll)
3
= (o)]l Poop <CH
where |1:112 is the Euclidean norm. Combining D.8 and D.9.
we conclude that
sup | (a) - Gx Lou
6
e.
3
3 3
a
26

---

E Details in Section 4.1
E.1 Proof of Proposition 3
Proof. Since W is bijective, and a is surjective, it is enough to show that + Wu + K(u) is surjective. We observe that for E (D))).
Wu + K(u) =
is equivalent to
Hz(u): == K(u) + W-1==
We will show that He (D)) + (D) has fixed point for each E L2(D). By the Leray-Schaude theorem, see Gilbarg and Trudinger [2001 Theorem 11.3), H [22 + I2)
has fixed point if the union UosAgi Vi is bounded, where the sets
VA = {ue (D): u= (=)(
E (D) ='u=HH((M)
{u E (D) - = w - ((u)
-x-'==
are parametrized by 0 < 1. As the map - W-1K(u) is coercive, there is such that for ||]|L >r.
(aru + WV-KK(u), u) IP(D)" 2 |W-12|l1a(D)>
|| (2D)
Thus, we have that for ||u|| L2(D)" >r
(W-1K K(u) W-1 10--2,,00 u) L2(D)"
||ul|22DD)- (au + (- u) L2(D)" (au W- -2,22,,) (2,4))200 L2(D)"
|
W
L2(D)"
a0> -1
|| ||
||ul
and, hence, for all [|))) > and A E (0, 1] we have u & V2. Thus
U
Va c B(0,
XE(0.1)
Again, by the Leray-Schauder theorem (see Gilbarg and Trudinger 2000 Theorem 11.3)), H, has fixed point.
E.2 Examples for Proposition
Example We consider the case where n and D C Rdd a bounded interval. We consider the non-linear integral operator,
K(u)(x): = (( I E D,
and k(æ, y. is bounded, that there is Cg > 0 such that
1k(x, y, t) Ck, 2,y D.R.
If w- llop is small enough such that
1>
W W-1 Ck D),
op
27

---

then, for a (Iw- Cx(D),1). au + W- (u) coercive. Indeed, we have for (D).
(au WVlK(u), (2())
||u| (=
W |l [[(())](<)()) 2 aa | llop Cx|DI) 2 - For example, we can consider a kernel
|
20
k( y. - - +b(()()))
where ds R R. the sigmoid function defined by
(t(t) + 1ee
T are functions b, c E C(D x D) such that
Σ |s < || IIpp DD]
Example 3. Again, consider the case where and D Rd a is bounded set. We assume that W E C* (D) satisfies O < C1 S ((x) S C2. For simplicity, we assume that D = We consider the non-linear integral operator
((u)(o)= = J. k(x, y, u(z))u(y)d, D,
(E.1)
where
k( = E (1 u)t + (z v)),
(E.2)
in which Twire IR + IR is the wavelet functet defined by
Twire (t) Im (E (we
and bj.bj.c3 E C(D x D) are such that the az(z, y) are nowhere vanishing functions, that is, aj(x,v) u) # O for all a.y E D D. the and its generalizations (e.g. activation functions which do decay only exponentially).
The next lemma holds for any activation function with exponential decay, including the activation function wire and settles the key condition for Proposition] 3 to hold.
Lemma 3. Assume that |D| and the activation function E + R be continuous. Assume that there exists M1, mg > O such that
lo(1) S Mie-maldy L R.
Let aj bys oj C(DxD) be such that aj a;(xz 1) are nowhere vanishing functions, that is, az v) + for all F, E D x D. Moreover. let K (D) - (D) be non-linear integral operator given in E. with kernel satisfying E.2 Let > 0 and 0 < co S W(x) S C1 Then function
(D) - (D), F(u) = am + W- (u)) (u) is coercive.
Proof. As D is compact, there ap > 0 such that for all j = 1,2, J we have (z,(z, g)l 2 do a.c. and (z, b) by a.e. We point out that [o(()) < Mr. Next.
W l/ll
(E.3)
28

---

we consider À > 0 and u E (D) and the sets
Di(A) = fo D: |u(x)] 2 eΛ}, D2(A) fo D: |u(x) <e>].
Let s > 0 be such that
| (llL<(DDDD Mie 7.
Then, for x E D2(A),
W (0lellllsxxxplll//eomm)) + ()((()))((())
W (yle
a X. 4
After is chosen in the above, we choose An 2 bo/x(1,bo/aag)) (aoe)) to be sufficiently large that
for all [t] 2 εAu it holds
| (i - bol)t < a 4
Here, we observe that, as Ag (ao),, we have that for all |t| 2 EAO. aolel 2 Ao we have for E D2(A),
bb > 0. Then, when
| ( (0) u(z)
a
4
Ao 2 1, we have When u E (D) has the norm ||u||C(D) A
1/o ((x)) (at.-
S. (J | Miee -oola (ux:)] bu) |m(=))dr)) ) ju(v) ldy
| [wleolllelooooll(ssoooo v)(()) + by(s,, v)) (u(x)dd (u())dy
|u]le<()) + + ((ull(ra(
[uliz(D))
Hence,
(au + ((-)
a
w||
[1
2
and the function u + ou + W-1 is coercive.
E.3 Proof of Proposition 4
Proof. (Injectivity) Assume that
o(Wu1 K(u1) + b) = o(Wu2 + +((2)) b).
29

---

where U2 E (2)".. Since is injective and W LD - (D))) is bounded linear bijective, we have
M1 + W ((n)) = U2 + ((2)) =:
Since the mapping + 1o-lW-1-0 "(uu (u) is contraction (because W-1 is contraction). by the Banach fixed-point theorem, the mapping + 2 W- (u) admit unique fixed-point in (D)", which implies that 41 1/2.
(Surjectivity) Since a is surjective, is enough to show that a + Wu + ((u) + b is surjective. Let E (D)). Since the mapping u - W-1 W- W-1 ((u) contraction, by Banach fixed-point theorem, there is u² E (D)" such that
ur -w - K((') = Wa K(x)(())))))) xb=
E.4 Examples for Proposition
Example 4. We consider the case of n = 1, and D C [0, We consider Volterra operators
K(u)(x) / k(x,y, v. u(x), u(y))
where r = (Z xa) and = (y1,..., Ya). We recall that K is Volterra operator if و for all j = 1.2......
k( v, t. #
(E.4)
In particular, when D = (a, C R is an interval, the Volterra operators are of the form K(w)(x) =
[ k(z, u. u())
and if is considered as a time variable, the Volterra operators are causal in the sense that the value of K (x) the time depends only uly) at the times y S.
Assume that k(s,, y, 1) C(D D IR) is bounded and uniformly Lipschitz smooth in the I and variables, that is, C(D D. C (R R)).
Next, we consider the non-linear operator (D) + IZ(D).
F(u) = u =K)uuu.
(E.5)
Assume that u, w E (D) are such that u + K(u) = w + K(w), so that w - u = K(u) K(w) Next, we will show that then w. denote and D(21) = D n ((0,2) 21 x [0, (d-)) Then for E D(z1) the Volterra property of the kernel implies that
[u(x) - -0(=-(())) < / [k(2, y. (())),(()))))))) - k(x, y. w(x).
<
|K v, (x) (())))(v)) k(a, v. w(x),
D(2)
[k(,,,, u())) k(x, v, m (z).
D(a))
1k(2, w())),,ww)))) w(v))((()))))))))))))))))))))) - k(x, w()), (())
D(a))
< $ --
30

---

so that all 0 < 21 <t
1|u w W(lEA(D(EL))
= =fo
.... So (J²
(Jo...
(()) dx2 do
- 21 11 2|/lllcp==moou<xx)))l 2HMeeppmoooreeepppp-oooooooooooe walea//llluuuluuuuooslllllluuee
$ 2 (1Klle | +bbllloooyylllllllooooooooorrrrr ||u -
Thus, when 21 so small that
21 ( (I (MllsooDDD <1,
find that I|u - L(DG)()) O, that is, u(x) w())- for D(21). Using the same
arguments as above, we see for all k E N that that u in D(kzı) then u w in D( + 1)21). Using induction, we see that w D. Hence, the operator + F(u) is injective in L (2))).
Example 5. We consider derivatives of Volterra operators the domain C [0, Let K (D(D) + LD) be non-linear operator
K(u)= = S k( u(v)))(y)dy.
(E.6)
where k(r, 1) satisfies E.4 (D) be
bounded, and k E C(D D; D.CUIIR R)). Let Fi L²D
Fi(u) = u + K(u).
(F.7)
Then the Fréchet derivative of K at up LD) to the direction L²) is
Du = w(x) / k (1 vo(v))w(y)dy,
(E.8)
where
ka (1 mo(u)) = uo(u) k(z,v,t)
+ k( v, vo(v)
(F.9)
at
(=
a Volterra operator satisfying la (1, y, #
Yi s xy for all j 1, ...,.... ...... d. As seen in Example 4 the operator DFil L2(D) L2(D) injective.
(B.
F Details in Section 4.2
In this appendix, we prove Theorem We recall that in the theorem, weconsider the case when n and D C R is a bounded interval, and the operator F, is of the form
(i) ((x)u(x)+ / k(a, u. u(y)
where W E (D) satisfies 0 < C1 < ((x) S C2 and the function (, + k(x, y, s) is in
CD x D x R) and in D x D R its three derivatives and the derivatives of W are uniformly bounded by Co. that is.
|kl|o>(xxDx) So,, | co.
(F.I)
31

---

We recall that the identical embedding H (D) - (D) is bounded and compact by Sobolev's embedding theorem.
As we will consider kernels k(x, y, uo())), we will consider the non-linear operator Fi mainly as an operator in a Sobolev space H²
The Frechet derivative of F1 at to direction w, denoted Aug w DF lai (a (w) is given by
Aug w = (x))w(()) + J. k( U. uo(v))w(()dd + 1. uoly) ou
Ok (x, uolv)) (F.2)
The condition F.11 implies that
F -
(F.3)
is a locally Lipsichitz smooth function and the operator
Aug (D) - (D):
given in F.2 is defined for all uo C(D) as a bounded linear operator.
When X' is Banach space, let (0, R) = {ur E X:: ||l||x < R) and (x = fu E X ||l||x I R} be the open and closed balls in X, respectively.
We consider the Hölder spaces C (D) and their image in (leaky) ReLU-type functions. Let and (s) (s) ReLU(-s). We will consider the image of the closed ball of C (D) in the map aa. that is o. (L.a (0, R)) {oa og C(D): Ilollc.-(D) RJ.
We will below assume that for all un C(D) the integral operator
Auo: H (D) (D) is an injective operator.
(F.4)
This condition is valid when ((u) is Volterra operator, see Examples 4 and 5 As the integral operators Aug are Fredholm operators having index zero, this implies that the operators (F.4) are bijective.
The inverse operator A=1 DD) (D) can be written
A = ((x)u(x)) - S Timo (1,)))())d
(F.5)
where Kun: Kukiun E CC D) and W (D).
We will consider the inverse function of the map Fi in a set y calBein(D)) (0. R)) that is compact subset of the Sobolev space (D). To this end, we will cover the set ) with small balls (D) (g). Eu), = 2,2.....J of H' (D), centred at g, FIş)). where by E H' (D). We will show below that when g (D) (gj, 25p), that is, g is 2e1-close to the function Gj in (D),, the inverse map of F, can be written as a limit (FT (g).g) = limm- (15((;; g) in H HD)2), where Fi (v;)) )) + A,, 93)
u - A7 (
Hi 4u
that is, near 9j we can approximate FT1 as composition Hmm of 2m layers of neural operators.
To glue the local inverse maps together, we use partition of unity in the function space ) that is given by integral neural operators
(2 w) dr,1 o 07,2 ..... oi (u, w), where (1, (F (T..... (v,w), w),
where / belongs in some finite index set C Z some 6 > 0, some y E D (f 1,... to). s(7, = / (v, w) = that maps a pair (v, w) to the first function Here, Fz. (v, w) are integral neural operators with distributional kernels
(e, (x) = S, k ke. (x, y. u(x), w(y))dy.
32

---

where (x,, v. v(r), w(y)) = v(z)1{(- th,s ( ( (w()))6 - 2). and 1,4 is the indicator function
of set A and y 8(y 2) is the Dirac delta distribution at the point D. Using these, write the inverse of Fi at g E aa
(T) lim Σ
(0) 3(0) 9
(F.6)
m--000
SEZ
where j(i) E f1,2...., J are suitably chosen and the limit is taken in the norm topology of /1' (D). This result is summarized in the following theorem, that modified version of Theorem for for the inverse operator FT- in (F.6) where we have refined the partition of unity 4 that we indexes Z I 0 instead of j E J/..
This result is summarized in following theorem:
Theorem 4. Assume that Fi satisfies the above assumptions [.I and (F4) and that Fi: (D) +
(D) is bijection. Let y C on (B(Bc..< Rt) be a compact subset of the Sobolev space and 0. Then the inverse of F1 HD) (D) in can written limit (.6) that is, as a limit of integral neural operators.
(D), where
Observe that Theorem 4 includes the case where a 1, in which case da Id and y C oa (B ( (0, R)) B (D) (0, R)). We note that when is leaky ReLU-function with parameter > 0, Theorem
g can be applied to compute the inverse of 0a Fi that is given by AT' o o- where --- 01/a- Note that the assumption that y C oa (B
(0, Rt) makes it possible to apply Theorem ijnn the case when one trains deep neural networks having layers da Fi and the parameter a of the leaky Rel.U-function a free parameter which also trained.
Proof. As the operator F1 can be multiplied by function Wxx) -! |, it is sufficient to consider the case when (xx) =
Below, the fact that as D C R, Sobolev's embedding theorem yields that the embedding (D) C(D) is bounded and there is Css 0 such that
[|llcc()) S Cs||u]|an<D)
(F.7)
For clarity, we denote below the norm of C(D) by [|]]L=<(D)-
Next we considre the Frechet derivatives of F1. We recall that the 1st Frechet derivative of Fj at wo the operator Aup The 2nd Frechet derivative Fi at up directions w1 and w2
/
2 Ok Du (a, v. uo (oly)) (w)zz()))))) f uo(y) Ouz (r, y, mo(u)) (v) (v)
Ok
D2 Filus (W1, W2)
/ plz, p (y)
=
where
Ok
Ok.
p(x,8) =
(x, y, uo(y)) + uo(v) (z. y. uo uo(y)),
(F.8)
Du
ou
and
0
821:
0h (I, (r,y, vo())) + vo(y) ar (x, y. uo(v))
p(z,y)
= 2
(F.9)
Ox
Juor
Thus,
(F.10)
||D (w), w2) (
<
When we freeze the function u in kernel & to ug, we denote
Ky u(x) = J k(r, v. uo(v))u(y)dy,
33

---

Lemma 4. For uo. E C(D) we have
11K,, - (101 & (Dl)) - (D)
and
||Au; - Ac01L1(())-=1110) & (11 +
| |o/l uoll (F.1I)
Proof. Denote
Muo vxx) J uv(y) 0k du
(x, (o()))oyy)d)
Nug,o v(x) S. (1) (, Ou
Ōk
We have
Mu2 -M,," (Naz,us Na 2) + (N v Nuruu
By Schur's test for continuity of integral operators,
& sup / [k(2,, mo(v))ddy) ZED
1/2 sup vED.
1/2
[K (((())))11()))))
|k(2, uo(v)))))))
and
|M ll (((()))))()))
Ok sup / |ualv)
(x, no mo(w))ld)))
1/2
sup yED
|uo(y)
Ōk (x. un (v)) dx
1/2
Du
ou
and
|kuz Kin (12(())(11)))))))))))))))
/ [k(x, 1. uz()) k(x, y. u1(v))ldy)
1/2
< sup
zED
x sup / Ik (l1,y,u2(00 uz()) k(0, y, un (v)))de MED
1/2
< <(IMOODDDDD / (uz(v) - un
1/2
1/2
(IKllo)/DDDX)) sup
(22(3) un (x)) dz
VED
| (/Inato) (1) (v))
1/2
1/2
(uz(y) -u -1)()))
sup WED
1/2
1/2
(DI|1/1/1u (
uille3(D) (101/)/uu-ulll)))))
[ sup (uz(y) - uz vED
[k||)DDDXX 1///4|/uz - un 1112//DD
112 un (+()
1/2
(DD(O DIl U2 un -ulle><0)
34

---

and
Nuga, - (
/ sup or DEEDJA
1/2
[12(v)k(z, u2(1) w2 (dy)
x
sup (u2lt) (z())((>, u, (2))
MED
( wn 1n ())))()))))))))))())) lll2 11111/2200
1/2
Dileloo- (D)
and
|| Naga - Nu 1 ((()))
(sup / (wz(v)-u - (u))) (v))
<
SEn
x sup / ((2(z)) - - (w)) v. (v))
1/2
yED
so that
[Ma, Ma, Mc
s [bIlc + | "Ille>(D)-
Also, when D₂
do de
Dr Kun KunllLe(D)-1100))
1/2
1/2
< sup DDDJD
(Dkk(,
sup yED
DDzk(a, uo(v)) dx)
< [Xllx(DDDXX)
and
Dr, Kn, D2 (mm)l1(())-1(()))
s sup / D
1/2
DDk((,, y, v, D2k(a, mo(v))ldy)
x sup / [Dak(z, (v)) - (,, uo(v)) ven
1/2
< (Il / |ul(y) - vo(u))dy
1/2
1/2
(I sup
sup DDD D
|n [1) uo(v)) dx
< | (1, (ui(w) mo(v))ldy mo(v))l)
1/2
1/2
|ui(v) - (o(())) (w)))
sup WED D
1/2
1/2
-uolla(D)) )
[D] sup D
ln (1(1)) un uo(w)))
[x(c=()xXXX) (|)c<D)xx))D))//11,, 11 4 uoll!?DD || up (D(D)
1/2
s (klc=(DxD<R) DII 4n - 1D)
Thus.
K n0 (L(())HH(
35

---

and
(
s (ullxoop)|/lccxxxxxxx))
and
[ Kanll13(D)- [M]cc(xxxxx]]illl -olle><DD)
Similarly,
|| Mu Muo (LA((D)))HH()) I |Mccapxxxx + |uellc(()))] (oalloo)))))(((+////////o)))))) (D)
As Au, = Ku, + Mu, the claim follows.
As the embedding (D) - C(D) is bounded and has norm Cs. Lemma 4 implies that for all R > 0 there is
Cr. (= + CsR),
such that the map,
Up ト Duol uy B (0, R),
(F.12)
Lipschitz map B (0, R) L(H (D)) with Lipschitz constant C2 (R), that is,
| 1u - DF lea |l (D) s (x))ll - (2)
(F.13)
As Mg + Auo DFilus is continuous, the inverse A=1 : H' + H (D) exists for all Wo E C(D), and the embedding (D) C(D) is compact, we have that for all R > 0 there is (R) > 0 such that
A Il | S CB()), for all uo E Bgp (0, R).
(F.14)
Let R1, R2 > 0 be such that y C B (0, R1) and x = OT) c B (0, R2). Below, we denote C Cz= and CB = (R2)..
Next we consider inverse of Fi in y. To this end, let us consider Ep > 0, which we choose later to be small enough. As y BBg(00 (0, R) compact there are finite number of elements gj Fi (vz) )
where 1y E X. .,, such that
y c U (0) j=1
We observe that for uo, E X,
A
A A (An, Auo)
and hence the Lipschitz constant of A- u+ A 1 L(H (D), H (D)) satisfies
Lip(A-1) < CA C3CL,
(F.15)
see
Let us consider fixed j and gj E V. When satisfies
llg - g))l < 250.
(F.16)
the equation
Fi = g, u XE
is equivalent to the fixed point equation
u = u - - - (i;))) + A,, 93),
that is equivalent to the fixed point equation
H, (u)
36

---

for the function : + (D),
11flw) = u ( Fi(w))) + Ar, (9- 95). Note that H2 depends on g. and thus we later denote H = H. We observe that
Hy(0;) - U/ Ao, 93)-
(F.17)
Let u, E BH We have
Fi(u) = Fi + A. - u) + B. b), |B - w|ll s Collu - ull?,
where, see F.1O).
Co = 3 1/2 (1
so that for in U2 E B (0, 2R2),
81 U2 A (Fi()) - Fi(42))
U1 U2 An (Fi(U) Fi(u2)) -AA - (P(()()( - Fi(u2)),
and
[u] u2 - A (Fi - (12))))l
|4 (Bug (u1 - u2)) H²(D)
=
A (D) Bug (w1 u2)
A DD uellt (
M CgCo -11 - u2ll u2ll?
and
[1 (F) -=(2))ll))
|| A7, (p (w)))llmm)
/1 |, - un (D))
L Lips (.. (A
- -a
4Cab -o (D)
see F.2 and hence, when ||-1 u, Sr Ra,
(u)( (,(u2)l(='(D)
| 02 (R) - (12))))
I u2 (F (12))))) + -7)) - (1 H)))
- us -uilan()) CnColllu - 1 (D)) + 1l12 )))lm))) + Ca(CCn 4CoR2)lluu will) 1
Chrllug un (I))
where
Cn = 2C + CC 4C
We now choose
min( 1
R2). 2C
We consider
en < 1 1
80 2C11"
Then, we have
r > 2C -CHr).
37

---

Then, we have that LipT n² (1) (H2) 1 a = Chr < 2.2 and
r || H (D) l1 - و مامlا - a),
and for all u B (0, R2) such that | uylه S r, we have || - (1-)((9-)))lmmo & (1 a)r. Then,
[1,(u) My (D))
(u) (D) + (1) 1)]lm(D)
1n allu By ()) + 11u, A., (g 93) my)lm(D)
M ar + I14,, (g 9)) (D) (r) r,
that is, H2 maps (D) (1g,r) to itself. By Banach fixed point theorem, H3 B (B((D(()))))) B (D) (0g,r) has a fixed point.
Let us denote
H₂ (
u ) 100001
H)( g
Fi(e,)) ( "T )
A 9;) )
9
By the above, when we choose En to have a value
1
< 8CB 2CH²
the map Fg has right inverse map R, By (gj. 25u), that is,
(i( = 9. for B (gj. 25p).
(F.18)
it holds that R, Bu (gj, 250) + BH(DD (1g, r), and by Banach fixed point theorem it is given by the limit
R₂(9) lim Wjm:: g Bmigg 220).
(F.19)
in (D), where
Wj.0 = Uj
(F.20)
Wj.m-1 H (w..m)
(F.21)
We can write for g E By (g); 200);
R₂(g)
)
U1 ) g
lim H3m m=+00
9
where the limit takes space in H H (D)² and
H H, Hy o... H,,
(F.22)
is the composition m operators Hy. This implies that R, can be written as a limit finite iterations of neural operators H3 (we will consider how the operator An can written as a neural operator below).
As y C. oa (B(B.r. R)), there are finite number of points yı E D. 1,2, Lo and E > 0 such that the sets
1 Z(i 12 ... ie.) = y: (is ( S g(ve)= < (ie + 7)8,, for all f),
where i,, dg..........................dd ........
in ......., Z, satisfy the condition
(F.23)
If 12,, de, ()) Brr (g,, EO) # then Z(i,, i2. ... .e, ny C BH 25a).
To show (.23). we will below use the mean value theorem for function
a₂ E Y, where # E Cl. (D). First, let us consider the case when the parameter a of the leaky ReLU function o is strictly positive. Without loss of generality,
assume that D [0, and ye hl, where
38

---

h = 1/6 and é= = 0.1.....44 to We consider E 3 n Z(11, ie, C oa Cr (0, R)) of the form g = 0a o v. As a non-zero, the inequality (ir - 1) S g(vr) < (ie + t) is equivalent to 01/aa(it )= S v(ye) < Pija((ie + }),, and thus
o /1aa(tt)) As z w(ye) (iee( + Ae,
(F.24)
where A max(1, a, 1/0), that is, for g = (v) Z(i i2...., de,) the values v(ge) are known within small errors. By applying mean value theorem on the interval [[E 1 1),, (h) for function v we see that there is a E 1),, th) such that
de (x') v(th) v((E - 1)h)
dr
and thus by (F.24).
du (x') (x') de,t s $2A h²
(F.25)
dr
where
de, h (o1 01/a( 1)=1))
(F.26)
Observe that these estimates are useful when E much smaller that h. As g = on oo E C
de aa ( (D)) (0, R)), we have E B BD) (0, R), so that du E B (D) (0, R) satisfies (.25
implies that
de (z) de,t S 2A E1
+ Rh for all æ E [(e Eh].
(F.27)
dæ
h
Moreover, (.24 and E B B (D) (0, R)) imply
[b(x) - σ /i (bte))) oia
< AE + Rh,
(F.28)
for x E [(e- - 11),, Eh).
Let E2 A./A. When we first choose Eg to be large enough (so that h = 1/6o is small) and then & be small enough, we may assume that
E1 + RIC AEi + Rh) <
1
max
82.
(F.29)
8
Then for any two functions 9.9 E y n Z(i, i2, i2. 1 C oa ( (B, Rt) of the form
g oa v. g' v' the inequalities F.27 and F.28 imply
du (1) dr (5:) < -2.
du'
(F.30)
dx |o(x) v' r'(x:) < 4 62
for x E D. us d' E e Bc.. (0, R). this implies
1
||u v' Il (D)
2
F2.
As the embedding (D) + (D) is continuous and has norm less than on the interval D [0, see that
|| - v' ('llms < <
and thus
|gg I'lmm < A62 = En-
(F31)
To prove 1223
assume that (Z(11, iz,.... :ta) n) n (oj.) # 0, and g Z(i iz. ...... ie, nny. By the assumption, there exists 9* E (Z(in, i2.... ( (.) Using (3I1), we have
lgg 9i9l (D)) S $11g g" (' + 11g" 9)llT(D) $ 2
39

---

Thus, g E (Ir 250), which implies that the property F23 follows.
We next consider the case when the parameter of the leaky relu function 0a is zero, Again, we assume that D = [0, 1] and = he, where h = 1/6o and & = 0, 1....... la- We consider g E y n Ja (v) and an interval [41,, (6+1 C D, where 1 < f < 8o 2. We will consider four cases. First, if g does not obtain the value zero on the interval [2 h, (41 + 1)] the mean value theorem implies that there is x' E [G,,, (61 + 1)h] such that 1)h))/h. Second, if g does not obtain the value zero either of the intervals [(1 ((111)h,, 1)h, h1h] or [(2 1)h, (61 + 2)h.. we can use the mean value theorem to estimate the derivatives of g and at some point of these intervals similarly to the first case. Third, if g does not vanish identically on the interval h, (1)+1)hl but obtains the value zero the both intervals [41 - 1)h, 614] and (f1 + 1),, (41 + 2)h]. the function v has two zeros on the interval [(G: 1)h, (f1 2)h] and the mean value theorem implies that there is x' E [(6] 1)h, (6 2)h] such that du (a')
(.1......... (te) ] C 0 (B. (0, R)) of the form
dg (x') = de (x') is equal to d= (g(Eah) - 9(((1]
da
zz
O. Fourth, if none of the above cases are valid, 9 vanishes identically the interval h,, (e, + 1)h]. In all these cases the fact that [Iullct.( (T) S R implies that the derivative of g can be estimated on the whole interval [e h. (41 + 1)h] within small error. Using these observations we see for any 62,63 > that if ye E D [d,,dg] C R, L 1, 1,2,... .......6. .. are a sufficiently dense grid in D and E1 to be small enough, then the derivatives of any two functions 9.9' E E n Z(11, .2.......... C oa ( (D) (0, R)) of the form 9 = (1()), g oo(c') satisfly lgg g' g'lllx((a (ea.2aa))--a))) E2- As the embedding C2 ([d1 + d2 -3)) + H ([d1 + E3, d2 E3))
de
is continuous,
[o..())llr ((er ca G Vēa.
[Vo,(v)] ((d2-e,,)))) S Ca
Vēg.
and E2 and Ey can be chosen to be arbitrarily small, we see that the property FF23 follows. Thus the property (F23) is shown in all cases.
By our assumptions y C aal (B (0, R)) and hence g E y implies that | I AR. This implies that )/ Z( ((1, .2, dea) is empty if there is l such that Liel > 2AR//11 + 1. Thus, there is a finite set I C t such that
..,,
y U z(i).
(F.32)
TET
z ny /
for all T
(F.33)
where notation / = (f,, i2 ... it)) E Zhe On the other hand, we have chosen gj y such that (1, .}} such that there exists g E z(7) n (D) (0), By F23 this implies that
= .,...,, cover V. This implies that for all 7 E Z there is j = ji) E
z(i) (I) 25o).
(F.34)
Thus, see that Z(i), E I is disjoint covering of y, and by F.34 in each set Z(i) Te.Ter the map g + (g) we have constructed a right inverse of the map F1.
Below, we denote s(ī, E) = isE1. Next we construct partition of unity in y using maps
F (1, /
/
where
k2, (,0)), = u(x) thst (1 - 2).
Then,
(1, v(x), if - S S w(z) < hh,
O, otherwise.
Next, for all E I we define the operator
4 HD(D) x y HD))
11(r, w) m Φे. के. Di w),
40

---

where : x > - (D) x y' are the maps
Φi w) (F (T,, (v, w) w),
and IT (v. w) = maps pair (v, w) to the first function u. It satisfies
87(u. w) = { v, if 22 w(ye) s 2 for all e
0, otherwise.
Observe that here s(7, () = igg is close to the value ( (ye). Now we can write for g E
(r) = Σ
22 (()
FE
with suitably chosen j(1) E ( J}.
Let us finally consider A_d where un C(D.. Let us denote
Kug w = J, uo uo(v)
Ōk (z, y. uo(y))
Du
and Juo Ku Kuo be the integral operator with kernel
Ok
Juo (x, v) = k(r, U. mo()) + uo(y) Du
(x, y. mo(v))
We have
(I + - Juo + Jno (1 + Juo)= Jnor
so that when we write the linear bounded operator
A.i = (1 + Jun) (D) - HD))).
integral operator
(I+ + - -1(())= =0+ [ /. mus (x,, y)
we have
(I+ + Jun) =o()
v(x) Jun v(x)
+/ J Jno (Zx, of "io (v.0') + (Ss jno (I,, of mooo (gl D') (a', yydr') } dy ely)dy
u(x) / (x,
where
Fam (I, u) jun (I, 3) + / (Jro (y. ddy + / / Jus (x,, ym (V. V x')i (2' uddd dy'
This implies that the operator A-l = (I + Juo)- is neural operator, too. Observe that Fng (x, v), OrJan (2, CTD D).
This proves Theorem 3
41


Reviews:
[{"review_id": "nSgZhqXdEq", "reviewer": "Reviewer_kfSm", "summary": "This study investigates the injectivity and bijectivity of neural operators (NOs) in the infinite-rank setting which is less investigated than their finite-rank counter, such as invertible flow networks. This study is based on the finite-rank analysis by Puthawala et al. (2022a). As a previous work, Alberti et al. (2022) have shown a global injectivity of an infinite-rank NN based on wavelet expansion. The NO in consideration is formulated as a composite of hidden layers of the form $(\\mathcal{L}v)(x) := \\sigma( T(v)(x) + b(x) )$. In Section 2, the iff conditions for linear NOs are stated. Particularly Prop 2 (DSS condition for ReLU) is shown by extending the finite-rank results by Puthawala et al. (2022a), and Prop 3 (for bijective activation) is shown based on Fredholm theory. In Section 3, $cc$-universalities of (Theorem 1) injective linear NOs, and (Theorem 2) injective finite-rank NOs are shown. It is remarkable that different from finite-rank results, Theorem 1 does not require any assumption between input and output dimensions. In Section 4, a nonlinear NO with each layer being of the form $\\sigma \\circ F_1(u) = \\sigma(Wu + K(u))$, which covers attention mechanisms, is considered, (Props 3 and 4) sufficient conditions of the surjectivity and bijectivity are stated using the Leray-Schauder fix point theorem, and (Theorem 3) the inverses of bijective nonlinear NOs are constructed.", "questions": "please refer to comments in the Weaknesses section.\n", "limitations": "n/a", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Modern deep learning tasks tend to be formulated in the *infinite-dimensional* setup, and the authors established a mathematical foundation for a *general class of NOs*.\n", "weaknesses": "(Minor comments)\n- I believe that the NOs in consideration cover a wide range of practical examples, but it would broaden their potential readers if the authors could explicitly showcase such examples (eg., not just ones mentioned in l.252 and Example 1, but also subnetworks, operator transformers, and integral autoencoders, etc...)\n- Definition 1 would be clearer if the description in ll.77-80 by sentence “and $T_\\ell$ : … are sums of …” is associated with an expression such as $T_\\ell = …$.\n- Some citations in ll.339-341 are duplicated.\n- Additional literature (not mandatory as these are in finite-dimensional settings\n  - https://proceedings.neurips.cc/paper/2020/hash/2290a7385ed77cc5592dc2153229f082-Abstract.html\n  - https://arxiv.org/abs/2204.07415\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable comments, constructive feedback, and endorsement. We have addressed your feedback below\n\n\n_I believe that the NOs in consideration cover a wide range of practical examples, but it would broaden their potential readers if the authors could explicitly showcase such examples (eg., not just ones mentioned in l.252 and Example 1, but also subnetworks, operator transformers, and integral autoencoders, etc...)._\n\nThis is a great suggestion. Just after l.39, we have included more examples of model applications including graph neural operators, IAE-nets, Operator transformers, and Factorized Fourier Neural operators. We have cited applications where neural operators are used as well, especially in the context of inverse problems where injectivity is a critical property.\n\n_Definition 1 would be clearer if the description in ll.77-80 by sentence “and_  $T_\\ell$ : … are sums of …” _is associated with an expression such as_ $T_\\ell =$.\n\nWe agree that this change will make the description clearer. We have included such a change in the new version.\n\n_Some citations in ll.339-341 are duplicated._\n\nThe duplicated reference have been removed.\n\n_Additional literature (not mandatory as these are in finite-dimensional settings)._\n\nWe intended to cite these papers to help fill out the story in finite dimensions but, in an oversight, did not. We have included reference to them in the related work section, and referenced them again the section on universality."}, {"review_id": "aeWXjgRFdz", "reviewer": "Reviewer_GKdz", "summary": "This paper provides a theoretical analysis of the injectivity and surjectivity of neural operators.\n\nSection 2 discusses the injectivity of a single layer of neural operators. For the ReLU activation, the injectivity of the layer is characterized by the directed spanning set. On the other hand, if the activation is bijective, the injectivity of the layer is equivalent to the injectivity of its constituent.\n\nSection 3 of the paper examines how injectivity can be preserved when composing layers. The paper shows that integral neural operators with L^2-integral kernels possess a universal approximation property for continuous operators. A truncated series expansion-based approximation, implementable for this family, is demonstrated to maintain the universal approximation property.\n\nSection 4 considers nonlinear integral operators and presents a sufficient condition for a layer in this class to be bijective. Additionally, this document proposes a method for constructing the inverse of a nonlinear integral neural operator layer. The inverse is expressed as a limit of integral neural operators.", "questions": "Please kindly correct me if any of my understandings in the above comments are incorrect.\n\n- Line 32: Should \"infinite-dimension setting\" be \"infinite-rank setting\"?\n- Line 33: What does \"infinite-rank Sobolev space\" mean? I have not been able to find a definition of the rank of Sobolev spaces either in this paper or elsewhere.\n\n[minor suggestions]\n\n- Line 27 “a operators” → “an operator”\n- Line 21 “Bayesian UQ” → “Bayesian uncertainty quantification” (if UQ stands for that)\n- Line 23 “existence and uniqueness” → “existence and uniqueness of the solutions”\n- Line 47 “and their implementation” → “and that their implementation”\n- Line 48 “universality approximation” → “universal approximation”", "limitations": "None in particular.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "[originality]\n\n- As far as I understand, this paper is the first to provide a rigorous framework for analyzing the injectivity and bijectivity of neural operators.\n\n[quality]\n\n- The paper is written with mathematical rigor, and there are no noticeable flaws in the logical development. However, I have not been able to check the proofs in detail.\n\n[clarity]\n\n- The sections have clear purposes, with concise explanations of the goals in each section.\n\n[significance]\n\n- This paper presents universality results and identifies conditions under which injectivity or bijectivity can be guaranteed. These theoretical foundations can improve the models and facilitate their application by increasing their reliability and enhancing understanding.", "weaknesses": "[originality]\n\n- None in particular\n\n[quality]\n\n- The paper contains a noticeable number of grammatical and typographical errors. It is recommended that an up-to-date automatic checker is used to revise the paper and correct these issues.\n- The model classes, including layers, can be assigned specific symbols to make them stand out in the paper.\n- Line 33 mentions \"infinite-rank Sobolev spaces,\" but I could not find the definition of the notion of rank of a Sobolev space in the paper or elsewhere on the internet.\n\n[clarity]\n\n- Throughout the introduction, it remains unclear what is meant by \"finite-rank case\" and \"infinite-rank case.\" This issue may be resolved if the definition of the rank of Sobolev spaces is provided or if it is made clear that it refers to the rank of the target operator.\n\n[significance]\n\n- The theoretical results presented in the paper do not appear to have any immediate tangible implications for the models or learning algorithms of neural operators.\n- The motivation of the paper needs to be elaborated as it is currently unclear why it is important to understand the conditions under which the neural operators have injectivity or bijectivity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review, suggestions, and endorsement. We have address your feedback below\n\n_The paper contains a noticeable number of grammatical and typographical errors. It is recommended that an up-to-date automatic checker is used to revise the paper and correct these issues._\n\nWe apologize for any and all grammatical and typographical errors. We reviewed all of the text and typography and fixed and removed all errors.\n\n_The model classes, including layers, can be assigned specific symbols to make them stand out in the paper._\n\nWe appreciate any suggestions that improve clarity, and want to make sure that we understand the suggestion exactly, and hoped to get more clarification on this point. Is the suggestion to use a specific symbol to represent an injective versus non-injective layer? For example, $\\mathcal L^{\\textrm{inj}}$ for the injective case and $\\mathcal L^{\\textrm{bijec}}$ for the bijective case?\n\n_Line 33 mentions \"infinite-rank Sobolev spaces,\" but I could not find the definition of the notion of rank of a Sobolev space in the paper or elsewhere on the internet._\n\n The prefix infinite-rank was meant purely for emphasis, but we agree that it is confusing and the emphasis isn't needed. We have removed it.\n\n_Throughout the introduction, it remains unclear what is meant by \"finite-rank case\" and \"infinite-rank case.\" This issue may be resolved if the definition of the rank of Sobolev spaces is provided or if it is made clear that it refers to the rank of the target operator._\n\nIn order to make this distinction clearer, we have added an additional sentence at the start of the introduction that frames and defined the finite-rank and infinite-rank cases, and draws attention to their essential differences. We think this has made the distinction clearer, and makes the work more accessible.\n\n_The theoretical results presented in the paper do not appear to have any immediate tangible implications for the models or learning algorithms of neural operators._\n\nIt is true that our work is principally theoretical and does not make concrete recommendations for e.g. learning algorithms, but we think it has useful application and takeaways. Mainly by giving an algebraic perspective on neural operators, and by laying the ground work for rigorous application of Bayesian inversion via neural operators. We elaborate on these points more in the global comment.\n\nAdditionally, the proof of Theorem 1, that establishes universality of injective operators, proceed by `giving a recipe' for construction an arbitrarily good injective approximator. This recipe can be followed in application, and so may be a guide for injective approximation. We have included statements to this effect just after the statement of Theorem 1.\n\n_The motivation of the paper needs to be elaborated as it is currently unclear why it is important to understand the conditions under which the neural operators have injectivity or bijectivity._\n\nWe realized that our motivation could be made even stronger. We have done just this in the global comment above.\n\n_minor suggestions_\n\nThank you for the suggestions. All of the suggested changes have been made. \n"}, {"review_id": "wolQInBNiq", "reviewer": "Reviewer_KyUH", "summary": "The authors present theoretical results in the field of operator learning, specifically dealing with operators are injective and surjective. They build on existing finite-dimensional work and consider the infinite-dimensional case of learning mappings between infinite-rank Sobolev spaces. The paper contributes several theoretical results in this context, including characterizing the conditions for layer-wise injectivity and bijectivity given certain activation functions, universal approximation results for injective linear neural operators, and sufficient conditions for surjective/bijective nonlinear integral operators. The paper lays the groundwork for analysis of learning injective and bijective operators in the infinite-dimensional setting. ", "questions": "Line 107: Is it possible to give a more intuitive description of Definition 2 and why it is called a directed spanning set?\n\nLine 234: Not sure why it is a natural conclusion from using the sup norm that the approximation does not smooth out non-smooth operators. Can you clarify this comment?\n\nLine 215: This is a very interesting remark, can you imagine other orthonormal bases and discuss whether they would be worth formulating as a neural operator architecture?\n\nLine 249: What does it mean for integral transform with kernel the attention mechanism in a transformer to have be injective, practically/in a particular problem? ", "limitations": "This theoretical work naturally pertains to tasks where injectivity and bijectivity are desirable, however neural operators are of course used in much broader contexts. As the authors discuss, their contribution is largely theoretical and an important next step for this work would be explicit constructions of injective neural operators/inverses that are good approximators. ", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper has several significant theoretical results and answers many natural questions one would have about the theory of injectivity and bijectivity in neural operators. Examples and practical implementation results in finite-rank cases are also described in the paper and appendix, which is helpful and grounds the theoretical results in practice. The writing is very clear and the proofs are clean. Overall, it is a very thorough paper.", "weaknesses": "Just a note on writing, too many proofs and examples are black-boxed and put in the appendix. At least a description of various proof techniques in the main text would be helpful. The paper would also benefit from more motivation of injective/bijective models in downstream tasks/generative modeling, describing any particular successes or failures different activation functions have had, etc. More discussion on how this theory should guide practice would also improve the significance of the paper.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review, suggestions, and strong endorsement. Please find answers to your questions and feedback below.\n\n_Just a note on writing, too many proofs and examples are black-boxed and put in the appendix. At least a description of various proof techniques in the main text would be helpful._\n\nWe find it difficult to balance including sufficient 'teasers' for the proofs of the main results against other page-length considerations. We realized, in light of your comment, that the proof of Theorem 2 lacked intuition. We would therefore add a few sentences giving some intuition about the main proof ideas just after line 231.\n\n_The paper would also benefit from more motivation of injective/bijective models in downstream tasks/generative modeling, describing any particular successes or failures different activation functions have had, etc. More discussion on how this theory should guide practice would also improve the significance of the paper._\n\nThis point is well taken. We could have done a better job of more motivation and application discussion. In our global comment at the top, for some more discussion that we'd like to include.\n\n_Line 107: Is it possible to give a more intuitive description of Definition 2 and why it is called a directed spanning set?_\n\nYes, it is possible. To make the definition more digestible, and explain the notes, we would like to add the following text just after line 110, between Def. 2 and Prop. 2. This addition would give some intuition for the $\\ker(T|_{S(v,T+b)})$ term of Eqn. 2.1.\n\n\"The name directed spanning set arises from the kernel term of Eqn. (2.1). The indices of $S(v,T + b)$ are those that are directed (positive) in the direction of $v$. If $T$ restricted to these indices together span $L^2(D)^n$, then the kernel term is $\\{0\\}$, and the condition automatically satisfied. Hence, The DSS conditions measures the extend to which the set of indices which are directed w.r.t. form a span of the input space of L.\"\n\n_Line 234: Not sure why it is a natural conclusion from using the sup norm that the approximation does not smooth out non-smooth operators. Can you clarify this comment?_\n\nThe issue of smoothening out nonsmooth operators was a feature of some prior work that is not present in our work, but this point wasn't clearly made in Remark 1. We would like to modify the content of Remark 1 to make this point more clear. Please see our proposed new text for Remark 1 below.\n\n\"Observe that in our finite-rank approximation result, we only require that the target function G+ is continuous and bounded, but not smooth. This differs from prior work that requires smoothness of the function to be approximated.\"\n\n_Line 215: This is a very interesting remark, can you imagine other orthonormal bases and discuss whether they would be worth formulating as a neural operator architecture?_\n\nWe're glad that you found it interesting! Your follow up question is interesting as well. We can't think of any other NO architecture that can be formed by a particular choice of basis. We would, however, like to include the following remark just after line 217 to make this point more clear. \n\n\"Lemma 2 and Remark 3 in the appendix C.3 give a 'recipe' to construct the projection $B$ such that the composition $B \\circ T$ (interpreted as augmenting finite rank neural operator $T$ with one layer $B$) is injective. \nThe projection $B$ is constructed by using an orthogonal sequence $\\{\\xi_k\\}_k$ subject to the condition (3.1), which that does not over-leap the range of $T$.\nThis condition is automatically satisfied for any orthogonal base $\\{\\varphi_k\\}_k$.\nThat is, if we find an orthogonal sequence $\\{\\xi_k\\}_k$ that is 'easy' to compute, then, we can construct 'easy' projection $B$.\nThis could yield practical implications in guiding the choice of the orthogonal basis ${\\varphi_k}_k$ for the neural operator's design.\"\n\n_Line 249: What does it mean for integral transform with kernel the attention mechanism in a transformer to have be injective, practically/in a particular problem?_\n\nAppealing to the analogy that the attention mechanism allows a layer to 'focus' on particular pieces of a signal, an injective attention mechanism could be considered one that `allows focus, but not blindspots.' Here, a 'blindspot' means a (complete) loss of information.\n"}, {"review_id": "mMi1udMbsG", "reviewer": "Reviewer_ZQ9v", "summary": "The paper considers the question of when neural operators, which have infinite-dimensional inputs and outputs, are injective and bijective. This question is answered in quite some generality in different settings and under different assumptions.", "questions": "Quite a few questions were already asked in the section on Weaknesses. The authors should address them.", "limitations": "--- No solid rationale is provided for the whole premise of the paper.\n\n--- A lot of abstract theory with few examples, if any. Practical utility of the \nconcept is totally unclear from the current version.\n\n--- Unclear if the framework survives contact with reality in the form of finite-dimensional\nrepresentation of functions (for instance sampling on a uniform grid) that can lead to\naliasing errors.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "--- A careful and at times, deep, analysis for the questions under consideration is provided.", "weaknesses": "1. Relevance and Scope: While appreciating the depth of the functional analysis that is presented, this reviewer is left a bit perplexed about the rationale behind all this heavy machinery. The authors do not really motivate why one needs bijective neural operators in the first place. There is some boilerplate on generative models in the discussion but this does not occupy any centerstage. If the fundamental question is itself not posed properly, the interest of the subsequent analysis becomes rather limited. The authors should clearly motivate why they consider these questions in the first place and explain it well to the reader. \n\n2. Clarity: The paper has excessively abstract notation and presentation. It needs to be reorganized to bring out the main contributions well. Here are some specific questions about the clarity: \n\n    \na) Why introduce Directed spanning sets in Definition 1 (which is almost impossible to follow as no intuition is provided into what it is) and Proposition 1, when immediately afterwards you have Proposition 2 which provides the answer for a bijective activation function. Who cares about ReLU when you already have a result on Leaky ReLU which is heavily used to begin with. \n\n  b) Similarly what is the use of the local layer wise analysis when you anyway consider the global end-to-end neural operator in Section 3 ? \n\n  c) In Lemma 1, what does condition 3.1 even mean in practice ? How can one check it ? \n\n  d) The authors have to concede that having a universal approximation theorem does not mean much, see Lanthaler et al and Kovachki et al 2021a where this issue of universal approximation is critiqued as the system size can grow (even double) exponentially. So having a universal approximation theorem does not imply any kind of efficient approximation but it is a necessary condition at best. \n\n  e) Does section 3.3 imply that FNO and WNO are injective and bijective neural operators ? If not, why not ? If yes, under what conditions ? \n\n  f) Can you find a non-trivial example for section 4.1, apart from Example 1 which appears to be trivial. Has anyone ever implemented example 1? If so, what is its empirical performance. \n\n  g) What is the rationale for section 4.2, why would I want to explicitly compute the inverse of the neural operator -- in what situations is that useful ? Can you give an example ?\n\nAll the questions clearly indicate that the paper is not well-written and needs a substantial rewrite. \n\n3. Concreteness: The main limitation in my view is that lack of exemplification of the results, except for example 1 -- the authors should give many more examples of practical utility, with conditions on the weight matrices, kernels as well as activation functions so that either available or de novo neural operators fall into their theoretical framework. This will enable the readers to better understand the significance of this work. \n\n4. Novelty: In many places in the paper, the authors refer to Puthawala et al 2022 -- how does the current paper differ from this reference ? A thorough elaboration of these differences is essential for judging the novelty of this paper. \n\n5. Finite-dimensional representations and Aliasing: The authors always assume that one has access to functions as both inputs and outputs. This is far from the case and in practice, one has to work with finite-dimensional representations of the functions, see for instance Fanaskov and Oseledets, Spectral neural operators, 2022 for a discussion on this issue. This use of finite-dimensional representation can lead to what are called aliasing errors. For instance, FNO has aliasing errors -- how do these errors affect your theoretical considerations -- in particular, does aliasing destroy the bijection property of the neural operator. Addressing this issue is crucial for any relevance for the theory, particularly in the construction of inverse neural operators", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for your comments, careful eye, and fair criticisms.\n\n_Relevance and Scope_\n\nWe should have motivated the application of injectivity and invertibility more. We will, please see the global comment.\n\n_Why introduce...begin with._\n\nYou make a good point. The for these two cases back-to-back is to use the ReLU activation to 'build a bridge' from the Euclidean case to infinite dimensions, and draw contrasts between them. We did not do a good enough job making this parallel clear, and so we will include a sentence at line 92 saying the following.\n\n\"Deriving a condition for layer-wise injectivity with bijective activation functions is trivial in a finite dimensional setting. With a ReLU activation function it requires the so-called Directed Spanning Set (DSS) condition. This condition is not automatic, but will hold with high probability for random weight matrices if they are expansive enough.\n\nIn this section we derive a generalization of the DSS condition and show that it is much more restrictive than the infinite dimensional setting. We then present a less restrictive condition that is met when the activation function is bijective, e.g. a leaky-ReLU activation is used.\"\n\n_Similarly what...Section 3?_ \n\nAlthough it would appear that the end-to-end condition supersedes the layer-wise one, it is not the case. The layer-wise case is less restrictive and has different applications & implications then global result does not. We didn't do enough to make this apparent in the manuscript. Therefore, we would like to add the following sentences just after Line 97.\n\n\"Although it may appear that the end-to-end result is strictly stronger than the layerwise result, this is not the case. The layerwise result is an exact characterization, whereas the end-to-end result is sufficient for injectivity, but not necessary. The layerwise analysis is also constructive, and so gives a rough guide for the construction of injective networks, whereas the global analysis is less so. Finally, the layerwise condition has different applications, such as network of stochastic depth, see e.g. [Huang et al. 2016] or [Benitez et al. 2023]. End-to-end injectivity by enforcing layerwise injectivity is straight forward, whereas deriving a sufficient condition for any depth is more daunting.\"\n\n_In Lemma...check it ?_\n\nTo make condition 3.1 clearer, and to provide a 'hands on' example, we would like to include the following  Condition 3.1 has a straight-forward interpretation. We think that this example provides a reprieve from the abstraction, and gives a simpler takeaway.\n\n\"Lemma 1 and Eqn. 3.1 may be interpreted as saying that if _some_ orthonormal sequence $\\\\{\\xi_k\\\\}\\_{k \\in \\mathbb{N}}$ exists that doesn't overlap the range of $T$, then $T$ may be embedded in a small space without losing injectivity. As an example of such a $\\\\{\\xi_k\\\\}\\_{k \\in \\mathbb{N}}$, if the range $\\mathrm{Ran}(T)$ of $T$ is included in the continuous function space (for instance, a finite rank neural operator with continuous basis like FNOs), then we may choose $\\\\{\\xi_k\\\\}\\_{k \\in \\mathbb{N}}$ to be a discontinuous basis and the condition is automatically met.\"\n\n_The authors...at best._\n\nIndeed, having a universal approximation theorem is only a starting point for showing efficient approximation but, we argue, even such a starting point wasn't achieved until recently in the finite-dimensional case. The proof of our universal approximation result is compatible with future efficient (quantitative) approximation results that may arise for neural operators. To make this point clear, we would like to include the following sentences just after Line 185.\n\n\"The proof for universal approximation theorem is constructive. If, in the future, efficient approximation bounds for neural operators are given, such bounds can likely be used directly in our universality proof to generate corresponding efficient approximation bounds for injective neural operators.\"\n\n_Does section...what conditions?_\n\nIn short, it does give such conditions. We have (abstract) conditions that apply perfectly well to FNO or WNO. It all depends on the choice of basis. It is clear that we did not do a good enough job of drawing attention to this in the main text. Therefore we would like to modify lines 218 - 221 to say the following.\n\n\"We show that Propositions 1, 2 (characterization of layerwise injectivity/bijectivity), and Lemma 1 (global injectivity) all have natural analogues for finite rank operator $K_{\\ell,N}$ in Proposition 6 and Lemma 3 in Appendix C. These conditions applies out-of-the-box to both Fourier Neural Operators (FNO) and Wavelet Neural Operators (WNO). We also show the universal approximation in the case of finite rank approximation.\"\n\n_Can you...empirical performance._\n\nMotivated by your comment, we derived another four examples that illustrate non-triviality of the results in Section 4.1. The entire derivation and proof is worked out, but is too long to include in this rebuttal. Please see the global comment for the setup of these examples.\n\n_What is...an example?_\n\nThere are two main rationales for Section 4.2, some abstract involving the algebra of groups and other involves justifying the discussion of operator encoder-type networks. We feel we didn't do a good enough job of bring them to the fore. Therefore, we would like to add the following sentences just after Line 294.\n\n\"The proof that neural operators may be inverted with other neural operators provides a theoretical justification for Integral Auto Encoder networks (IAE-nets, Ong et al. 2022) where an encoder/decoder pair that parallel the roles of the finite-dimensional VAEs (Kingma & Welling 2023). This section proves that the decoder half of a IAE-net is provably able to inverse the encoder half. Our analysis also shows that injective differential operators (as arise in PDE) and integral operator encoders form a formal algebra under operator composition.\""}, {"review_id": "3vYRIoPefv", "reviewer": "Reviewer_GL1W", "summary": "This paper extends several known results on ReLU networks from finite dimensional domains to the more challenging infinite dimensional domains. Namely:\n(a) Conditions for injectiviity of infinite dimensional ReLU networks are provided\n(b) Universality of infinite dimensional ReLU networks was proven\n(c) Extension to network with non-linear kernels are provided\n", "questions": "To convince me, mostly rebuttal should focus on importance of the problems discussed in the paper.\n\nI note some minor comments and typos I saw while reading the paper:\nLine 7 `the case the case'\nLine 34 'on its face' not sure that is an expression in english\nLine 43, 53 114 and throughout `the equivalent condition' should be `an equivalent condition'\nDefinition 1 should be more explicit IMHO: What are bias functions? What do the linear kernels do? Better to just right out the formula\nLine 105 be *the* ReLU activation", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "Strengths: From a mathematical standpoint the paper proves some natural and  non-trivial theorems ", "weaknesses": "I think this paper is not appropriate for this venue: it is not clear to me why the ML world should care that much about infinite dimensional ReLU operators, and the authors do not make an effort to explain this. There is certainly room for purely theoretical papers in Neurips and there are many such papers. However, I believe that, even if the proofs are above the paygrade of most Neurips attendants, it should be at least clear to said attendants why the questions are interesting. I do not feel this is the case here    ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for your comments and careful eye.\n\n_I think this paper is not appropriate for this venue: it is not clear to me why the ML world should care that much about infinite dimensional ReLU operators, and the authors do not make an effort to explain this. There is certainly room for purely theoretical papers in Neurips and there are many such papers. However, I believe that, even if the proofs are above the paygrade of most Neurips attendants, it should be at least clear to said attendants why the questions are interesting. I do not feel this is the case here._\n\nWe could have done a better job of valorizing the results, and emphasizing that our analysis goes far beyond the ReLU case. The ReLU activation function is of interest only in section 2.1. We think that our results, properly valorized, are well-suited for NeurIPS not because the paygrade of the proofs, but because the results are practical and important. That said, it is clear from your comment and others, that our paper still needs more motivation. Therefore, we would like to include some more applications of injectivity \\& invertibility. For more details, please see the global comment to all reviewers. We hope that including those comments will better motivate the focus on injectivity/invertibility, and address your criticisms.\n\n_I note some minor comments and typos I saw while reading the paper: Line 7 the case the case'_\n\nThank you for pointing this out. It has been addressed.\n\n_Line 34 'on its face' not sure that is an expression in english Line 43_\n\nWe have replaced the phrase 'on its face' with the phrase 'on first inspection.'\n\n_Line 43, 53 114 and throughout the equivalent condition' should be `an equivalent condition.'_\n\nAll have been fixed, thank you.\n\n_Definition 1 should be more explicit IMHO: What are bias functions?_\n\nWe have made Definition 1 more explicit by adding equation definition of $T_\\ell$, .\n\nBias functions are analogous to bias vectors. We believe that their meaning will become clear by defining $T_\\ell$ more explicitly.\n\n_What do the linear kernels do?_\n\nA linear kernel is a kernel in the sense of a convolution. That is, they are the filter of a convolution. That is, an operator of the form $v \\mapsto \\int_\\Omega k(x - y) v(y) dy$, where $x$ is the independent variable. A nonlinear kernel is an operator of the form $v \\mapsto \\int_\\Omega k(x,y,v(x),v(y))v(y) dy$ where again $x$ is independent. Note that this map is no longer linear in $v$.\n\n_Better to just right out the formula Line 105 be the ReLU activation._\n\nYou are right, that is more clear. This change has been made."}]

Author rebuttal:
We appreciate all the reviewers' valuable comments, and close attention.
We have a few things that we would like to say globally, in response to points brought up by multiple reviewers. We have also replies to individual reviewers below.

Several reviewers remarked that the question of injectivity for neural operators could be better motivated. We agree. To make the motivation of the work stronger, we would like to add the two remarks to the paper. The two remarks will in the introduction just after Line 39, and will then be briefly restated in the conclusion.


(1) The first remark forms a connection between the algebra of injective/invertable neural operators, and pseudodifferential operators. Pseudodifferential operators revolutionized the theory of linear PDE and so we believe that making such a connection is valuable, and lays the groundwork for an algebraic study of neural operators.

``Our work draws parallels between neural operators and pseudodifferential operators [Taylor, Princeton Mathematical Series 1981], a class that contains many inverses of linear partial differential operators and integral operators. The connection to pseudodifferential operators provided an algebraic perspective to linear PDE [Kohn \& Nirenberg, Communications on Pure and Applied Mathematics 1965, Shubin, Springer 1987]. An important fact in the analysis  of pseudodifferential operators, is that the inverses of certain operators, e.g. elliptic pseudodifferential operator, are themselves pseudodifferential operator. By proving an analogous result in section 4.2, that the inverse of invertible NO are themselves given by NO, we draw an important and profound connection between (non)linear partial differential equations and NO. 
    ''

(2) The second described the application of injective neural operators to Bayesian inverse problems. In short, there are a lot of ways to incorrectly discretize inverse problems that makes Bayesian methods discretization dependent. Injective and invertible neural operators are natural ways to do this correctly.

"There are significant benefits to applying Bayesian solution methods to inverse and imaging problems in infinite dimensions. This, for example, allows one to study functions of continuous space \& time variables. These infinite dimensional models can then be approximated by a finite dimensional model without losing discretization invariance, see [Stuart, Acta Numerica 2010].  Crucially, discretization must be done `at the last possible moment,' or else performance degrades as a discretization becomes finer, see [Lassas-Siltanen, Inverse Problems 2004] and also [Lassas-Saksman-Siltanen, Inverse Problems in Imaging 2009]. By formulating machine learning problems in infinite dimensional function spaces and then approximating these methods using finite dimensional subspaces, we avoid bespoke ad-hoc methods and instead obtain methods that apply to any discretization.

One example is the following. Let $\mathcal M$ be the submanifold of $X=L^2([0,1]^2)$ or $X=L^2([0,1]^3)$ corresponding to natural images or 3D medical model. Let $K\subset \mathbb{R}^D$ be a manifold with the same topology as $\mathcal M$, $\iota:\mathbb{R}^D\to X$ be an embedding, and define Let $K_1=\iota(K)\subset \mathcal M$. Given $\mu$, a measure supported on $M$, the task is to find a neural operator $f_\theta\colon X\to X$ that maps (pushes forward) the uniform distribution on the model space $K_1$ to $\mu$ and so thus maps $K_1$ to $\mathcal M$. If $f_\theta\colon X\to X$ is bijective, computing likelihood functions in statistical analysis is made easier via the change of variables formula. Further, we may interpret $f_\theta^{-1}$ as encoder and $f_\theta$ the corresponding decoder, which parameterized elements of $\mathcal M$. As everything is formulated in infinite dimension function space $X$, we obtain discretization invariant methods.''

Motivated by the remarks of Reviewer ZQ9v, we would also like to include the following examples as other functions for which the machinery developed in Section 4.1 applies. We think that these addition flesh out the applications of that section.

The below four points are all other functions, for which we may prove 

(A) We can obtain surjectivity (by illustrating coercivity) of neural operators of the form $F(U) = Wu + \sigma_2(K (u))$  where $\sigma_1$ is bijective, $\sigma_2$ bounded. 

(B) We can obtain surjectivity (by illustrating coercivity) of neural operators of the form $F(U) = \alpha u + K (u)$ where $K(u) = \int_D a(x,y,u(x),u(y))dy$ where $a(x,y,s_1,s_2)$ is continuous and is such that $\exists R > 0$, $c_1 < \alpha$ so that for all $|(s_1,s_2)| > R$, $\mathrm{sign}(s_1)\mathrm{sign}(s_2)a(x,y,s_1,s_2) \geq -c_1$.

(C) In imaging applications layer wise coercivity of neural networks have been a useful property, see for example [Li et al., Inverse Problems, 2022].

(D) We may also show that coercivity is preserved by perturbations in a bounded domain. This makes it possible to study non-linear and non-positive perturbations of physical models. For example, in quantum mechanics when a non-negative energy potential $|\phi|^4$ is replaced by a Mexican hat potential $-C|\phi|^2 + |\phi|^4$, as occurs in the study of magnetization, superconductors and the Higgs field.

Meta-review: The paper analyzes the injectivity and bijectivity of neural operators. Injectivity is important when input has to be recovered from outputs such as encoding and decoding in autoencoders.  The main contribution is characterizing sufficient conditions under which the neural operator is bijective. This is an extension of Puthawala et al to infinite-dimensional settings. Reviewers agree that this paper is theoretically sound. While there were concerns about the lack of motivation and examples, the rebuttal provides more details and examples. Concerns regarding aliasing and weight matrices can be included in discussions as limitations. For the camera-ready version, the writing has to be improved based on reviews. In particular, the following changes are required: 
- A solid introduction that better motivates readers to follow theoretical findings. A short intro to neural operators will make the paper more accessible. Furthermore, the authors need to more elaborate on connections to Puthawala's paper. There are interesting arguments in rebuttal that can be added to the introduction.
- Motivation behind injectivity requires more elaboration. Authors need to give specific applications that rely on injectivity and bijectivity with more reasoning. 
- Sections 4.1 and 4.2 need further discussions. In particular, it is important to address the concerns of reviewer ZQ9v
- Include limitations such as aliasing and squared weight matrices in discussions
Decision: Accept (poster)