Title: Globally injective and bijective neural operators
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Learning

Operator Learning

Functional Analysis

Injectivity

Bijectivity

Universal approximation
Abstract: Recently there has been great interest in operator learning, where networks learn operators between function spaces from an essentially infinite-dimensional perspective. In this work we present results for when the operators learned by these networks are injective and surjective. As a warmup, we combine prior work in both the finite-dimensional ReLU and operator learning setting by giving sharp conditions under which ReLU layers with linear neural operators are injective. We then consider the case when the activation function is pointwise bijective and obtain sufficient conditions for the layer to be injective. We remark that this question, while trivial in the finite-rank setting, is subtler in the infinite-rank setting and is proven using tools from Fredholm theory. Next, we prove that our supplied injective neural operators are universal approximators and that their implementation, with finite-rank neural networks, are still injective. This ensures that injectivity is not 'lost' in the transcription from analytical operators to their finite-rank implementation with networks. Finally, we conclude with an increase in abstraction and consider general conditions when subnetworks, which may have many layers, are injective and surjective and provide an exact inversion from a 'linearization.’ This section uses general arguments from Fredholm theory and Leray-Schauder degree theory for non-linear integral equations to analyze the mapping properties of neural operators in function spaces. These results apply to subnetworks formed from the layers considered in this work, under natural conditions. We believe that our work has applications in Bayesian uncertainty quantification where injectivity enables likelihood estimation and in inverse problems where surjectivity and injectivity corresponds to existence and uniqueness of the solutions, respectively.

Globally injective and bijective neural operators

Abstract

Recently there has been great interest in operator learning, where networks learn operators between function spaces from an essentially infinite-dimensional perspective. In this work we present results for when the operators learned by these networks are injective and surjective. As a warmup, we combine prior work in both the finite-dimensional ReLU and operator learning setting by giving sharp conditions under which ReLU layers with linear neural operators are injective. We then consider the case when the activation function is pointwise bijective and obtain sufficient conditions for the layer to be injective. We remark that this question, while trivial in the finite-rank setting, is subtler in the infinite-rank setting and is proven using tools from Fredholm theory. Next, we prove that our supplied injective neural operators are universal approximators and that their implementation, with finite-rank neural networks, are still injective. This ensures that injectivity is not 'lost' in the transcription from analytical operators to their finite-rank implementation with networks. Finally, we conclude with an increase in abstraction and consider general conditions when subnetworks, which may have many layers, are injective and surjective and provide an exact inversion from a 'linearization.' This section uses general arguments from Fredholm theory and Leray-Schauder degree theory for non-linear integral equations to analyze the mapping properties of neural operators in function spaces. These results apply to subnetworks formed from the layers considered in this work, under natural conditions. We believe that our work has applications in Bayesian uncertainty quantification where injectivity enables likelihood estimation and in inverse problems where surjectivity and injectivity corresponds to existence and uniqueness of the solutions, respectively.

1 Introduction

In this work, we produce results at the intersection of two fields: neural operators (NO), and injective and bijective networks. Neural operators [Kovachki et al., 2021a,b] are neural networks that take a infinite dimensional perspective on approximation by directly learning an operator between Sobolev spaces. Injectivity and bijectivity on the other hand are fundamental properties of networks that enable likelihood estimation by the change of variables formula, are critical in applications to inverse problems, and are useful properties for downstream applications.

The key contribution of our work is the translation of fundamental notions from the finite-rank setting to the infinite-rank setting. By the 'infinite-dimension setting' we refer to the case when the object of approximation is a mapping between Sobolev spaces. This task, although straight-forward on first inspection, often requires dramatically different arguments and proofs as the topology, analysis and notion of noise are much simpler in the finite-rank case as compared to the infinite-rank case. We see our work as laying the groundwork for the application of neural operators to generative models

in function spaces. In the context of operator extensions of traditional V AEs [Kingma and Welling, 2013], injectivity of a decoder forces distinct latent codes to correspond to distinct outputs.

Our work draws parallels between neural operators and pseudodifferential operators Taylor [1981], a class that contains many inverses of linear partial differential operators and integral operators. The connection to pseudodifferential operators provided an algebraic perspective to linear PDE Kohn and Nirenberg [1965]. An important fact in the analysis of pseudodifferential operators, is that the inverses of certain operators, e.g. elliptic pseudodifferential operators, are themselves pseudodifferential operators. By proving an analogous result in section 4.2, that the inverse of invertible NO are themselves given by NO, we draw an important and profound connection between (non)linear partial differential equations and NO.

We also believe that our methods have applications to the solution of inverse problems with neural networks. The desire to use injective neural networks is one of the primary motivations for this work. These infinite dimensional models can then be approximated by a finite dimensional model without losing discretization invariance, see Stuart [2010]. Crucially, discretization must be done 'at the last possible moment,' or else performance degrades as the discretization becomes finer, see Lassas and Siltanen [2004] and also Saksman et al. [2009]. By formulating machine learning problems in infinite dimensional function spaces and then approximating these methods using finite dimensional subspaces, we avoid bespoke ad-hoc methods and instead obtain methods that apply to any discretization.

More details on our motivations for and applications of injectivity & bijectivity of neural operators are given in Appendix A.

1.1 Our Contribution

In this paper, we give a rigorous framework for the analysis of the injectivity and bijectivity of neural operators. Our contributions are as follows:

(A) We show an equivalent condition for the layerwise injectivity and bijectivity for linear neural operators in the case of ReLU and bijective activation functions (Section 2). In the particular ReLU case, the equivalent condition is characterized by a directed spanning set (Definition 2).

(B) We prove that injective linear neural operators are universal approximators, and that their implementation by finite rank approximation is still injective (Section 3). We note that universal approximation theorem (Theorem 1) in the infinite dimensional case does not require an increase in dimension, which deviate from the finite dimensional case Puthawala et al. [2022a, Thm. 15].

(C) We zoom out and perform a more abstract global analysis in the case when the input and output dimensions are the same. In this section we 'coarsen' the notion of layer, and provide a sufficient condition for the surjectivity and bijectivity of nonlinear integral neural operators with nonlinear kernels . This application arises naturally in the context of subnetworks and transformers. We construct their inverses in the bijective case (Section 4).

1.2 Related Works

In the finite-rank setting, injective networks have been well-studied, and shown to be of theoretical and practical interest. See Gomez et al. [2017], Kratsios and Bilokopytov [2020], Teshima et al. [2020], Ishikawa et al. [2022], Puthawala et al. [2022a] for general references establishing the usefulness of injectivity or any of the works on flow networks for the utility of injectivity and bijectivity for downstream applications, [Dinh et al., 2016, Siahkoohi et al., 2020, Chen et al., 2019, Dinh et al., 2014, Kingma et al., 2016], but their study in the infinite-rank setting is comparatively underdeveloped. These works, and others, establish injectivity in the finite-rank setting as a property of theoretical and practical interest. Our work extends Puthawala et al. [2022a] to the infinite-dimensional setting as applied to neural operators, which themselves are a generalization of multilayer perceptrons (MLPs) to function spaces. Moreover, our work includes not only injectivity, but also surjectivity in the non-ReLU activation case, which Puthawala et al. [2022a] has not focused on.

Examples of works in these setting include neural operators Kovachki et al. [2021b,a], DeepONet Lu et al. [2019], Lanthaler et al. [2022], and PCA-Net Bhattacharya et al. [2021], De Hoop et al. [2022]. The authors of Alberti et al. [2022] recently proposed continuous generative neural networks (CGNNs), which are convolution-type architectures for generating L 2 ( R ) -functions, and provided the sufficient condition for the global injectivity of their network. Their approach is the wavelet basis expansion, whereas our work relies on an independent choice of basis expansion.

1.3 Networks considered and notation

Let D ⊂ R d be an open and connected domain, and L 2 ( D ; R h ) be the L 2 space of R h -value function on D given by L 2 ( D ; R h ) := L 2 ( D ; R ) ×··· × L 2 ( D ; R ) ︸ ︷︷ ︸ h = L 2 ( D ) h .

Definition 1 (Integral and pointwise neural operators) . We define an integral neural operator G : L 2 ( D ) d in → L 2 ( D ) d out and layers L ℓ : L 2 ( D ) d ℓ → L 2 ( D ) d ℓ +1 by

where σ : R → R is a non-linear activation operating element-wise, and k ℓ ( x, y ) ∈ L 2 ( D × D ; R d ℓ +1 × d ℓ ) are integral kernels, and W ℓ ∈ C ( D ; R d ℓ +1 × d ℓ ) are pointwise multiplications with matrices, and b ℓ ∈ L 2 ( D ) d ℓ +1 are bias functions ( ℓ = 1 , ..., L ). Here, T 0 : L 2 ( D ) d in → L 2 ( D ) d 1 and T L +1 : L 2 ( D ) d L +1 → L 2 ( D ) d out are mappings (lifting operator) from the input space to the feature space and mappings (projection operator) from the feature spaces to the output space, respectively.

The layers T 0 and T L +1 play a special role in the neural operators. They are local linear operators and serve to lift and project the input data from and to finite-dimensional space respectively. These layers may be 'absorbed' into the layers L 1 and L L without loss of generality (under some technical conditions), but are not in this text to maintain consistency with prior work. Prior work assumes that d in < d 1 , we only assume that d in ≤ d 1 for lifting operator T 0 : L 2 ( D ) d in → L 2 ( D ) d 1 . This would seemingly play an important role in the context of injectivity or universality, but we find that our analysis does not require that d in < d 1 at all. In fact, as elaborated in Section 3.2, we may take d in = d ℓ = d out for ℓ = 1 , . . . , L and our analysis is the same.

2 Injective linear neural operator layers with ReLU and bijective activations

In this section we present sharp conditions under which a layer of a neural operator with ReLU activation is injective. The Directed Spanning Set (DSS) condition, described by Def. 2 is a generalization of the finite-dimensional DSS [Puthawala et al., 2022a] which guarantees layerwise injectivity of ReLU layers. Extending this condition from finite to infinite dimensions is not automatic. The finite-dimensional DSS will hold with high probability for random weight matrices if they are expansive enough ([Puthawala et al., 2022a, Theorem7]). However, the infinite-dimensional DSS is much more restrictive than the finite dimensional setting. We then present a less restrictive condition that is met when the activation function is bijective, e.g. a leaky-ReLU activation is used.

Although it may appear that the end-to-end result is strictly stronger than the layerwise result, this is not the case. The layerwise result is an exact characterization, whereas the end-to-end result is sufficient for injectivity, but not necessary. The layerwise analysis is also constructive, and so gives a rough guide for the construction of injective networks, whereas the global analysis is less so. Finally, the layerwise condition has different applications, such as network of stochastic depth, see e.g. Huang et al. [2016], Benitez et al. [2023]. End-to-end injectivity by enforcing layerwise injectivity is straightforward, whereas deriving a sufficient condition for networks of any depth is more daunting.

We denote by

where σ : R → R is a non-linear activation function, T ∈ L ( L 2 ( D ) n , L 2 ( D ) m ) , and b ∈ L 2 ( D ) m , where L ( L 2 ( D ) n , L 2 ( D ) m ) is the space of linear bounded operators from L 2 ( D ) n to L 2 ( D ) m . The aim of this section is to characterize the injectivity condition for the operator v ↦→ σ ( Tv + b ) mapping from L 2 ( D ) n to L 2 ( D ) m , which corresponds to layer operators L ℓ . Here, T : L 2 ( D ) n → L 2 ( D ) m is linear.

2.1 ReLU activation

Let ReLU : R → R be ReLU activation, defined by ReLU( s ) = max { 0 , s } . With this activation function, we introduce a definition which we will find sharply characterizes layerwise injectivity.

Definition 2 (Directed Spanning Set) . We say that the operator T + b has a directed spanning set (DSS) with respect to v ∈ L 2 ( D ) n if

where T | S ( v,T + b ) ( v ) = ( T i v ) i ∈ S ( v,T + b ) and

The name directed spanning set arises from the ker( T | S ( v,T + b ) ) term of (2.1). The indices of S ( v, T + b ) are those that are directed (positive) in the direction of v . If T restricted to these indices together span L 2 ( D ) n , then ker( T | S ( v,T + b ) ) is { 0 } , and the condition is automatically satisfied. Hence, the DSS condition measures the extent to which the set of indices, which are directed w.r.t. v , form a span of the input space.

Proposition 1. Let T ∈ L ( L 2 ( D ) n , L 2 ( D ) m ) and b ∈ L 2 ( D ) m . Then, the operator ReLU ◦ ( T + b ) : L 2 ( D ) n → L 2 ( D ) m is injective if and only if T + b has a DSS with respect to every v ∈ L 2 ( D ) n in the sense of Definition 2.

See Section B for the proof. Puthawala et al. [2022a] has provided the equivalent condition for the injectivity of ReLU operator in the case of the Euclidean space. However, proving analogous results for operators in function spaces require different techniques. Note that because Def. 2 is a sharp characterization of injectivity, it can not be simplified in any significant way. The condition restrictive Def. 2 is, therefore, difficult to relax while maintaining generality. This is because for each function v , multiple components of the function Tv + b are strictly positive in the entire domain D , and cardinality | S ( v ; T + b ) | of S ( v ; T + b ) is larger than n . This observation prompts us to consider the use of bijective activation functions instead of ReLU, such as leaky ReLU function, defined by σ a ( s ) := ReLU( s ) -a ReLU( -s ) where a > 0 .

2.2 Bijective activation

If σ is injective, then injectivity of σ ◦ ( T + b ) : L 2 ( D ) n → L 2 ( D ) m is equivalent to the injectivity of T . Therefore, we consider the bijectivity in the case of n = m . As mentioned in Section 1.3, an significant example is T = W + K , where W ∈ R n × n is injective and K : L 2 ( D ) n → L 2 ( D ) n is a linear integral operator with a smooth kernel. This can be generalized to Fredholm operators (see e.g., Jeribi [2015, Section 2.1.4]), which encompasses the property for identity plus a compact operator. It is well known that a Fredholm operator is bijective if and only if it is injective and its Fredholm index is zero. We summarize the above observation as follows:

Proposition 2. Let σ : R → R be bijective, and let T : L 2 ( D ) n → L 2 ( D ) m and b ∈ L 2 ( D ) m . Then, σ ◦ ( T + b ) : L 2 ( D ) n → L 2 ( D ) m is injective if and only if T : L 2 ( D ) n → L 2 ( D ) m is injective. Furthermore, if n = m and T ∈ L ( L 2 ( D ) n , L 2 ( D ) n ) is the linear Fredholm operator, then, σ ◦ ( T + b ) : L 2 ( D ) n → L 2 ( D ) n is bijective if and only if T : L 2 ( D ) n → L 2 ( D ) n is injective with index zero.

We believe that this characterization of layerwise injectivity is considerably less restrictive than Def. 2, and the characterization of bijectivity in terms of Fredholm theory will be particularly useful in establishing operator generalization of flow networks.

3 Global analysis of injectivity and finite-rank implementation

In this section we consider global properties of the injective and bijective networks that constructed in Section 2. First we construct end-to-end injective networks that are not layerwise injective. By doing this, we may avoid the dimension increasing requirement that would be necessary from a layerwise analysis. Next we show that injective neural operators are universal approximators of continuous functions. Although the punchline resembles that of [Puthawala et al., 2022a, Theorem 15], which relied on Whitney's embedding theorem, the arguments are quite different. The finite-rank case has dimensionality restrictions, as required by degree theory, whereas our infinite-rank result does not. Finally, because all implementations of neural operators are ultimately finite-dimensional, we present a theorem that gives conditions under which finite-rank approximations to injective neural operators are also injective.

3.1 Global analysis

By using the characterization of layerwise injectivity discussed in Section 2, we can compose injective layers to form L L ◦ · · · L 1 ◦ T 0 , a injective network. Layerwise injectivity, however, prevents us from getting injectivity of T L +1 ◦ L L ◦ · · · L 1 ◦ T 0 by a layerwise analysis if d L +1 > d out , as is common in application [Kovachki et al., 2021b, Pg. 9]. In this section, we consider global analysis and show that T L +1 ◦ L L ◦ · · · L 1 ◦ T 0 , nevertheless, remains injective. This is summarized in the following lemma.

Lemma 1. Let ℓ ∈ N with ℓ < m , and let the operator T : L 2 ( D ) n → L 2 ( D ) m be injective. Assume that there exists an orthogonal sequence { ξ k } k ∈ N in L 2 ( D ) and a subspace S in L 2 ( D ) such that Ran( π 1 T ) ⊂ S and

where π 1 : L 2 ( D ) m → L 2 ( D ) is the restriction operator defined in (C.1). Then, there exists a linear bounded operator B ∈ L ( L 2 ( D ) m , L 2 ( D ) ℓ ) such that B ◦ T : L 2 ( D ) n → L 2 ( D ) ℓ is injective.

See Section C.1 in Appendix C for the proof. T and B correspond to L L ◦ · · · L 1 ◦ T 0 (from lifting to L -th layer) and T L +1 (projection), respectively. The assumption (3.1) on the span of ξ k encodes a subspace distinct from the range of T . In Remark 3 of Appendix C, we provide an example that satisfies the assumption (3.1). Moreover, in Remark 4 of Appendix C, we show the exact construction of the operator B by employing projections onto the closed subspace, using the orthogonal sequence { ξ k } k ∈ N . This construction is given by the combination of "Pairs of projections" discussed in Kato [2013, Section I.4.6] with the idea presented in [Puthawala et al., 2022b, Lemma 29].

3.2 Universal approximation

We now show that the injective networks that we consider in this work are universal approximators. We define the set of integral neural operators with L 2 -integral kernels by

and

The following theorem shows that L 2 -injective neural operators are universal approximators of continuous operators.

Theorem 1. Let D ⊂ R d be a Lipschitz bounded domain, and G + : L 2 ( D ) d in → L 2 ( D ) d out be continuous such that for all R > 0 there is M > 0 so that

We assume that either (i) σ ∈ A L 0 ∩ BA is injective, or (ii) σ = ReLU . Then, for any compact set K ⊂ L 2 ( D ) d in , ϵ ∈ (0 , 1) , there exists L ∈ N and G ∈ NO inj L ( σ ; D,d in , d out ) such that

See Section C.3 in Appendix C for the proof. For the definitions of A L 0 and BA , see Definition 3 in Appendix C. For example, ReLU and Leaky ReLU functions belong to A L 0 ∩ BA (see Remark 5 (i)).

We briefly remark on the proof of Theorem 1 emphasizing how its proof differs from a straightforward extension of the finite-rank case. In the proof we first employ the standard universal approximation theorem for neural operators ([Kovachki et al., 2021b, Theorem 11]). We denote the approximation of G + by ˜ G , and define the graph of ˜ G as H : L 2 ( D ) d in → L 2 ( D ) d in × L 2 ( D ) d out . That is H ( u ) = ( u, ˜ G ( u )) . Next, utilizing Lemma 1, we construct the projection Q such that Q ◦ H becomes an injective approximator of G + and belongs to NO L ( σ ; D,d in , d out ) . The proof for universal approximation theorem is constructive. If, in the future, efficient approximation bounds for neural operators are given, such bounds can likely be used directly in our universality proof to generate corresponding efficient approximation bounds for injective neural operators.

This approach resembles the approach in the finite-rank space Puthawala et al. [2022a, Theorem 15], but unlike that theorem we don't have any dimensionality restrictions. More specifically, in the case of Euclidean spaces R d , Puthawala et al. [2022a, Theorem 15] requires that 2 d in +1 ≤ d out before all continuous functions G + : R d in → R d out can be uniformly approximated in compact sets by injective neural networks. When d in = d out = 1 this result is not true, as is shown in Remark 5 (iii) in Appendix C using topological degree theory [Cho and Chen, 2006]. In contrast, Theorem 1 does not assume any conditions on d in and d out . Therefore, we can conclude that infinite dimensional case yields better approximation results than the finite dimensional case.

This surprising improvement in restrictions in infinite-dimensions can be elucidated by an analogy to Hilbert's hotel paradox, see [Burger and Starbird, 2004, Sec 3.2]. In this analogy, the orthonormal bases { φ k } k ∈ N and Ψ j,k ( x ) = ( δ ij φ k ( x )) d i =1 play the part of guests in the hotel with N floor, each of which as d rooms. A key step in the proof of Theorem 1 is that there is a linear isomorphism S : L 2 ( D ) d → L 2 ( D ) (i.e., a rearrangement of guests) which maps Ψ j,k to φ b ( j,k ) , where b : [ d ] × N → N is a bijection.

3.3 Injectivity-preserving transfer to Euclidean spaces via finite-rank approximation

In the previous section, we have discussed injective integral neural operators. The conditions are given in terms of integral kernels, but such kernels may not actually be implementable with finite width and depth networks, which have a finite representational power. A natural question to ask, therefore, is how should these formal integral kernels be implemented on actual finite rank networks, the so-called implementable neural operators? In this section we discuss this question.

We consider linear integral operators K ℓ with L 2 -integral kernels k ℓ ( x, y ) . Let { φ k } k ∈ N be an orthonormal basis in L 2 ( D ) . Since { φ k ( y ) φ p ( x ) } k,p ∈ N is an orthonormal basis of L 2 ( D × D ) , integral kernels k ℓ ∈ L 2 ( D × D ; R d ℓ +1 × d ℓ ) in integral operators K ℓ ∈ L ( L 2 ( D ) d ℓ , L 2 ( D ) d ℓ +1 ) has the expansion

where C ( ℓ ) k,p ∈ R d ℓ +1 × d ℓ whose ( i, j ) -th component c ( ℓ ) k,p,ij is given by

Here, we denote ( u, φ k ) ∈ R d ℓ by ( u, φ k ) = ( ( u 1 , φ k ) L 2 ( D ) , ..., ( u d ℓ , φ k ) L 2 ( D ) ) . By truncating by N finite sums, we approximate L 2 -integral operators K ℓ ∈ L ( L 2 ( D ) d ℓ , L 2 ( D ) d ℓ +1 ) by finite rank operator K ℓ,N with rank N , having the form

The choice of orthonormal basis { φ k } k ∈ N is a hyperparameter. If we choose { φ k } k as Fourier basis and wavelet basis, then network architectures correspond to Fourier Neural Operators (FNOs) [Li et al., 2020b] and Wavelet Neural Operators (WNOs) [Tripura and Chakraborty, 2023], respectively. We show that Propositions 1, 2 (characterization of layerwise injectivity), and Lemma 1 (global injectivity) all have natural analogues for finite rank operator K ℓ,N in Proposition 5 and Lemma 2 in Appendix D. These conditions applies out-of-the-box to both FNOs and WNOs.

Lemma 2 and Remark 3 in the appendix D give a 'recipe' to construct the projection B such that the composition B ◦ T (interpreted as augmenting finite rank neural operator T with one layer B ) is injective. The projection B is constructed by using an orthogonal sequence { ξ k } k ∈ N subject to the condition (3.1), which does not over-leap the range of T . This condition is automatically satisfied for any orthogonal base { φ k } k ∈ N . This could yield practical implications in guiding the choice of the orthogonal basis { φ k } k ∈ N for the neural operator's design.

We also show the universal approximation in the case of finite rank approximation. We denote NO L,N ( σ ; D,d in , d out ) by the set of integral neural operators with N rank, that is the set (3.2) replacing L 2 -integral kernel operators K ℓ with finite rank operators K ℓ,N with rank N (see Definition 4). We define by

Theorem 2. Let D ⊂ R d be a Lipschitz bounded domain, and N ∈ N , and G + : L 2 ( D ) d in → L 2 ( D ) d out be continuous with boundedness as in (3.3). Assume that the non-linear activation function σ is either ReLU or Leaky ReLU. Then, for any compact set K ⊂ L 2 ( D ) d in ∩ (span { φ k } k ≤ N ) d in , ϵ ∈ (0 , 1) , there exists L ∈ N , N ′ ∈ N with

and G N ′ ∈ NO inj ′ ( σ ; D,d in , d out ) such that

See Section D.4 in Appendix D for the proof. In the proof, we make use of Puthawala et al. [2022a, Lemma 29], which gives rise to the assumption (3.4). We do not require any condition on d in and d out as well as Theorem 1.

Remark 1. Observe that in our finite-rank approximation result, we only require that the target function G+ is continuous and bounded, but not smooth. This differs from prior work that requires smoothness of the function to be approximated.

3.4 Limitations

We assume square matrices for the bijectivity and construction of inversions. Weight matrices in actual neural operators are not necessarily square. Lifting and projection are local operators which map the inputs into a higher-dimensional feature space and project back the feature output to the output space. We also haven't addressed possible aliasing effects of our injective operators. We will relax the square assumption and investigate the aliasing of injective operators in future work.

4 Subnetworks & nonlinear integral operators: bijectivity and inversion

So far our analysis of injectivity has been restricted to the case where the only source of nonlinearity are the activation functions. In this section we consider a weaker and more abstract problem where nonlinearities can also arise from the integral kernel with surjective activation function, such as leaky ReLU . Specifically, we consider layers of the form

where W ∈ L ( L 2 ( D ) n , L 2 ( D ) n ) is a linear bounded bijective operator, and K : L 2 ( D ) n → L 2 ( D ) n is a non-linear operator. This arises in the non-linear neural operator construction by Kovachki et al. [2021b] or in Ong et al. [2022] to improve performance of integral autoencoders. In this construction, each layer L ℓ is written as

where W ℓ ∈ R d ℓ +1 × d ℓ independent of x , and K ℓ : L 2 ( D ) d ℓ → L 2 ( D ) d ℓ +1 is the non-linear integral operator.

This relaxing of assumptions is motivated by a desire to obtain theoretical results for both subnetworks and operator transformers. By subnetworks, we mean compositions of layers within a network. This includes, for example, the encoder or decoder block of a traditional VAE. By neural operator we mean operator generalizations of finite-rank transformers, which can be modeled by letting K be an integral transformation with nonlinear kernel k of the form

where A and B are matrices of free parameters, and the (integral) softmax is taken over x . This specific choice of k can be understood as a natural generalization of the attention mechanism in transformers, see [Kovachki et al., 2021b, Sec. 5.2] for further details.

4.1 Surjectivity and bijectivity

Critical to our analysis is the notion of coercivity. Apart from being a useful theoretical tool, layerwise coercivity of neural networks is a useful property in imaging applications, see e.g.Li et al. [2020a] Recall from Showalter [2010, Sec 2, Chap VII] that a non-linear operator K : L 2 ( D ) n → L 2 ( D ) n is coercive if

Proposition 3. Let σ : R → R be surjective and W : L 2 ( D ) n → L 2 ( D ) n be linear bounded bijective (then the inverse W -1 is bounded linear), and let K : L 2 ( D ) n → L 2 ( D ) n be a continuous and compact mapping. Moreover, assume that the map u ↦→ αu + W -1 K ( u ) is coercive with some 0 < α < 1 . Then, the operator σ ◦ F 1 is surjective.

See Section E.1 in Appendix E for the proof. An example K satisfying the coercivity (4.2) condition is given here.

Example 1. We simply consider the case of n = 1 , and D ⊂ R d is a bounded interval. We consider the non-linear integral operator

The operator u ↦→ αu + W -1 K ( u ) with some 0 < α < 1 is coercive when the non-linear integral kernel k ( x, y, t ) satisfies certain boundedness conditions. In Examples 2 and 3 in Appendix E, we show that these conditions are met by kernels k ( x, y, t ) of the form

̸

where σ : R → R is the sigmoid function σ s : R → R , and a, b, c ∈ C ( D × D ) or by a wavelet activation function σ wire : R → R , see Saragadam et al. [2023], where σ wire ( t ) = Im ( e iωt e -t 2 ) and a, b, c ∈ C ( D × D ) , and a j ( x, y ) = 0 .

In the proof of Proposition 3, we utilize the Leray-Schauder fix point theorem. By employing the Banach fix point theorem under a contraction mapping condition (4.3), we can obtain bijectivity as follows:

Proposition 4. Let σ : R → R be bijective. Let W : L 2 ( D ) n → L 2 ( D ) n be bounded linear bijective, and let K : L 2 ( D ) n → L 2 ( D ) n . If W -1 K : L 2 ( D ) n → L 2 ( D ) n is a contraction mapping, that is, there exists ρ ∈ (0 , 1) such that

then, the operator σ ◦ F 1 is bijective.

See Section E.3 in Appendix E for the proof. We note that if K is compact linear, then assumption (4.3) implies that W + K is an injective Fredholm operator with index zero, which is equivalent to

σ ◦ ( W + K ) being bijective as observed in Proposition 2. That is, Proposition 4 requires stronger assumptions when applied to the linear case.

̸

Assumption (4.3) implies the the injectivity of K . An interesting example of injective operators arises when K are Volterra operators. When D ⊂ R d is bounded and K ( u ) = ∫ D k ( x, y, u ( y )) u ( y ) dy , where we denote x = ( x 1 , . . . , x d ) and y = ( y 1 , . . . , y d ) , we recall that K is a Volterra operator if k ( x, y, t ) = 0 implies y j ≤ x j for all j = 1 , 2 , . . . , d . A well known fact, as discussed in Example 4 in Appendix E, is that if K ( u ) is a Volterra operator whose kernel k ( x, y, t ) ∈ C ( D × D × R n ) is bounded and uniformly Lipschitz in t -variable then F : u ↦→ u + K ( u ) is injective.

Remark 2. A similar analysis (illustrating coercivity) shows that operators of the following two forms are bijective.

Operators of the form F ( U ) := αu + K ( u ) where K ( u ) := ∫ D a ( x, y, u ( x ) , u ( y )) dy where a ( x, y, s 1 , s 2 ) is continuous and is such that ∃ R > 0 , c 1 < α so that for all | ( s 1 , s 2 ) | > R , sign( s 1 )sign( s 2 ) a ( x, y, s 1 , s 2 ) ≥ -c 1 .

Layers of the form ( L ℓ v )( x ) := σ 1 ( Wu + σ 2 ( K ( u ))) where σ 1 is bijective and σ 2 bounded.

Finally we remark that coercivity is bounded preserved by perturbations in a bounded domain. This makes it possible to study non-linear and non-positive perturbations of physical models. For example, in quantum mechanics when a non-negative energy potential | ϕ | 4 is replaced by a Mexican hat potential -C | ϕ | 2 + | ϕ | 4 , as occurs in the study of magnetization, superconductors and the Higgs field.

4.2 Construction of the inverse of a non-linear integral neural operator

The preceding sections clarified sufficient conditions for surjectivity and bijectivity of the non-linear operator F 1 . We now consider how to construct the inverse of F 1 in a compact set Y . We find that constructing inverses is possible in a wide variety of settings and, moreover, that the inverses themselves can be given by neural operators. The proof that neural operators may be inverted with other neural operators provides a theoretical justification for Integral Auto Encoder networks [Ong et al., 2022] where an infinite encoder/decoder pair play a role parallel to those of encoder/decoders in finite-dimensional VAEs [Kingma and Welling, 2013]. This section proves that the decoder half of a IAE-net is provably able to inverse the encoder half. Our analysis also shows that injective differential operators (as arise in PDEs) and integral operator encoders form a formal algebra under operator composition. We prove this in the rest of this section, but first we summarize the main three steps of the proof.

First, by using the Banach fixed point theorem and invertibility of derivatives of F 1 we show that, locally, F 1 may be inverted by an iteration of a contractive operator near g j = F 1 ( v j ) . This makes local inversion simple in balls which cover the set Y . Second, we construct partition of unity functions Φ j that masks the support of each covering ball and allows us to construct one global inverse that simply passes through to the local inverse on the appropriate ball. Third and finally, we show that each function used in both of the above steps are representable using neural operators with distributional kernels.

As a simple case, let us first consider the case when n = 1 , and D ⊂ R is a bounded interval, and the operator F 1 of the form

where W ∈ C 1 ( D ) satisfies 0 < c 1 ≤ W ( x ) ≤ c 2 and the function ( x, y, s ) ↦→ k ( x, y, s ) is in C 3 ( D × D × R ) and in D × D × R its three derivatives and the derivatives of W are uniformly bounded by c 0 , that is,

The condition (4.4) implies that F 1 : H 1 ( D ) → H 1 ( D ) , contains locally Lipschitz smooth functions. Furthermore, F 1 : H 1 ( D ) → H 1 ( D ) is Fréchet differentiable at u 0 ∈ C ( D ) , and we denote Fréchet

derivative of F 1 at u 0 by A u 0 , which can be written as the integral operator (F.2). We will assume that for all u 0 ∈ C ( D ) , the integral operator

This happens for example when K ( u ) is a Volterra operator, see Examples 4 and 5. As the integral operators A u 0 are Fredholm operators having index zero, this implies that the operators (4.5) are bijective. The inverse operator A -1 u 0 : H 1 ( D ) → H 1 ( D ) can be written by the integral operator (F.5).

We will consider the inverse function of the map F 1 in Y ⊂ σ a ( B C 1 ,α ( D ) (0 , R )) = { σ a ◦ g ∈ C ( D ) : ∥ g ∥ C 1 ,α ( D ) ≤ R } , which is a set of the image of Hölder spaces C n,α ( D ) through (leaky) ReLU-type functions σ a ( s ) = ReLU( s ) -a ReLU( -s ) with a ≥ 0 . We note that Y is a compact subset the Sobolev space H 1 ( D ) , and we use the notations B X ( g, R ) and B X ( g, R ) as open and closed balls with radius R > 0 at center g ∈ X in Banach space X .

To this end, we will cover the set Y with small balls B H 1 ( D ) ( g j , ε 0 ) , j = 1 , 2 , . . . , J of H 1 ( D ) , centered at g j = F 1 ( v j ) , where v j ∈ H 1 ( D ) . As considered in detail in Appendix F, when g is sufficiently near to the function g j in H 1 ( D ) , the inverse map of F 1 can be written as a limit ( F -1 1 ( g ) , g ) = lim m →∞ H ◦ j m ( v j , g ) in H 1 ( D ) 2 , where

that is, near g j we can approximate F -1 1 as a composition H ◦ j m of 2 m layers of neural operators.

To glue the local inverse maps together, we use a partition of unity Φ ⃗ i , ⃗ i ∈ I in the function space Y , where I ⊂ Z ℓ 0 is a finite index set. The function Φ ⃗ i are given by neural operators

where some ϵ 1 > 0 , s ( ⃗ i, ℓ ) ∈ R are some suitable values near g j ( ⃗ i ) ( y ℓ ) , some y ℓ ∈ D ( ℓ = 1 , ..., ℓ 0 ), and π 1 is the map π 1 ( v, w ) = v that maps a pair ( v, w ) to the first function v . Here, F z,s,h ( v, w ) are integral neural operators with distributional kernels F z,s,h ( v, w )( x ) = ∫ D k z,s,h ( x, y, v ( x ) , w ( y )) dy, where k z,s,h ( x, y, v ( x ) , w ( y )) = v ( x ) 1 [ s -1 2 h,s + 1 2 h ) ( w ( y )) δ ( y -z ) , and 1 A is a indicator function of a set A and y ↦→ δ ( y -z ) is the Dirac delta distribution at the point z ∈ D . Using these, we can write the inverse of F 1 at g ∈ Y as

where j ( ⃗ i ) ∈ { 1 , 2 , . . . , J } .

This result is summarized in following theorem which is proven in Appendix F.

Theorem 3. Assume that F 1 satisfies the above assumptions (4.4) and (4.5) and that F 1 : H 1 ( D ) → H 1 ( D ) is a bijection. Let Y ⊂ σ a ( B C 1 ,α ( D ) (0 , R )) be a compact subset of the Sobolev space H 1 ( D ) , where α > 0 and a ≥ 0 . Then the inverse of F 1 : H 1 ( D ) → H 1 ( D ) in Y can written as a limit (4.6) that is, as a limit of integral neural operators.

5 Discussion and Conclusion

In this paper, we provided a theoretical analysis of injectivity and bijectivity for neural operators. In the future, we will further develop applications of our theory, particularly in the areas of generative models and inverse problems and integral autoencoders. We gave a rigorous framework for the analysis of the injectivity and bijectivity of neural operators including when either the ReLU activation or bijective activation functions are used. We further proved that injective neural operators are universal approximators and their finite-rank implementation are still injective. Finally, we ended by considering the 'coarser' problem of non-linear integral operators, as arises in subnetworks, operator transformers and integral autoencoders.

Acknowledgments

MLwas partially supported by Academy of Finland, grants 273979, 284715, 312110. M.V. de H. was supported by the Simons Foundation under the MATH + X program, the National Science Foundation under grant DMS-2108175, and the corporate members of the Geo-Mathematical Imaging Group at Rice University.

References

Giovanni S. Alberti, Matteo Santacesaria, and Silvia Sciutto. Continuous generative neural networks. arXiv:2205.14627 , 2022.

Jose Antonio Lara Benitez, Takashi Furuya, Florian Faucher, Xavier Tricoche, and Maarten V de Hoop. Fine-tuning neural-operator architectures for training and generalization. arXiv preprint arXiv:2301.11509 , 2023.

Kaushik Bhattacharya, Bamdad Hosseini, Nikola B Kovachki, and Andrew M Stuart. Model reduction and neural networks for parametric pdes. The SMAI journal of computational mathematics , 7: 121-157, 2021.

Edward B Burger and Michael Starbird. The heart of mathematics: An invitation to effective thinking . Springer Science & Business Media, 2004.

Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems , 32, 2019.

Yeol Je Cho and Yu-Qing Chen. Topological degree theory and applications . CRC Press, 2006.

Maarten De Hoop, Daniel Zhengyu Huang, Elizabeth Qian, and Andrew M Stuart. The cost-accuracy trade-off in operator learning with neural networks. arXiv preprint arXiv:2203.13181 , 2022.

Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516 , 2014.

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803 , 2016.

David Gilbarg and Neil S. Trudinger. Elliptic partial differential equations of second order . Classics in Mathematics. Springer-Verlag, Berlin, 2001. ISBN 3-540-41160-7. Reprint of the 1998 edition.

Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. Advances in neural information processing systems , 30, 2017.

Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14 , pages 646-661. Springer, 2016.

Isao Ishikawa, Takeshi Teshima, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Universal approximation property of invertible neural networks. arXiv preprint arXiv:2204.07415 , 2022.

Aref Jeribi. Spectral theory and applications of linear operators and block operator matrices , volume 9. Springer, 2015.

Tosio Kato. Perturbation theory for linear operators , volume 132. Springer Science & Business Media, 2013.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improving variational inference with inverse autoregressive flow. arXiv preprint arXiv:1606.04934 , 2016.

Joseph J Kohn and Louis Nirenberg. An algebra of pseudo-differential operators. Communications on Pure and Applied Mathematics , 18(1-2):269-305, 1965.

Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research , 22:Art-No, 2021a.

Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481 , 2021b.

Anastasis Kratsios and Ievgen Bilokopytov. Non-euclidean universal approximation. Advances in Neural Information Processing Systems , 33:10635-10646, 2020.

Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis. Error estimates for deeponets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications , 6(1):tnac001, 2022.

Matti Lassas and Samuli Siltanen. Can one use total variation prior for edge-preserving bayesian inversion? Inverse problems , 20(5):1537, 2004.

Housen Li, Johannes Schwab, Stephan Antholzer, and Markus Haltmeier. NETT: solving inverse problems with deep neural networks. Inverse Problems , 36(6):065005, 23, 2020a. ISSN 0266-5611. doi: 10.1088/1361-6420/ab6d57. URL https://doi.org/10.1088/1361-6420/ab6d57 .

Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 , 2020b.

Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193 , 2019.

Yong Zheng Ong, Zuowei Shen, and Haizhao Yang. Iae-net: Integral autoencoders for discretizationinvariant learning. arXiv preprint arXiv:2203.05142 , 2022.

Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica , 8:143-195, 1999.

Michael Puthawala, Konik Kothari, Matti Lassas, Ivan Dokmani´ c, and Maarten de Hoop. Globally injective relu networks. Journal of Machine Learning Research , 23(105):1-55, 2022a.

Michael Puthawala, Matti Lassas, Ivan Dokmanic, and Maarten De Hoop. Universal joint approximation of manifolds and densities by simple injective flows. In International Conference on Machine Learning , pages 17959-17983. PMLR, 2022b.

Matti Lassas Saksman, Samuli Siltanen, et al. Discretization-invariant bayesian inversion and besov space priors. arXiv preprint arXiv:0901.4220 , 2009.

Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Gua Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. arXiv preprint arXiv:2301.05187 , 2023.

Ralph E Showalter. Hilbert space methods in partial differential equations . Courier Corporation, 2010.

Ali Siahkoohi, Gabrio Rizzuti, Philipp A Witte, and Felix J Herrmann. Faster uncertainty quantification for inverse problems with conditional normalizing flows. arXiv preprint arXiv:2007.07985 , 2020.

Andrew M Stuart. Inverse problems: a bayesian perspective. Acta numerica , 19:451-559, 2010.

M.E. Taylor. Pseudodifferential Operators . Princeton mathematical series. Princeton University Press, 1981.

Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling-based invertible neural networks are universal diffeomorphism approximators. Advances in Neural Information Processing Systems , 33:3362-3373, 2020.

Tapas Tripura and Souvik Chakraborty. Wavelet neural operator for solving parametric partial differential equations in computational mechanics problems. Computer Methods in Applied Mechanics and Engineering , 404:115783, 2023.

Appendix

A Motivation behind our injectivity & bijectivity

PDEs-based inverse problems: We consider the following partial differential equations (PDEs) of the form

where D ⊂ R d is a bounded domain, L a is partial differential operator with a coefficient a ∈ A ( D ; R d a ) := { a : D → R d a } , and B is some boundary operator, e.g. Dirichlet or Neumann, and function f & g are fixed. When a solution u ∈ U ( D ; R d u ) := { u : D → R d a } is uniquely determined, we may define the solution operator G : A ( D ; R d a ) →U ( D ; R d u ) by putting G ( a ) := u . Note that the operator G is, in general, non-linear even if partial differential Operator L a is linear (e.g., L a = ∆ + a ). The aim of the inverse problem is to evaluate G -1 , the inverse of the solution operator. When G is non-injective, the problem is termed ill-posed .

The key link between PDE and our injective neural operators is as follows. Can G be approximated by injective neural operators N inj ? If so, then N inj can be a surrogate model to solve an ill-posed inverse problem. In general, this is possible, even if G is non-injective, per the results in Section 3.

Moreover, by the results of Section 4, we may conduct the inverse N -1 inj of surrogate model N inj with another neural operator.

Approximation of pushes forward measure: Let M be the submanifold of X = L 2 ([0 , 1] 2 ) or X = L 2 ([0 , 1] 3 ) corresponding to natural images or 3D medical model. Let K ⊂ R D be a manifold with the same topology as M , ι : R D → X be an embedding, and let K 1 = ι ( K ) ⊂ M . Given µ , a measure supported on M , the task is to find a neural operator f θ : X → X that maps (pushes forward) the uniform distribution on the model space K 1 to µ and so thus maps K 1 to M . If f θ : X → X is bijective, computing likelihood functions in statistical analysis is made easier via the change of variables formula. Further, we may interpret f -1 θ as an encoder and f θ as the corresponding decoder, which parameterized elements of M . As everything is formulated in infinite dimension function space X , we obtain discretization invariant methods.

B Proof of Proposition 1 in Section 2

Proof. We use the notation T | S ( v,T + b ) ( v ) = ( T i v ) i ∈ S ( v,T + b ) . Assume that T + b has a DSS with respect to every v ∈ L 2 ( D ) n in the sense of Definition 2, and that

where v (1) , v (2) ∈ L 2 ( D ) n . Since T + b has a DSS with respect to v (1) , we have for i ∈ S ( v (1) , T + b )

By assuming (B.1), we have for i / ∈ S ( v (1) , T ) ,

Then, we have

that is, which implies that

Thus,

In addition,

Thus, Combining (B.2) and (B.3), and (2.1) as v = v (1) , we conclude that v (1) -v (2) = 0 .

Conversely, assume that there exists a v ∈ L 2 ( D ) n such that

̸

For i ∈ S ( v, T + b ) , we have by u ∈ Ker( T i ) ,

For i / ∈ S ( v, T + b ) , we have by u ∈ X ( v, T + b ) ,

Therefore, we conclude that

Then there is u = 0 such that

̸

We define, for α ∈ (0 , 1) ,

with k ∈ N and j ∈ [ m -ℓ ] . We note that { φ α k,j } k ∈ N ,j ∈ [ m -ℓ ] is an orthonormal sequence in L 2 ( D ) m . We set

̸

where u = 0 , that is, ReLU ◦ ( T + b ) is not injective.

C Details in Sections 3.1 and 3.2

C.1 Proof of Lemma 1

Proof. The restriction operator, π ℓ : L 2 ( D ) m → L 2 ( D ) ℓ ( ℓ < m ), acts as follows,

Since L 2 ( D ) is a separable Hilbert space, there exists an orthonormal basis { φ k } k ∈ N in L 2 ( D ) . We denote by

for k ∈ N and j ∈ [ m -ℓ ] . Then, { φ 0 k,j } k ∈ N ,j ∈ [ m -ℓ ] is an orthonormal sequence in L 2 ( D ) m , and

It holds for 0 < α < 1 / 2 that is equivalent to

that is, Q α maps from Ran( P V ⊥ α ) to

It follows that is injective.

Indeed, for u ∈ L 2 ( D ) m and 0 < α < 1 / 2 ,

which implies that ∥ ∥ ∥ P V ⊥ α -P V ⊥ 0 ∥ ∥ ∥ op ≤ 2 α .

We will show that the operator

is injective. Assuming that for a, b ∈ L 2 ( D ) n ,

Denoting by P V α ( T ( a ) -T ( b )) = ∑ k ∈ N ,j ∈ [ m -ℓ ] c k,j φ α k,j ,

From (3.1), we obtain that c kj = 0 for all k, j . By injectivity of T , we finally get a = b . We define Q α : L 2 ( D ) m → L 2 ( D ) m by

By the same argument as in Section I.4.6 Kato [2013], we can show that Q α is injective and

C.2 Remarks following Lemma 1

Remark 3. Lemma 1 and Eqn. (3.1) may be interpreted as saying that if some orthonormal sequence { ξ k } k ∈ N exists that doesn't overlap the range of T , then T may be embedded in a small space without losing injectivity. An example that satisfies (3.1) is the neural operator whose L -th layer operator L L consists of the integral operator K L with continuous kernel function k L , and with continuous activation function σ . Indeed, in this case, we may choose the orthogonal sequence { ξ k } k ∈ N in L 2 ( D ) as a discontinuous functions sequence 1 so that span { ξ k } k ∈ N ∩ C ( D ) = { 0 } . Then, by Ran( L L ) ⊂ C ( D ) d L , the assumption (3.1) holds.

Remark 4. In the proof of Lemma 1, an operator B ∈ L ( L 2 ( D ) m , L 2 ( D ) ℓ ) ,

appears, where P V ⊥ α is the orthogonal projection onto orthogonal complement V ⊥ α of V α with

in which φ α k,j is defined for α ∈ (0 , 1) , k ∈ N and j ∈ [ ℓ ] ,

Here, { φ k } k ∈ N is an orthonormal basis in L 2 ( D ) . Futhermore, Q α : L 2 ( D ) m → L 2 ( D ) m is defined by

where P V ⊥ 0 is the orthogonal projection onto orthogonal complement V ⊥ 0 of V 0 with

The operator Q α is well-defined for 0 < α < 1 / 2 because it holds that

This construction is given by the combination of "Pairs of projections" discussed in Kato [2013, Section I.4.6] with the idea presented in [Puthawala et al., 2022b, Lemma 29].

C.3 Proof of Theorem 1

We begin with

Definition 3. The set of L -layer neural networks mapping from R d to R d ′ is

where σ : R → R is an element-wise nonlinear activation function. For the class of nonlinear activation functions,

and

where B R (0) := { u ∈ L 2 ( D ) d in | ∥ u ∥ L 2 ( D ) d in ≤ R } . By Theorem 11 of Kovachki et al. [2021b], there exists L ∈ N and ˜ G ∈ NO L ( σ ; D,d in , d out ) such that

We write operator ˜ G by

where

We remark that kernel functions ˜ k ℓ are continuous because neural operators defined in Kovachki et al. [2021b] parameterize the integral kernel function by neural networks, thus,

We define the neural operator H : L 2 ( D ) d in → L 2 ( D ) d in + d out by

where K ℓ and b ℓ are defined as follows. First, we choose K inj ∈ L ( L 2 ( D ) d in , L 2 ( D ) d in ) as a linear injective integral operator 2 .

(i) When σ 1 ∈ A L 0 ∩ BA is injective,

The set of integral neural operators with L 2 -integral kernels is

Proof. Let R > 0 such that

Then, the operator H : L 2 ( D ) d in → L 2 ( D ) d in + d out has the form of

For the case of (ii), we have used the fact

Thus, in both cases, H is injective.

In the case of (i), as σ ∈ A L 0 , we obtain the estimate where

.

Then we evaluate for a ∈ K ( ⊂ B R (0)) ,

In the case of (ii), we find the estimate, for a ∈ K ,

From (C.6) (especially, Ran( π 1 H ) ⊂ C ( D ) ) and Remark 3, we can choose an orthogonal sequence { ξ k } k ∈ N in L 2 ( D ) such that (3.1) holds. By applying Lemma 1, as T = H , n = d in , m = d in + d out , ℓ = d out , we find that

is injective. Here, P V ⊥ α and Q α are defined as in Remark 4; we choose 0 < α << 1 such that

where P V ⊥ 0 is the orthogonal projection onto

By the same argument as in the proof of Theorem 15 in Puthawala et al. [2022a], we can show that

Furthermore, since B is a linear operator, B ◦ K L +1 is also a linear operator with integral kernel ( Bk L +1 ( · , y )) ( x ) , where k L +1 ( x, y ) is the kernel of K L +1 . This implies that

We get, for a ∈ K ,

Using (C.7) and (C.8), we then obtain

Combining (C.9) and (C.10), we conclude that

C.4 Remark following Theorem 1

Remark 5. We make the following observations using Theorem 1:

(i) ReLU and Leaky ReLU functions belong to A L 0 ∩ BA due to the fact that { σ ∈ C ( R ) | σ is not a polynomial } ⊆ A 0 (see Pinkus [1999]), and both the ReLU and Leaky ReLU functions belong to BA (see Lemma C.2 in Lanthaler et al. [2022]). We note that Lemma C.2 in Lanthaler et al. [2022] solely established the case for ReLU. However, it holds true for Leaky ReLU as well since the proof relies on the fact that the function x ↦→ min(max( x, R ) , R ) can be exactly represented by a two-layer ReLU neural network, and a two-layer Leaky ReLU neural network can also represent this function. Consequently, Leaky ReLU is one of example that satisfies (ii) in Theorem 1.

(ii) We emphasize that our infinite-dimensional result, Theorem 1, deviates from the finitedimensional result. Puthawala et al. [2022a, Theorem 15] assumes that 2 d in +1 ≤ d out due to the use of Whitney's theorem. In contrast, Theorem 1 does not assume any conditions on d in and d out , that is, we are able to avoid invoking Whitney's theorem by employing Lemma 1.

(iii) We provide examples that injective universality does not hold when L 2 ( D ) d in and L 2 ( D ) d out are replaced by R d in and R d out : Consider the case where d in = d out = 1 and G + : R → R is defined as G + ( x ) = sin( x ) . We can not approximate G + : R → R by an injective function G : R → R in the set K = [0 , 2 π ] in the L ∞ -norm. According to the topological degree theory (see Cho and Chen [2006, Theorem 1.2.6(iii)]), any continuous function G : R → R which satisfies ∥ G -G + ∥ C ([0 , 2 π ]) < ε satisfies the equation on both intervals I 1 = [0 , π ] , I 2 = [ π, 2 π ] deg ( G,I j , s ) = deg ( G + , I j , s ) = 1 for all s ∈ [ -1 + ε, 1 -ε ] , j = 1 , 2 . This implies that G : I j → R obtains the value s ∈ [ -1 + ε, 1 -ε ] at least once. Hence, G obtains the values s ∈ [ -1 + ε, 1 -ε ] at least two times on the interval [0 , 2 π ] and is it thus not injective. It is worth noting that the degree theory exhibits significant differences between the infinite-dimensional and finite-dimensional cases [Cho and Chen, 2006]).

D Details in Section 3.3

D.1 Finite rank approximation

We consider linear integral operators K ℓ with L 2 kernels k ℓ ( x, y ) . Let { φ k } k ∈ N be an orthonormal basis in L 2 ( D ) . Since { φ k ( y ) φ p ( x ) } k,p ∈ N is an orthonormal basis of L 2 ( D × D ) , integral kernels k ℓ ∈ L 2 ( D × D ; R d ℓ +1 × d ℓ ) in integral operators K ℓ ∈ L ( L 2 ( D ) d ℓ , L 2 ( D ) d ℓ +1 ) has the expansion

then integral operators K ℓ ∈ L ( L 2 ( D ) d ℓ , L 2 ( D ) d ℓ +1 ) take the form

where C ( ℓ ) k,p ∈ R d ℓ +1 × d ℓ whose ( i, j ) -th component c ( ℓ ) k,p,ij is given by

Here, we write ( u, φ k ) ∈ R d ℓ ,

We define K ℓ,N ∈ L ( L 2 ( D ) d ℓ , L 2 ( D ) d ℓ +1 ) as the truncated expansion of K ℓ by N finite sum, that is,

Then K ℓ,N ∈ L ( L 2 ( D ) d ℓ , L 2 ( D ) d ℓ +1 ) is a finite rank operator with rank N . Furthermore, we have

which implies that as N →∞ ,

D.2 Layerwise injectivity

We first revisit layerwise injectivity and bijectivity in the case of the finite rank approximation. Let K N : L 2 ( D ) n → L 2 ( D ) m be a finite rank operator defined by

where C k,p ∈ R m × n and ( u, φ p ) ∈ R n is given by

Let b N ∈ L 2 ( D ) n be defined by

D.3 Global injectivity

We revisit global injectivity in the case of finite rank approximation. As an analogue of Lemma 1, we have the following

in which b p ∈ R m . As analogues of Propositions 1 and 2, we obtain the following characterization.

Proposition 5. (i) The operator

is injective if and only if for every v ∈ (span { φ k } k ≤ N ) n ,

where S ( v, K N + b N ) ⊂ [ m ] and X ( v, K N + b N ) are defined in Definition 2, and

(ii) Let σ be injective. Then the operator

is injective if and only if C N is injective, where

Proof. The above statements follow from Propositions 1 and 2 by observing that u ∈ Ker( K N ) is equivalent to (cf. (D.2) and (D.3))

Lemma 2. Let N,N ′ ∈ N and n, m, ℓ ∈ N with N ′ m>N ′ ℓ ≥ 2 Nn +1 , and let T : L 2 ( D ) n → L 2 ( D ) m be a finite rank operator with N ′ rank, that is,

is injective. Then, there exists a finite rank operator B ∈ L ( L 2 ( D ) m , L 2 ( D ) ℓ ) with rank N ′ such that

and Lipschitz continuous, and is injective.

Proof. From (D.4), T : L 2 ( D ) n → L 2 ( D ) m has the form of

where ( T ( a ) , φ k ) ∈ R m . We define T : R Nn → R N ′ m by

where T ( a ) ∈ L 2 ( D ) m is defined by

in which a k ∈ R n , a = ( a 1 , ..., a N ) ∈ R Nn .

Since T : L 2 ( D ) n → L 2 ( D ) m is Lipschitz continuous, T : R Nn → R N ′ m is also Lipschitz continuous. As N ′ m > N ′ ℓ ≥ 2 Nn +1 , we can apply Lemma 29 from Puthawala et al. [2022a] with D = N ′ m , m = N ′ ℓ , n = Nn . According to this lemma, there exists a N ′ ℓ -dimensional linear subspace V ⊥ in R N ′ m such that

and is injective, where V ⊥ 0 = { 0 } N ′ ( m -ℓ ) × R N ′ ℓ . Furthermore, in the proof of Theorem 15 of Puthawala et al. [2022a], denoting

we are able to show that

is injective. Here, π N ′ ℓ : R N ′ m → R N ′ ℓ

where Q : R N ′ m → R N ′ m is defined by

We define B : L 2 ( D ) m → L 2 ( D ) ℓ by

where B k,p ∈ R ℓ × m , B = ( B k,p ) k,p ∈ [ N ′ ] . Then B : L 2 ( D ) m → L 2 ( D ) ℓ is a linear finite rank operator with N ′ rank, and

is injective because, by the construction, it is equivalent to

is injective.

D.4 Proof of Theorem 2

Definition 4. We define the set of integral neural operators with N rank by NO L,N ( σ ; D,d in , d out ) := { G N : L 2 ( D ) d in → L 2 ( D ) d out : G N = K L +1 ,N ◦ ( K L,N + b L,N ) ◦ σ · · · ◦ ( K 2 ,N + b 2 ,N ) ◦ σ ◦ ( K 1 ,N + b 1 ,N ) ◦ ( K 0 ,N + b 0 ,N ) , K ℓ,N ∈ L ( L 2 ( D ) d ℓ , L 2 ( D ) d ℓ +1 ) , K ℓ,N : f ↦→ ∑ k,p ≤ N C ( ℓ ) k,p ( f, φ k ) φ p , b ℓ,N ∈ L 2 ( D ; R d ℓ +1 ) , b ℓ,N = ∑ p ≤ N b ( ℓ ) p φ m , C ( ℓ ) k,p ∈ R d ℓ +1 × d ℓ , b ( ℓ ) p ∈ R d ℓ +1 , k, p ≤ N, d ℓ ∈ N , d 0 = d in , d L +2 = d out , ℓ = 0 , ..., L +2 } . (D.5)

Proof. Let R > 0 such that

and

where B R (0) := { u ∈ L 2 ( D ) d in | ∥ u ∥ L 2 ( D ) d in ≤ R } . As ReLU and Leaky ReLU function belongs to A L 0 ∩ BA , by Theorem 11 of Kovachki et al. [2021b], there exists L ∈ N and ˜ G ∈ NO L ( σ ; D,d in , d out ) such that

We write operator ˜ G by

where

We set ˜ G N ′ ∈ NO L,N ′ ( σ ; D,d in , d out ) such that

where ˜ K ℓ,N ′ : L 2 ( D ) d ℓ → L 2 ( D ) d ℓ +1 is defined by

where C ( ℓ ) k,p ∈ R d ℓ +1 × d ℓ whose ( i, j ) -th component c ( ℓ ) k,p,ij is given by

Since

there is a large N ′ ∈ N such that

Then, we have

We define the operator H N ′ : L 2 ( D ) d in → L 2 ( D ) d in + d out by

where K inj,N : L 2 ( D ) d in → L 2 ( D ) d in is defined by

As K inj,N : (span { φ k } k ≤ N ) d in → L 2 ( D ) d in is injective,

is injective. Furthermore, by the same argument (ii) (construction of H ) in the proof of Theorem 1,

because both of two-layer ReLU and Leaky ReLU neural networks can represent the identity map. Note that above K inj,N is an orthogonal projection, so that K inj,N ◦· · ·◦ K inj,N = K inj,N . However, we write above H N ′ ( a ) 1 as K inj,N ◦ · · · ◦ K inj,N ( a ) so that it can be considered as combination of ( L +2) layers of neural networks.

We estimate that for a ∈ L 2 ( D ) d in , ∥ a ∥ L 2 ( D ) d in ≤ R ,

Here, we repeat an argument similar to the one in the proof of Lemma 2: H N ′ : L 2 ( D ) d in → L 2 ( D ) d in + d out has the form of

where ( H N ′ ( a ) 1 , φ k ) ∈ R d in , ( H N ′ ( a ) 2 , φ k ) ∈ R d out . We define H N ′ : R Nd in → R Nd in + N ′ d out by

where H N ′ ( a ) = ( H N ′ ( a ) 1 , H N ′ ( a ) 2 ) ∈ L 2 ( D ) d in + d out is defined by

where a k ∈ R d in , a = ( a 1 , ..., a N ) ∈ R Nd in . Since H N ′ : L 2 ( D ) d in → L 2 ( D ) d in + d out is Lipschitz continuous, H N ′ : R Nd in → R N ′ d out is also Lipschitz continuous. As

we can apply Lemma 29 of Puthawala et al. [2022a] with D = Nd in + N ′ d out , m = N ′ d out , n = Nd in . According to this lemma, there exists a N ′ d out -dimensional linear subspace V ⊥ in R Nd in + N ′ d out such that

and

is injective, where V ⊥ 0 = { 0 } Nd in × R N ′ d out . Furthermore, in the proof of Theorem 15 of Puthawala et al. [2022a], denoting by

we can show that is injective, where π N ′ d out : R Nd in + N ′ d out → R N ′ d out

π

N

′

d

out

(

a, b

) :=

b,

(

a, b

)

∈

and Q : R Nd in + N ′ d out → R Nd in + N ′ d out is defined by

By the same argument in proof of Theorem 15 in Puthawala et al. [2022a], we can show that

We define B : L 2 ( D ) d in + d out → L 2 ( D ) d out

B k,p ∈ R d out × ( d in + d out ) , B = ( B k,p ) k,p ∈ [ N ′ ] , then B : L 2 ( D ) d in + d out → L 2 ( D ) d out is a linear finite rank operator with N ′ rank. Then,

is injective because by the construction, it is equivalent to

is injective. Furthermore, we have

Indeed, H N ′ ∈ NO L,N ′ ( σ ; D,d in , d out ) , B is the linear finite rank operator with N ′ rank, and multiplication of two linear finite rank operators with N ′ rank is also a linear finite rank operator with N ′ rank.

Finally, we estimate for a ∈ K ,

Using notation

where ∥·∥ 2 is the Euclidean norm. Combining (D.8) and (D.9), we conclude that

R

Nd in

×

R

N

′

d

out

,

E Details in Section 4.1

E.1 Proof of Proposition 3

Proof. Since W is bijective, and σ is surjective, it is enough to show that u ↦→ Wu + K ( u ) is surjective. We observe that for z ∈ L 2 ( D ) n ,

is equivalent to

We will show that H z : L 2 ( D ) n → L 2 ( D ) n has a fixed point for each z ∈ L 2 ( D ) n . By the Leray-Schauder theorem, see Gilbarg and Trudinger [2001, Theorem 11.3], H : L 2 ( D ) → L 2 ( D ) has a fixed point if the union ⋃ 0 <λ ≤ 1 V λ is bounded, where the sets

are parametrized by 0 < λ ≤ 1 .

As the map u ↦→ αu + W -1 K ( u ) is coercive, there is an r > 0 such that for ∥ u ∥ L 2 ( D ) n > r ,

Thus, we have that for ∥ u ∥ L 2 ( D ) n > r

and, hence, for all ∥ u ∥ L 2 ( D ) > r 0 and λ ∈ (0 , 1] we have u ̸∈ V λ . Thus

Again, by the Leray-Schauder theorem (see Gilbarg and Trudinger [2001, Theorem 11.3]), H z has a fixed point.

E.2 Examples for Proposition 3

Example 2. We consider the case where n = 1 and D ⊂ R d is a bounded interval. We consider the non-linear integral operator,

and k ( x, y, t ) is bounded, that is, there is C K > 0 such that

If ∥ ∥ W -1 ∥ ∥ op is small enough such that

then, for α ∈ ( ∥ ∥ W -1 ∥ ∥ op C K | D | , 1 ) , u ↦→ αu + W -1 K ( u ) is coercive. Indeed, we have for u ∈ L 2 ( D ) ,

For example, we can consider a kernel

where σ s : R → R is the sigmoid function defined by

T are functions a, b, c ∈ C ( D × D ) such that

Example 3. Again, we consider the case where n = 1 and D ⊂ R d a is a bounded set. We assume that W ∈ C 1 ( D ) satisfies 0 < c 1 ≤ W ( x ) ≤ c 2 . For simplicity, we assume that | D | = 1 . We consider the non-linear integral operator

where

in which σ wire : R → R is the wavelet function defined by

and a j , b j , c j ∈ C ( D × D ) are such that the a j ( x, y ) are nowhere vanishing functions, that is, a j ( x, y ) = 0 for all x, y ∈ D × D . the and its generalizations (e.g. activation functions which do decay only exponentially).

̸

The next lemma holds for any activation function with exponential decay, including the activation function σ wire and settles the key condition for Proposition 3 to hold.

Lemma 3. Assume that | D | = 1 and the activation function σ : R → R be continuous. Assume that there exists M 1 , m 0 > 0 such that

Let a j , b j , c j ∈ C ( D × D ) be such that a j ( x, y ) are nowhere vanishing functions, that is, a j ( x, y ) = 0 for all x, y ∈ D × D . Moreover, let K : L 2 ( D ) → L 2 ( D ) be a non-linear integral operator given in (E.1) with a kernel satisfying (E.2) . Let α > 0 and 0 < c 0 ≤ W ( x ) ≤ c 1 . Then function F : L 2 ( D ) → L 2 ( D ) , F ( u ) = αu + W -1 K ( u ) is coercive.

̸

Proof. As D is compact, there is a 0 > 0 such that for all j = 1 , 2 , . . . , J we have | a j ( x, y ) | ≥ a 0 a.e. and | b j ( x, y ) | ≤ b 0 a.e. We point out that | σ ( t ) | ≤ M 1 . Next,

we consider λ > 0 and u ∈ L 2 ( D ) and the sets

Let ε > 0 be such that

Then, for x ∈ D 2 ( λ ) ,

After ε is chosen as in the above, we choose λ 0 ≥ max(1 , b 0 / ( a 0 ε )) to be sufficiently large so that for all | t | ≥ ελ 0 it holds

Here, we observe that, as λ 0 ≥ b 0 / ( a 0 ε ) , we have that for all | t | ≥ ελ 0 , a 0 | t | -b 0 > 0 . Then, when λ ≥ λ 0 , we have for x ∈ D 1 ( λ ) ,

When u ∈ L 2 ( D ) has the norm ∥ u ∥ L 2 ( D ) = λ ≥ λ 0 ≥ 1 , we have

Hence,

and the function u → αu + W -1 K ( u ) is coercive.

E.3 Proof of Proposition 4

Proof. (Injectivity) Assume that

where u 1 , u 2 ∈ L 2 ( D ) n . Since σ is injective and W : L 2 ( D ) n → L 2 ( D ) n is bounded linear bijective, we have

Since the mapping u ↦→ z -W -1 K ( u ) is contraction (because W -1 K is contraction), by the Banach fixed-point theorem, the mapping u ↦→ z -W -1 K ( u ) admit a unique fixed-point in L 2 ( D ) n , which implies that u 1 = u 2 .

(Surjectivity) Since σ is surjective, it is enough to show that u ↦→ Wu + K ( u ) + b is surjective. Let z ∈ L 2 ( D ) n . Since the mapping u ↦→ W -1 z -W -1 b -W -1 K ( u ) is contraction, by Banach fixed-point theorem, there is u ∗ ∈ L 2 ( D ) n such that

E.4 Examples for Proposition 4

Example 4. We consider the case of n = 1 , and D ⊂ [0 , ℓ ] d . We consider Volterra operators

where x = ( x 1 , . . . , x d ) and y = ( y 1 , . . . , y d ) . We recall that K is a Volterra operator if

̸

In particular, when D = ( a, b ) ⊂ R is an interval, the Volterra operators are of the form

and if x is considered as a time variable, the Volterra operators are causal in the sense that the value of K ( u )( x ) at the time x depends only on u ( y ) at the times y ≤ x .

Assume that k ( x, y, t, s ) ∈ C ( D × D × R × R ) is bounded and uniformly Lipschitz smooth in the t and s variables, that is, k ∈ C ( D × D ; C 0 , 1 ( R × R )) .

Next, we consider the non-linear operator F : L 2 ( D ) → L 2 ( D ) ,

Assume that u, w ∈ L 2 ( D ) are such that u + K ( u ) = w + K ( w ) , so that w -u = K ( u ) -K ( w ) . Next, we will show that then u = w . We denote and D ( z 1 ) = D ∩ ([0 , z 1 ] × [0 , ℓ ] d -1 ) ,. Then for x ∈ D ( z 1 ) the Volterra property of the kernel implies that

so that for all 0 < z 1 < ℓ ,

∥ u -w ∥ 2 L 2 ( D ( z 1 )) = ∫ z 1 0 (∫ ℓ 0 · · · ∫ ℓ 0 1 D ( x ) | u ( x ) -w ( x ) | 2 dx d dx d -1 . . . dx 2 ) dx 1 ≤ z 1 ℓ d -1 ( 2 ∥ k ∥ C ( D × D ; C 0 , 1 ( R × R )) ∥ u -w ∥ L 2 ( D ( z 1 )) ∥ u ∥ L 2 ( D ( z 1 )) + ∥ k ∥ L ∞ ( D × D × R × R ) ∥ u -w ∥ L 2 ( D ( z 1 )) √ | D ( z 1 ) | ) 2 ≤ z 1 ℓ d -1 ( ∥ k ∥ C ( D × D ; C 0 , 1 ( R × R )) ∥ u ∥ L 2 ( D ) + ∥ k ∥ L ∞ ( D × D × R × R ) √ | D | ) 2 ∥ u -w ∥ 2 L 2 ( D ( z 1 )) . Thus, when z 1 is so small that z 1 ℓ d -1 ( ∥ k ∥ C ( D × D ; C 0 , 1 ( R n )) ∥ u ∥ L 2 ( D ) + ∥ k ∥ L ∞ ( D × D × R × R ) √ | D | ) 2 < 1 ,

we find that ∥ u -w ∥ L 2 ( D ( z 1 )) = 0 , that is, u ( x ) -w ( x ) = 0 for x ∈ D ( z 1 ) . Using the same arguments as above, we see for all k ∈ N that that if u = w in D ( kz 1 ) then u = w in D (( k +1) z 1 ) . Using induction, we see that u = w in D . Hence, the operator u ↦→ F ( u ) is injective in L 2 ( D ) .

Example 5. We consider derivatives of Volterra operators in the domain D ⊂ [0 , ℓ ] d . Let K : L 2 ( D ) → L 2 ( D ) be a non-linear operator

where k ( x, y, t ) satisfies (E.4) , is bounded, and k ∈ C ( D × D ; C 0 , 1 ( R × R )) . Let F 1 : L 2 ( D ) → L 2 ( D ) be

Then the Fréchet derivative of K at u 0 ∈ L 2 ( D ) to the direction w ∈ L 2 ( D ) is

where

is a Volterra operator satisfying

̸

As seen in Example 4, the operator DF 1 | u 0 : L 2 ( D ) → L 2 ( D ) is injective.

F Details in Section 4.2

In this appendix, we prove Theorem 3.We recall that in the theorem, weconsider the case when n = 1 , and D ⊂ R is a bounded interval, and the operator F 1 is of the form

where W ∈ C 1 ( D ) satisfies 0 < c 1 ≤ W ( x ) ≤ c 2 and the function ( x, y, s ) ↦→ k ( x, y, s ) is in C 3 ( D × D × R ) and in D × D × R its three derivatives and the derivatives of W are uniformly bounded by c 0 , that is,

We recall that the identical embedding H 1 ( D ) → L ∞ ( D ) is bounded and compact by Sobolev's embedding theorem.

As we will consider kernels k ( x, y, u 0 ( y )) , we will consider the non-linear operator F 1 mainly as an operator in a Sobolev space H 1 ( D ) .

The Frechet derivative of F 1 at u 0 to direction w , denoted A u 0 w = DF 1 | u 0 ( w ) is given by

The condition (F.1) implies that

is a locally Lipsichitz smooth function and the operator

given in (F.2), is defined for all u 0 ∈ C ( D ) as a bounded linear operator.

When X is a Banach space, let B X (0 , R ) = { v ∈ X : ∥ v ∥ X < R } and B X (0 , R ) = { v ∈ X : ∥ v ∥ X ≤ R } be the open and closed balls in X , respectively.

We consider the Hölder spaces C n,α ( D ) and their image in (leaky) ReLU-type functions. Let a ≥ 0 and σ a ( s ) = ReLU( s ) -a ReLU( -s ) . We will consider the image of the closed ball of C 1 ,α ( D ) in the map σ a , that is σ a ( B C 1 ,α ( D ) (0 , R )) = { σ a ◦ g ∈ C ( D ) : ∥ g ∥ C 1 ,α ( D ) ≤ R } .

We will below assume that for all u 0 ∈ C ( D ) the integral operator

This condition is valid when K ( u ) is a Volterra operator, see Examples 4 and 5. As the integral operators A u 0 are Fredholm operators having index zero, this implies that the operators (F.4) are bijective.

The inverse operator A -1 u 0 : H 1 ( D ) → H 1 ( D ) can be written as

where ˜ k u 0 , ∂ x ˜ k u 0 ∈ C ( D × D ) and ˜ W ∈ C 1 ( D ) .

We will consider the inverse function of the map F 1 in a set Y ⊂ σ a ( B C 1 ,α ( D ) (0 , R )) that is a compact subset of the Sobolev space H 1 ( D ) . To this end, we will cover the set Y with small balls B H 1 ( D ) ( g j , ε 0 ) , j = 1 , 2 , . . . , J of H 1 ( D ) , centred at g j = F 1 ( v j ) , where v j ∈ H 1 ( D ) . We will show below that when g ∈ B H 1 ( D ) ( g j , 2 ε 0 ) , that is, g is 2 ε 1 -close to the function g j in H 1 ( D ) , the inverse map of F 1 can be written as a limit ( F -1 1 ( g ) , g ) = lim m →∞ H ◦ j m ( v j , g ) in H 1 ( D ) 2 , where

that is, near g j we can approximate F -1 1 as a composition H ◦ j m of 2 m layers of neural operators.

To glue the local inverse maps together, we use a partition of unity in the function space Y that is given by integral neural operators

where ⃗ i belongs in some finite index set I ⊂ Z ℓ 0 , some ϵ 1 > 0 , some y ℓ ∈ D ( ℓ = 1 , ..., ℓ 0 ), s ( ⃗ i, ℓ ) = i ℓ ϵ 1 .π 1 ( v, w ) = v that maps a pair ( v, w ) to the first function v Here, F z,s,h ( v, w ) are integral neural operators with distributional kernels

where k z,s,h ( x, y, v ( x ) , w ( y )) = v ( x ) 1 [ s -1 2 h,s + 1 2 h ) ( w ( y )) δ ( y -z ) , and 1 A is the indicator function of a set A and y ↦→ δ ( y -z ) is the Dirac delta distribution at the point z ∈ D . Using these, we can write the inverse of F 1 at g ∈ Y as

where j ( ⃗ i ) ∈ { 1 , 2 , . . . , J } are suitably chosen and the limit is taken in the norm topology of H 1 ( D ) . This result is summarized in the following theorem, that is a modified version of Theorem 3 for the inverse operator F -1 1 in (F.6) where we have refined the partition of unity Φ ⃗ i so that we use indexes ⃗ i ∈ I ⊂ Z ℓ 0 instead of j ∈ { 1 , . . . , J } .

This result is summarized in following theorem:

Theorem 4. Assume that F 1 satisfies the above assumptions (F.1) and (F.4) and that F 1 : H 1 ( D ) → H 1 ( D ) is a bijection. Let Y ⊂ σ a ( B C 1 ,α ( D ) (0 , R )) be a compact subset of the Sobolev space H 1 ( D ) , where α > 0 and a ≥ 0 . Then the inverse of F 1 : H 1 ( D ) → H 1 ( D ) in Y can written as a limit (F.6) that is, as a limit of integral neural operators.

Observe that Theorem 4 includes the case where a = 1 , in which case σ a = Id and Y ⊂ σ a ( B C 1 ,α ( D ) (0 , R )) = B C 1 ,α ( D ) (0 , R )) . We note that when σ a is a leaky ReLU-function with parameter a > 0 , Theorem 4 can be applied to compute the inverse of σ a ◦ F 1 that is given by F -1 1 ◦ σ -1 a , where σ -1 a = σ 1 /a . Note that the assumption that Y ⊂ σ a ( B C 1 ,α ( D ) (0 , R )) makes it possible to apply Theorem 4 in the case when one trains deep neural networks having layers σ a ◦ F 1 and the parameter a of the leaky ReLU-function is a free parameter which is also trained.

Proof. As the operator F 1 can be multiplied by function W ( x ) -1 , it is sufficient to consider the case when W ( x ) = 1 .

Below, we use the fact that as D ⊂ R , Sobolev's embedding theorem yields that the embedding H 1 ( D ) → C ( D ) is bounded and there is C S > 0 such that

For clarity, we denote below the norm of C ( D ) by ∥ u ∥ L ∞ ( D ) .

Next we considre the Frechet derivatives of F 1 . We recall that the 1st Frechet derivative of F 1 at u 0 is the operator A u 0 . The 2nd Frechet derivative of F 1 at u 0 to directions w 1 and w 2 is

where and

Thus,

(F.10)

When we freeze the function u in kernel k to be u 0 , we denote

Lemma 4. For u 0 , u 1 ∈ C ( D ) we have

and

Proof. Denote

We have

By Schur's test for continuity of integral operators,

and and

and and

so that

Also, when D x v = dv dx ,

and

Thus,

and and

Similarly,

As the embedding H 1 ( D ) → C ( D ) is bounded and has norm C S , Lemma 4 implies that for all R > 0 there is

is Lipschitz map B H 1 (0 , R ) →L ( H 1 ( D ) , H 1 ( D )) with a Lipschitz constant C L ( R ) , that is,

As u 0 ↦→ A u 0 = DF 1 | u 0 is continuous, the inverse A -1 u 0 : H 1 ( D ) → H 1 ( D ) exists for all u 0 ∈ C ( D ) , and the embedding H 1 ( D ) → C ( D ) is compact, we have that for all R > 0 there is C B ( R ) > 0 such that

Let R 1 , R 2 > 0 be such that Y ⊂ B H 1 (0 , R 1 ) and X = F -1 1 ( Y ) ⊂ B H 1 (0 , R 2 ) . Below, we denote C L = C L (2 R 2 ) and C B = C B ( R 2 ) .

Next we consider inverse of F 1 in Y . To this end, let us consider ε 0 > 0 , which we choose later to be small enough. As Y ⊂ B H 1 (0 , R ) is compact there are finite number of elements g j = F 1 ( v j ) ∈ Y , where v j ∈ X , j = 1 , 2 , . . . , J such that

such that the map, see (F.11).

Let us consider a fixed j and g j ∈ Y . When g satisfies

is equivalent to the fixed point equation

that is equivalent to the fixed point equation

We observe that for u 0 , u 1 ∈ X ,

and hence the Lipschitz constant of A -1 · : u ↦→ A -1 u , X →L ( H 1 ( D ) , H 1 ( D )) satisfies

the equation

for the function H j : H 1 ( D ) → H 1 ( D ) ,

Note that H j depends on g , and thus we later denote H j = H g j . We observe that

Let u, v ∈ B H 1 (0 , 2 R 2 ) . We have

where, see (F.10),

so that for u 1 , u 2 ∈ B H 1 (0 , 2 R 2 ) ,

and and

see (F.2), and hence, when ∥ u -v j ∥ ≤ r ≤ R 2 ,

where

We now choose

We consider

Then, we have

Then, we have that Lip B H 1 (0 , 2 R 2 ) → H 1 ( D ) ( H j ) ≤ a = C H r < 1 2 , and

and for all u ∈ B H 1 (0 , R 2 ) such that ∥ u -v j ∥ ≤ r , we have ∥ A -1 v j ( g -g j ) ∥ H 1 ( D ) ≤ (1 -a ) r . Then,

that is, H j maps B H 1 ( D ) ( v j , r ) to itself. By Banach fixed point theorem, H j : B H 1 ( D ) ( v j , r ) → B H 1 ( D ) ( v j , r ) has a fixed point.

Let us denote

By the above, when we choose ε 0 to have a value

the map F 1 has a right inverse map R j in B H 1 ( g j , 2 ε 0 ) , that is,

it holds that R j : B H 1 ( g j , 2 ε 0 ) → B H 1 ( D ) ( v j , r ) , and by Banach fixed point theorem it is given by the limit

in H 1 ( D ) , where

We can write for g ∈ B H 1 ( g j , 2 ε 0 ) ,

where the limit takes space in H 1 ( D ) 2 and

is the composition of m operators H j . This implies that R j can be written as a limit of finite iterations of neural operators H j (we will consider how the operator A -1 v j can be written as a neural operator below).

As Y ⊂ σ a ( B C 1 ,α ( D ) (0 , R )) , there are finite number of points y ℓ ∈ D , ℓ = 1 , 2 , . . . , ℓ 0 and ε 1 > 0 such that the sets

where i 1 , i 2 , . . . , i ℓ 0 ∈ Z , satisfy the condition

̸

(F.23)

To show (F.23), we will below use the mean value theorem for function g = σ a ◦ v ∈ Y , where v ∈ C 1 ,α ( D ) . First, let us consider the case when the parameter a of the leaky ReLU function σ a is strictly positive. Without loss of generality, we can assume that D = [0 , 1] and y ℓ = hℓ , where

h = 1 /ℓ 0 and ℓ = 0 , 1 , . . . , ℓ 0 . We consider g ∈ Y ∩ Z ( i 1 , i 2 , . . . , i ℓ 0 ) ⊂ σ a ( B C 1 ,α ( D ) (0 , R )) of the form g = σ a ◦ v . As a is non-zero, the inequality ( i ℓ -1 2 ) ε ≤ g ( y ℓ ) < ( i ℓ + 1 2 ) ε is equivalent to σ 1 /a (( i ℓ -1 2 ) ε ) ≤ v ( y ℓ ) < σ 1 /a (( i ℓ + 1 2 ) ε ) , and thus

where A = 1 2 max(1 , a, 1 /a ) , that is, for g = σ a ( v ) ∈ Z ( i 1 , i 2 , . . . , i ℓ 0 ) the values v ( y ℓ ) are known within small errors. By applying mean value theorem on the interval [( ℓ -1) h, ℓh ] for function v we see that there is x ′ ∈ [( ℓ -1) h, ℓh ] such that

Observe that these estimates are useful when ε 1 is much smaller that h . As g = σ a ◦ v ∈ Y ⊂ σ a ( B C 1 ,α ( D ) (0 , R )) , we have v ∈ B C 1 ,α ( D ) (0 , R ) , so that dv dx ∈ B C 0 ,α ( D ) (0 , R ) satisfies (F.25) implies that

Moreover, (F.24) and v ∈ B C 1 ,α ( D ) (0 , R ) imply

and thus by (F.24), where

for all x ∈ [( ℓ -1) h, ℓh ] .

Let ε 2 = ε 0 /A. When we first choose ℓ 0 to be large enough (so that h = 1 /ℓ 0 is small) and then ε 1 to be small enough, we may assume that

Then for any two functions g, g ′ ∈ Y ∩ Z ( i 1 , i 2 , . . . , i ℓ 0 ) ⊂ σ a ( B C 1 ,α ( D ) (0 , R )) of the form g = σ a ◦ v, g ′ = σ a ◦ v ′ the inequalities (F.27) and (F.28) imply

for all x ∈ D . As v, v ′ ∈ B C 1 ,α ( D ) (0 , R ) , this implies

As the embedding C 1 ( D ) → H 1 ( D ) is continuous and has norm less than 2 on the interval D = [0 , 1] , we see that

̸

and thus

To prove (F.23), we assume that ( Z ( i 1 , i 2 , . . . , i ℓ 0 ) ∩ Y ) ∩ B H 1 ( D ) ( g j , ε 0 ) = ∅ , and g ∈ Z ( i 1 , i 2 , . . . , i ℓ 0 ) ∩Y . By the assumption, there exists g ∗ ∈ ( Z ( i 1 , i 2 , . . . , i ℓ 0 ) ∩Y ) ∩ B H 1 ( D ) ( g j , ε 0 ) . Using (F.31), we have

Thus, g ∈ B H 1 ( D ) ( g j , 2 ε 0 ) , which implies that the property (F.23) follows.

We next consider the case when the parameter a of the leaky relu function σ a is zero. Again, we assume that D = [0 , 1] and y ℓ = hℓ , where h = 1 /ℓ 0 and ℓ = 0 , 1 , . . . , ℓ 0 . We consider g ∈ Y ∩ Z ( i 1 , i 2 , . . . , i ℓ 0 ) ⊂ σ a ( B C 1 ,α ( D ) (0 , R )) of the form g = σ a ( v ) and an interval [ ℓ 1 h, ( ℓ 1 +1) h ] ⊂ D , where 1 ≤ ℓ 1 ≤ ℓ 0 -2 . We will consider four cases. First, if g does not obtain the value zero on the interval [ ℓ 1 h, ( ℓ 1 +1) h ] the mean value theorem implies that there is x ′ ∈ [ ℓ 1 h, ( ℓ 1 +1) h ] such that dg dx ( x ′ ) = dv dx ( x ′ ) is equal to d = ( g ( ℓ 1 h ) -g ([( ℓ 1 -1) h )) /h . Second, if g does not obtain the value zero on either of the intervals [( ℓ 1 -1) h, ℓ 1 h ] or [( ℓ 1 +1) h, ( ℓ 1 +2) h ] , we can use the mean value theorem to estimate the derivatives of g and v at some point of these intervals similarly to the first case. Third, if g does not vanish identically on the interval [ ℓ 1 h, ( ℓ 1 +1) h ] but it obtains the value zero on the both intervals [( ℓ 1 -1) h, ℓ 1 h ] and [( ℓ 1 +1) h, ( ℓ 1 +2) h ] , the function v has two zeros on the interval [( ℓ 1 -1) h, ( ℓ 1 +2) h ] and the mean value theorem implies that there is x ′ ∈ [( ℓ 1 -1) h, ( ℓ 1 +2) h ] such that dv dx ( x ′ ) = 0 . Fourth, if none of the above cases are valid, g vanishes identically on the interval [ ℓ 1 h, ( ℓ 1 +1) h ] . In all these cases the fact that ∥ v ∥ C 1 ,α ( D ) ≤ R implies that the derivative of g can be estimated on the whole interval [ ℓ 1 h, ( ℓ 1 + 1) h ] within a small error. Using these observations we see for any ε 2 , ε 3 > 0 that if y ℓ ∈ D = [ d 1 , d 2 ] ⊂ R , ℓ = 1 , 2 , . . . , ℓ 0 are a sufficiently dense grid in D and ε 1 to be small enough, then the derivatives of any two functions g, g ′ ∈ Y ∩ Z ( i 1 , i 2 , . . . , i ℓ 0 ) ⊂ σ a ( B C 1 ,α ( D ) (0 , R )) of the form g = σ a ( v ) , g ′ = σ a ( v ′ ) satisfly ∥ g -g ′ ∥ H 1 ([ d 1 + ε 3 ,d 2 -ε 3 ]) < ε 2 . As the embedding C 1 ([ d 1 + ε 3 , d 2 -ε 3 ]) → H 1 ([ d 1 + ε 3 , d 2 -ε 3 ]) is continuous,

and ε 2 and ε 3 can be chosen to be arbitrarily small, we see that the property (F.23) follows. Thus the property (F.23) is shown in all cases.

By our assumptions Y ⊂ σ a ( B C 1 ,α ( D ) (0 , R )) and hence g ∈ Y implies that ∥ g ∥ C ( D ) ≤ AR. This implies that Y ∩ Z ( i 1 , i 2 , . . . , i ℓ 0 ) is empty if there is ℓ such that | i ℓ | > 2 AR/ε 1 +1 . Thus, there is a finite set I ⊂ Z ℓ 0 such that

where we use notation ⃗ i = ( i 1 , i 2 , . . . , i ℓ 0 ) ∈ Z ℓ 0 . On the other hand, we have chosen g j ∈ Y such that B H 1 ( D ) ( g j , ε 0 ) , j = 1 , . . . , J cover Y . This implies that for all ⃗ i ∈ I there is j = j ( ⃗ i ) ∈ { 1 , 2 , . . . , j } such that there exists g ∈ Z ( ⃗ i ) ∩ B H 1 ( D ) ( g j , ε 0 ) . By (F.23), this implies that

Thus, we see that Z ( ⃗ i ) , ⃗ i ∈ I is a disjoint covering of Y , and by (F.34), in each set Z ( ⃗ i ) ∩ Y , ⃗ i ∈ I the map g →R j ( g ) we have constructed a right inverse of the map F 1 .

Below, we denote s ( ⃗ i, ℓ ) = i ℓ ε 1 . Next we construct a partition of unity in Y using maps

where

Then,

Next, for all ⃗ i ∈ I we define the operator Φ ⃗ i : H 1 ( D ) ×Y → H 1 ( D ) ,

where ϕ ⃗ i,ℓ : H 1 ( D ) ×Y → H 1 ( D ) ×Y are the maps

and π 1 ( v, w ) = v maps a pair ( v, w ) to the first function v . It satisfies

Observe that here s ( ⃗ i, ℓ ) = i ℓ ε 1 is close to the value g j ( ⃗ i ) ( y ℓ ) . Now we can write for g ∈ Y

with suitably chosen j ( ⃗ i ) ∈ { 1 , 2 , . . . , J } .

Let us finally consider A -1 u 0 where u 0 ∈ C ( D ) . Let us denote

and J u 0 = K u 0 + ˜ K u 0 be the integral operator with kernel

so that when we write the linear bounded operator

We have as an integral operator

we have

where

This implies that the operator A -1 u 0 = ( I + J u 0 ) -1 is a neural operator, too. Observe that ˜ j u 0 ( x, y ) , ∂ x ˜ j u 0 ( x, y ) ∈ C ( D × D ) .

This proves Theorem 3.

Peer reviews:
comments: SUMMARY:
This paper extends several known results on ReLU networks from finite dimensional domains to the more challenging infinite dimensional domains. Namely:
(a) Conditions for injectiviity of infinite dimensional ReLU networks are provided
(b) Universality of infinite dimensional ReLU networks was proven
(c) Extension to network with non-linear kernels are provided

STRENGTHS:
Strengths: From a mathematical standpoint the paper proves some natural and  non-trivial theorems

WEAKNESSES:
I think this paper is not appropriate for this venue: it is not clear to me why the ML world should care that much about infinite dimensional ReLU operators, and the authors do not make an effort to explain this. There is certainly room for purely theoretical papers in Neurips and there are many such papers. However, I believe that, even if the proofs are above the paygrade of most Neurips attendants, it should be at least clear to said attendants why the questions are interesting. I do not feel this is the case here

QUESTIONS:
To convince me, mostly rebuttal should focus on importance of the problems discussed in the paper.

I note some minor comments and typos I saw while reading the paper:
Line 7 `the case the case'
Line 34 'on its face' not sure that is an expression in english
Line 43, 53 114 and throughout `the equivalent condition' should be `an equivalent condition'
Definition 1 should be more explicit IMHO: What are bias functions? What do the linear kernels do? Better to just right out the formula
Line 105 be *the* ReLU activation

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the question of when neural operators, which have infinite-dimensional inputs and outputs, are injective and bijective. This question is answered in quite some generality in different settings and under different assumptions.\n\nSTRENGTHS:\n--- A careful and at times, deep, analysis for the questions under consideration is provided.\n\nWEAKNESSES:\n1. Relevance and Scope: While appreciating the depth of the functional analysis that is presented, this reviewer is left a bit perplexed about the rationale behind all this heavy machinery. The authors do not really motivate why one needs bijective neural operators in the first place. There is some boilerplate on generative models in the discussion but this does not occupy any centerstage. If the fundamental question is itself not posed properly, the interest of the subsequent analysis becomes rather limited. The authors should clearly motivate why they consider these questions in the first place and explain it well to the reader. \n\n2. Clarity: The paper has excessively abstract notation and presentation. It needs to be reorganized to bring out the main contributions well. Here are some specific questions about the clarity: \n\n    \na) Why introduce Directed spanning sets in Definition 1 (which is almost impossible to follow as no intuition is provided into what it is) and Proposition 1, when immediately afterwards you have Proposition 2 which provides the answer for a bijective activation function. Who cares about ReLU when you already have a result on Leaky ReLU which is heavily used to begin with. \n\n  b) Similarly what is the use of the local layer wise analysis when you anyway consider the global end-to-end neural operator in Section 3 ? \n\n  c) In Lemma 1, what does condition 3.1 even mean in practice ? How can one check it ? \n\n  d) The authors have to concede that having a universal approximation theorem does not mean much, see Lanthaler et al and Kovachki et al 2021a where this issue of universal approximation is critiqued as the system size can grow (even double) exponentially. So having a universal approximation theorem does not imply any kind of efficient approximation but it is a necessary condition at best. \n\n  e) Does section 3.3 imply that FNO and WNO are injective and bijective neural operators ? If not, why not ? If yes, under what conditions ? \n\n  f) Can you find a non-trivial example for section 4.1, apart from Example 1 which appears to be trivial. Has anyone ever implemented example 1? If so, what is its empirical performance. \n\n  g) What is the rationale for section 4.2, why would I want to explicitly compute the inverse of the neural operator -- in what situations is that useful ? Can you give an example ?\n\nAll the questions clearly indicate that the paper is not well-written and needs a substantial rewrite. \n\n3. Concreteness: The main limitation in my view is that lack of exemplification of the results, except for example 1 -- the authors should give many more examples of practical utility, with conditions on the weight matrices, kernels as well as activation functions so that either available or de novo neural operators fall into their theoretical framework. This will enable the readers to better understand the significance of this work. \n\n4. Novelty: In many places in the paper, the authors refer to Puthawala et al 2022 -- how does the current paper differ from this reference ? A thorough elaboration of these differences is essential for judging the novelty of this paper. \n\n5. Finite-dimensional representations and Aliasing: The authors always assume that one has access to functions as both inputs and outputs. This is far from the case and in practice, one has to work with finite-dimensional representations of the functions, see for instance Fanaskov and Oseledets, Spectral neural operators, 2022 for a discussion on this issue. This use of finite-dimensional representation can lead to what are called aliasing errors. For instance, FNO has aliasing errors -- how do these errors affect your theoretical considerations -- in particular, does aliasing destroy the bijection property of the neural operator. Addressing this issue is crucial for any relevance for the theory, particularly in the construction of inverse neural operators\n\nQUESTIONS:\nQuite a few questions were already asked in the section on Weaknesses. The authors should address them.\n\nLIMITATIONS:\n--- No solid rationale is provided for the whole premise of the paper.\n\n--- A lot of abstract theory with few examples, if any. Practical utility of the \nconcept is totally unclear from the current version.\n\n--- Unclear if the framework survives contact with reality in the form of finite-dimensional\nrepresentation of functions (for instance sampling on a uniform grid) that can lead to\naliasing errors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present theoretical results in the field of operator learning, specifically dealing with operators are injective and surjective. They build on existing finite-dimensional work and consider the infinite-dimensional case of learning mappings between infinite-rank Sobolev spaces. The paper contributes several theoretical results in this context, including characterizing the conditions for layer-wise injectivity and bijectivity given certain activation functions, universal approximation results for injective linear neural operators, and sufficient conditions for surjective/bijective nonlinear integral operators. The paper lays the groundwork for analysis of learning injective and bijective operators in the infinite-dimensional setting.\n\nSTRENGTHS:\nThe paper has several significant theoretical results and answers many natural questions one would have about the theory of injectivity and bijectivity in neural operators. Examples and practical implementation results in finite-rank cases are also described in the paper and appendix, which is helpful and grounds the theoretical results in practice. The writing is very clear and the proofs are clean. Overall, it is a very thorough paper.\n\nWEAKNESSES:\nJust a note on writing, too many proofs and examples are black-boxed and put in the appendix. At least a description of various proof techniques in the main text would be helpful. The paper would also benefit from more motivation of injective/bijective models in downstream tasks/generative modeling, describing any particular successes or failures different activation functions have had, etc. More discussion on how this theory should guide practice would also improve the significance of the paper.\n\nQUESTIONS:\nLine 107: Is it possible to give a more intuitive description of Definition 2 and why it is called a directed spanning set?\n\nLine 234: Not sure why it is a natural conclusion from using the sup norm that the approximation does not smooth out non-smooth operators. Can you clarify this comment?\n\nLine 215: This is a very interesting remark, can you imagine other orthonormal bases and discuss whether they would be worth formulating as a neural operator architecture?\n\nLine 249: What does it mean for integral transform with kernel the attention mechanism in a transformer to have be injective, practically/in a particular problem?\n\nLIMITATIONS:\nThis theoretical work naturally pertains to tasks where injectivity and bijectivity are desirable, however neural operators are of course used in much broader contexts. As the authors discuss, their contribution is largely theoretical and an important next step for this work would be explicit constructions of injective neural operators/inverses that are good approximators.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical analysis of the injectivity and surjectivity of neural operators.\n\nSection 2 discusses the injectivity of a single layer of neural operators. For the ReLU activation, the injectivity of the layer is characterized by the directed spanning set. On the other hand, if the activation is bijective, the injectivity of the layer is equivalent to the injectivity of its constituent.\n\nSection 3 of the paper examines how injectivity can be preserved when composing layers. The paper shows that integral neural operators with L^2-integral kernels possess a universal approximation property for continuous operators. A truncated series expansion-based approximation, implementable for this family, is demonstrated to maintain the universal approximation property.\n\nSection 4 considers nonlinear integral operators and presents a sufficient condition for a layer in this class to be bijective. Additionally, this document proposes a method for constructing the inverse of a nonlinear integral neural operator layer. The inverse is expressed as a limit of integral neural operators.\n\nSTRENGTHS:\n[originality]\n\n- As far as I understand, this paper is the first to provide a rigorous framework for analyzing the injectivity and bijectivity of neural operators.\n\n[quality]\n\n- The paper is written with mathematical rigor, and there are no noticeable flaws in the logical development. However, I have not been able to check the proofs in detail.\n\n[clarity]\n\n- The sections have clear purposes, with concise explanations of the goals in each section.\n\n[significance]\n\n- This paper presents universality results and identifies conditions under which injectivity or bijectivity can be guaranteed. These theoretical foundations can improve the models and facilitate their application by increasing their reliability and enhancing understanding.\n\nWEAKNESSES:\n[originality]\n\n- None in particular\n\n[quality]\n\n- The paper contains a noticeable number of grammatical and typographical errors. It is recommended that an up-to-date automatic checker is used to revise the paper and correct these issues.\n- The model classes, including layers, can be assigned specific symbols to make them stand out in the paper.\n- Line 33 mentions \"infinite-rank Sobolev spaces,\" but I could not find the definition of the notion of rank of a Sobolev space in the paper or elsewhere on the internet.\n\n[clarity]\n\n- Throughout the introduction, it remains unclear what is meant by \"finite-rank case\" and \"infinite-rank case.\" This issue may be resolved if the definition of the rank of Sobolev spaces is provided or if it is made clear that it refers to the rank of the target operator.\n\n[significance]\n\n- The theoretical results presented in the paper do not appear to have any immediate tangible implications for the models or learning algorithms of neural operators.\n- The motivation of the paper needs to be elaborated as it is currently unclear why it is important to understand the conditions under which the neural operators have injectivity or bijectivity.\n\nQUESTIONS:\nPlease kindly correct me if any of my understandings in the above comments are incorrect.\n\n- Line 32: Should \"infinite-dimension setting\" be \"infinite-rank setting\"?\n- Line 33: What does \"infinite-rank Sobolev space\" mean? I have not been able to find a definition of the rank of Sobolev spaces either in this paper or elsewhere.\n\n[minor suggestions]\n\n- Line 27 “a operators” → “an operator”\n- Line 21 “Bayesian UQ” → “Bayesian uncertainty quantification” (if UQ stands for that)\n- Line 23 “existence and uniqueness” → “existence and uniqueness of the solutions”\n- Line 47 “and their implementation” → “and that their implementation”\n- Line 48 “universality approximation” → “universal approximation”\n\nLIMITATIONS:\nNone in particular.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study investigates the injectivity and bijectivity of neural operators (NOs) in the infinite-rank setting which is less investigated than their finite-rank counter, such as invertible flow networks. This study is based on the finite-rank analysis by Puthawala et al. (2022a). As a previous work, Alberti et al. (2022) have shown a global injectivity of an infinite-rank NN based on wavelet expansion. The NO in consideration is formulated as a composite of hidden layers of the form $(\\mathcal{L}v)(x) := \\sigma( T(v)(x) + b(x) )$. In Section 2, the iff conditions for linear NOs are stated. Particularly Prop 2 (DSS condition for ReLU) is shown by extending the finite-rank results by Puthawala et al. (2022a), and Prop 3 (for bijective activation) is shown based on Fredholm theory. In Section 3, $cc$-universalities of (Theorem 1) injective linear NOs, and (Theorem 2) injective finite-rank NOs are shown. It is remarkable that different from finite-rank results, Theorem 1 does not require any assumption between input and output dimensions. In Section 4, a nonlinear NO with each layer being of the form $\\sigma \\circ F_1(u) = \\sigma(Wu + K(u))$, which covers attention mechanisms, is considered, (Props 3 and 4) sufficient conditions of the surjectivity and bijectivity are stated using the Leray-Schauder fix point theorem, and (Theorem 3) the inverses of bijective nonlinear NOs are constructed.\n\nSTRENGTHS:\n- Modern deep learning tasks tend to be formulated in the *infinite-dimensional* setup, and the authors established a mathematical foundation for a *general class of NOs*.\n\nWEAKNESSES:\n(Minor comments)\n- I believe that the NOs in consideration cover a wide range of practical examples, but it would broaden their potential readers if the authors could explicitly showcase such examples (eg., not just ones mentioned in l.252 and Example 1, but also subnetworks, operator transformers, and integral autoencoders, etc...)\n- Definition 1 would be clearer if the description in ll.77-80 by sentence “and $T_\\ell$ : … are sums of …” is associated with an expression such as $T_\\ell = …$.\n- Some citations in ll.339-341 are duplicated.\n- Additional literature (not mandatory as these are in finite-dimensional settings\n  - https://proceedings.neurips.cc/paper/2020/hash/2290a7385ed77cc5592dc2153229f082-Abstract.html\n  - https://arxiv.org/abs/2204.07415\n\nQUESTIONS:\nplease refer to comments in the Weaknesses section.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-19"}