Title: Can neural operators always be continuously discretized?
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural Operators

Invertibility

Category Theory
Abstract: In this work we consider the problem of discretization of neural operators in a general setting. Using category theory, we give a no-go theorem that shows that diffeomorphisms between Hilbert spaces may not admit any continuous approximations by diffeomorphisms on finite spaces, even if the discretization is non-linear. This shows how infinite-dimensional Hilbert spaces and finite-dimensional vector spaces fundamentally differ. A key take-away is that to obtain discretization invariance, considerable effort is needed to ensure that finite-dimensional approximations of neural operator converge not only as sequences of functions, but that their representations converge in a suitable sense as well. With this perspective, we give several positive results. We first show that strongly monotone diffeomorphism operators always admit finite-dimensional strongly monotone diffeomorphisms. Next we show that bilipschitz neural operators may always be written via the repeated alternating composition of strongly monotone neural operators and invertible linear maps. We also show that such operators may be inverted locally via iteration provided that such inverse exists. Finally, we conclude by showing how our framework may be used `out of the box' to prove quantitative approximation results for discretization of neural operators.

Can neural operators always be continuously discretized?

Abstract

We consider the problem of discretization of neural operators between Hilbert spaces in a general framework including skip connections. We focus on bijective neural operators through the lens of diffeomorphisms in infinite dimensions. Framed using category theory, we give a no-go theorem that shows that diffeomorphisms between Hilbert spaces or Hilbert manifolds may not admit any continuous approximations by diffeomorphisms on finite-dimensional spaces, even if the approximations are nonlinear. The natural way out is the introduction of strongly monotone diffeomorphisms and layerwise strongly monotone neural operators which have continuous approximations by strongly monotone diffeomorphisms on finite-dimensional spaces. For these, one can guarantee discretization invariance, while ensuring that finite-dimensional approximations converge not only as sequences of functions, but that their representations converge in a suitable sense as well. Finally, we show that bilipschitz neural operators may always be written in the form of an alternating composition of strongly monotone neural operators, plus a simple isometry. Thus we realize a rigorous platform for discretization of a generalization of a neural operator. We also show that neural operators of this type may be approximated through the composition of finite-rank residual neural operators, where each block is strongly monotone, and may be inverted locally via iteration. We conclude by providing a quantitative approximation result for the discretization of general bilipschitz neural operators.

1 Introduction

Neural operators, first introduced in [30], have become more and more prominent in deep learning on function spaces. As opposed to traditional neural networks that learn maps between finite-dimensional Euclidean spaces, neural operators learn maps between infinite-dimensional function spaces yet may be trained and evaluated on finite-dimensional data through a rigorous notion of discretization. Neural operators are widely used in the field of scientific machine learning [6, 21, 38, 55, 56], among others, principally because of their discretization invariance. In this work, we consider the fundamental limits of this discretization. Throughout, we emphasize the importance of continuity under discretization.

A key ingredient in our analysis is the identification of properties of diffeomorphisms that may be induced by (bijective) neural operators, which are diffeomorphisms themselves. Diffeomorphisms exist in many contexts, for example, in generative models. These involve mapping one probability distribution or measure, µ , over some measurable space to another, X , via a push forward, that is, F # µ ( U ) = µ ( F -1 ( U )) for U ⊂ X . If µ admits a density dµ then, in finite dimensions, we may

use the change of variables formula to write dρ ( x ) = dµ ( F -1 ( x )) | JF ( x ) | -1 , where JF is the Jacobian of F . Clearly, F must be a bijection with full-rank Jacobian. In other words, F must be a diffeomorphism onto its range. This established diffeomorphisms as natural objects of interest in finite-dimensional machine learning, and helps account for their wide use [20, 25, 31, 44, 54]. In this work, we consider the extension of these efforts from finite to infinite dimensions implemented via neural operators. Although there is no analogue of the change of variables formula in infinite dimensions, we argue that it is, nonetheless, natural to consider the role of diffeomorphisms, and how they may be approximated via diffeomorphisms on finite-dimensional spaces.

The question of when operations between Hilbert spaces may be discretized continuously may be understood through an analogy to computer vision. Consider the task of learning a map from one image space to another, for example, a style transfer problem [16], where the mapping learned does not depend much on the resolution of the images provided. It is natural to think of the map as being defined between (infinite-resolution) continuum images, and then its application to images of a specific resolution. In this analogy, X is a function space (over images m : [0 , 1] 2 → R [29]) that is approximated with a finite-dimensional space R d and the transformation F : X → X is approximated by a map f : R d → R d , where each f acts on images of a particular resolution. An explicit transformation formula can be obtained when f is a diffeomorphism and has a smooth inverse.

We introduce a framework based on a generalized notion of neural operator layers including a skip connection and their restrictions to balls rather than compact sets. With bijective neural operators in mind, we give a perspective based on diffeomorphisms in infinite dimensions between Hilbert manifolds. We give a no-go theorem, framed with category theory, that shows that diffeomorphisms between Hilbert spaces may not admit any continuous approximations by diffeomorphisms on finitedimensional spaces, even if the underlying discretization is nonlinear. In this framing the discretization operation is modeled as a functor from the category of Hilbert spaces and C 1 -diffeomorphisms on them to their finite-dimensional approximations. A natural way to mitigate the no-go theorem is described by the introduction of strongly monotone diffeomorphisms and layerwise strongly monotone neural operators. We prove that all strongly monotone neural operator layers admit continuous approximations by strongly monotone diffeomorphsisms on finite-dimensional spaces. Wethen provide various conditions under which a neural operator layer is strongly monotone. Notably, a bilipschitz (and, hence, bijective) neural operator layer can always be represented by a composition of strongly monotone neural operator layers. Hence, such an operator may be continuously discretized. More constructively, any bilipschitz neural operator layer can be approximated by residual finite-rank neural operators, each of which are strongly monotone, plus a simple isometry. Moreover, these finite-rank residual neural operators are (locally) bijective and invertible, and their inverses are limits of compositions of finite-rank neural operators. Our framework may be used 'out of the box' to prove quantitative approximation results for discretization of neural operators.

1.1 Related work

Neural operators were first introduced in [30]. Alternative designs for mappings between function spaces are the DeepONet [34, 40], and the PCA-Net [7, 12]. In spite of the multitudinous applications of neural operators, the theory of the natural class of injective or bijective neural operators is comparatively underdeveloped; see, for example [2, 14].

Our work is concerned with the of discretization of neural operators through the lens of diffeomorphisms. For recent important work on analyzing the effect of discretization error of Fourier Neural Operators (FNOs) arising from aliasing, see [35]. Our work has connections to infinite-dimensional inference, see e.g. [19], and approximation theory, see e.g. [13] while bridging the gap with the theory of neural operators.

We give a no-go theorem that uses a category theory framing. This contributes to the use of category theory as an emerging tool in the analysis and understanding of neural networks at large. In this sense, we are in league with the recent work generalizing ideas from geometric machine learning using category theory [17].

Discretization obstructions have been encountered in other contexts. Numerical methods that approximate continuous models are known to sometimes fail in surprising ways. A basic example of this is the 'locking' phenomenon in the study of the Finite Elements Method (FEM). For example,

linear elements used to model bending of a curved surface or beam lock in such a way that the model exhibits a non-physical stiff response to deformations [4]. Understanding this has been instrumental in developing improved numerical methods, such a high order FEM [52]. Furthermore, in discretized statistical inverse problems [27, 36, 51], the introduction of Besov priors [48, 11] has been found to be essential.

Finally, our work extends prior work (not based on deep learning) in discretization of physical or partial differential equations based forward models in inverse problems. The analogous notion of discretization invariance of solution algorithms of inverse problems was studied in [37, 48, 51] and the lack of it (in imaging methods using Bayesian inversion with L 1 priors) in [36, 48]. By considering the neural operator as the physical model, our results state that discretization can be done locally in an appropriate way, together with constructing an inverse.

1.2 Our contributions

The key results in this paper comprise the following:

We prove a general no-go theorem showing that, under general circumstances, diffeomorphisms between Hilbert spaces may not admit continuous approximation by finitedimensional diffeomorphisms. In particular, neural operators corresponding to diffeomorphic maps, in general, cannot be approximated by finite-dimensional diffeomorphisms and their associated neural representations.

We show that strongly monotone neural operator layers admit continuous approximations by strongly monotone diffeomorphsisms on finite-dimensional spaces.

We show that bilipschitz neural operators can be represented in any bounded set as a composition of strongly monotone, diffeomorphic neural operator layers, plus a simple isometry. These can be approximated by finite-rank diffeomorphic neural operators, where each layer is strongly monotone. For these operators we give a quantitative approximation result.

2 Definitions and notation

In this section, we give the definitions and notation used throughout the paper. First, we summarize the relevant basic concepts from functional analysis. Then, we introduce generalized neural operators.

2.1 Elements of functional analysis

In this work, all Hilbert spaces, X , are endowed with their norm topology. We denote by B X ( r ) = B X (0 , r ) the ball in the space X having the center at zero and radius r > 0 . We denote by S ( X ) the set of all finite-dimensional linear subspaces V ⊂ X . The set S 0 ( V ) ⊂ S ( X ) is a partially ordered lattice. That is, if V 1 , V 2 ∈ S 0 ( X ) then there is a V 3 ∈ S 0 ( X ) so that V 1 ⊂ V 3 and V 2 ⊂ V 3 . 1 With Y standing for another Hilbert space, we denote by C n ( X ; Y ) the set of operators, F : X → Y , having n continuous (Fréchet) derivatives, and C n ( X ) = C n ( X ; X ) .

Next, we define what it means that a nonlinear operator or function F : X → X on an infinitedimensional Hilbert space, X , is approximated by operators or functions on finite-dimensional subspaces V ⊂ X . The key is that as V tends to X , the complexity of F V increases and one may hope that the approximation becomes better. We formalize this in the following definition.

Definition 1 ( ϵ V approximators and weak approximators) . (i) Let r > 0 , F ⊂ C n ( X ; X ) be a family of functions, and ⃗ ε = ( ε V ) V ∈ S 0 ( X ) be a sequence such that ε V → 0 as V → X . We say that a function

is an ⃗ ε -approximation operation for functions F in the ball B X (0 , r ) taking values in families F V ⊂ C 1 ( V ; V ) if A X maps a function F : X → X , where F ∈ F , to a sequence of functions ( F V ) V ∈ S 0 ( X ) , where F V ∈ F V , such that the following is valid: For all F : X → X satisfying ∥ F ∥ C n ( B X (0 ,r ); X ) ≤ M , we have

where P V : X → X is the orthogonal projection onto V , that is, Ran( P V ) = V .

(ii) We say that A : C n ( X ; X ) →× V ∈ S 0 ( X ) C ( V ; V ) , F → ( F V ) V ∈ S 0 ( X ) is a weak approximation operation for the family F ⊂ C n ( X ; X ) if for any F ∈ F and r > 0 it holds that

Note that the condition (i) is stronger than the condition (ii). An example of an approximation operation for the family F = C n ( X ) , that is, an ⃗ ε -approximation operation with all sequences ⃗ ε = ( ε V ) V ∈ S 0 ( X ) subject to ε V > 0 , is the linear discretization

Nonlinear discretization methods that do not rely on P V have been used, for example, in the numerical analysis of nonlinear partial differential equations. Here, X becomes an appropriate Sobolev space, and a Galerkin approximation is implemented through finite-dimensional subspaces, V , spanned by finite element basis functions. We present an example for the nonlinear equation, ∆ u ( t ) -g ( u ( t )) = ∆ x ( t ) where g is a smooth convex function, when F : x → u , in Appendix A.1.

Below, we will study whether a family, F ⊂ Diff 1 ( X ) , of C 1 diffeomorphisms on X can be approximated by C 1 diffeomorphisms, F V ⊂ Diff 1 ( V ) , on finite-dimensional subspaces, V . Of course, diffeomorphisms are bijective. Unless stated otherwise, from now on we will omit C 1 and implicitly assume that diffeomorphism are C 1 diffeomorphisms. We introduce two more notions that will play a key role in the further analysis.

Definition 2 (Strongly Monotone) . We say that a (nonlinear) operator F : X → X on Hilbert space, X , is strongly monotone if there exists a constant α > 0 so that

Definition 3 (Bilipschitz) . We say that F if bilipschitz there exist constants c > 0 and C < ∞ so that for all x 1 , x 2 ∈ X , c ∥ x 1 -x 2 ∥ ≤ ∥ F ( x 1 ) -F ( x 2 ) ∥ ≤ C ∥ x 1 -x 2 ∥ .

2.2 A general framework for neural operators

In this paper, we are concerned with the modeling of diffeomorphisms between Hilbert spaces by bijective neural operators. Our working definition of neural operator, which generalizes the traditional notion, is given below. We note the presence of a skip connection, which is essential.

Definition 4 (Generalized neural operator layer) . For Hilbert space X , a layer of a neural operator is a function F : X → X of the form

where T 1 : X → X and T 2 : X → X are compact linear operators 2 and G : X → X is a nonlinear operator in C 1 ( X ) . A generalized neural operator, H : X → X , is given by the composition

where each F ℓ , ℓ = 1 , . . . , L is of the form (4), the A ℓ : X → X are bounded linear operators and σ : X → X is a continuous operation (for example, a Nemytskii operator defined by a composition with a suitable activation function in function spaces).

The generalized neural operators can represent the classical neural operators [30, 33]. For an explicit construction, we refer to Appendix C.1. In the next section, we will study, under what conditions, bounded linear operators A ℓ , Nemytskii operators σ , and neural operator layers F ℓ , for which the generalized neural operator consists, can be continuously discretized.

Wenote that because G ∈ C 1 ( X ) in Definition 4, it follows that G ∈ L ∞ ( X ) and Lip X → X ( G ) < ∞ . Given a Hilbert space, X , a layer of a strongly monotone neural operator (respectively, a layer of a bilipschitz neural operator ,) is a function F : X → X that is strongly monotone (respectively, bilipschitz). Furthermore, a strongly monotone neural operator (respectively, a bilipschitz neural operator ), is a generalized neural operator with strongly monotone (respectively, bilipschitz), layers.

3 Category theoretic framework for discretization

'Well-behaved' operators between infinite-dimensional Hilbert spaces may have dramatically different behaviors than corresponding 'well-behaved' maps between finite-dimensional Euclidean spaces. This observation applies to discretization and neural operators versus neural networks. In this section we explore this. We first present a no-go theorem, that there are no procedures that continuously discretize an isotopy of diffeomorphisms. Next, we introduce strongly monotone neural operator layers, which are strongly monotone diffeomorphisms, and then prove that these allow a continuous approximation functor, that is, continuous approximations by strongly monotone diffeomorphisms on finite dimensional spaces. We finally show that bilipschitz neural operator layers admit a representation via strongly monotone operators and linear maps, allowing for their continuous approximation.

3.1 No-go theorem for discretization of diffeomorphisms on Hilbert spaces

In this section, we present our no-go theorem. To formulate the 'impossibility' of something, we must define what is meant by discretization and approximation. Before this, we give an informal statement of the no-go theorem.

Theorem 1 (No-go Theorem, Informal) . Let A be an approximation scheme that maps diffeomorphisms F on a Hilbert to a sequence of finite-approximations F V that are themselves diffeomorphisms. If F V converges to F as V → X , then A is not continuous, that is, there are maps F ( j ) that converge to F as j →∞ , but all F ( j ) V are far from F V .

We want to emphasize that most practical numerical algorithms are continuous so that the output depends (in some suitable sense) continuously on the input. This shows that there are no such numerical schemes that approximate infinite-dimensional diffeomorphisms with finite-dimensional ones. In order to prove our no-go theorem in the most general setting, we phrase it in terms of category theory. Namely, we formulate A (which will denote the approximation scheme) as a functor from the category of Hilbert spaces and diffeomorphisms thereon, to their finite-rank approximations.

Definition 5 (Category of Hilbert Space Diffeomorphisms) . We denote by D the category of Hilbert diffeomorphisms with objects O D that are pairs ( X,F ) of a Hilbert space X and a (possibly nonlinear) C 1 -diffeomorphism F : X → X and the set of morphisms (or arrows that 'map' objects to other objects) A that are either

(induced isomorphisms) Maps a ϕ that are defined for a linear isomorphism ϕ : X 1 → X 2 of Hilbert spaces X 1 and X 2 that maps the objects ( X 1 , F 1 ) ∈ O D to the object ( ϕ ( X 1 ) , ϕ ◦ F 1 ◦ ϕ -1 ) ∈ O D , or

(induced restrictions) Maps a X 1 ,X 2 that are defined for a Hilbert space X 1 , its closed subspace X 2 ⊂ X 1 , and an object ( X 1 , F 1 ) ∈ O D such that F 1 ( X 2 ) = X 2 . Then a X 1 ,X 2 maps to the object ( X 1 , F 1 ) ∈ O D to the object ( X 2 , F 1 | X 2 ) ∈ O D .

Definition 6 (Category of Approximation Sequences) . We denote by B the category of approximation sequences , that has objects O B that are of the form ( X,S 0 ( X ) , ( F V ) V ∈ S 0 ( X ) ) where X is a Hilbert space,

are partially ordered lattices, ⋃ V ∈ S 0 ( X ) V = X , and F V : V → V are C 1 -diffeomorphisms of spaces V ∈ S 0 ( X ) .

The set of morphisms A B consists of either

Maps A ϕ that are defined for a linear isomorphism ϕ : X 1 → X 2 of Hilbert spaces X 1 and X 2 , and lattices S 0 ( X 1 ) and S 0 ( X 2 ) = { ϕ ( V ) | V ∈ S 0 ( X 1 ) } , that maps the objects ( X 1 , S ( X 1 ) , ( F V ) V ∈ S ( X 1 ) ) to ( X 2 , S ( X 2 ) , ( ϕ ◦ F ϕ -1 ( W ) ◦ ϕ -1 ) W ∈ S ( X 2 ) ) , or

Maps A X 1 ,X 2 that are defined for a Hilbert space X 1 , its closed subspace X 2 ⊂ X 1 , and an object ( X 1 , S 0 ( X 1 ) , ( F V ) V ∈ S 0 ( X 1 ) ) such that F ( X 2 ) = X 2 and S 0 ( X 2 ) = { V ∈ S 0 ( X 1 ) | V ⊂ X 2 } is a partially ordered lattice. Then A X 1 ,X 2 maps the object ( X 1 , S 0 ( X 1 ) , ( F V ) V ∈ S 0 ( X 1 ) ) to the object ( X 2 , S 0 ( X 2 ) , ( F V ) V ∈ S 0 ( X 2 ) ) .

Next, we define the notion of an approximation or discretization functor. In practice, an approximation functor is an operator which maps a function F in an infinite dimensional space X to a function F V that operate in finite dimensional subspaces V of X in such a way that functions F V are close (in a suitable sense) to the function F .

Definition 7 (Approximation Functor) . We define the approximation functor , denoted by A : D → B , as the functor that maps each ( X,F ) ∈ O D to some ( X,S 0 ( X ) , ( F V ) V ∈ S 0 ( X ) ) ∈ O B so that the Hilbert space X stays the same. The approximation functor maps all morphisms a ϕ to A ϕ and morphisms a X 1 ,X 2 to A X 1 ,X 2 , and has the following the properties

(A) For all r > 0 and all ( X,F ) ∈ O

In separable Hilbert spaces this means that when the finite dimensional subspaces V ⊂ X grow to fill the whole Hilbert space X , then the approximations F V converge uniformly in all bounded subsets to F .

We recall the notation lim V → X used above: We consider ( S 0 ( X ) , ⊃ ) as a partially ordered set and say that real numbers y V converge to the limit y as V → X , and denote

if for all ϵ > 0 there is V 0 ∈ S 0 ( X ) such that for V ∈ S 0 ( X ) satisfying V ⊃ V 0 it holds that | y V -y | < ϵ .

Definition 8. We say that the approximation functor A is continuous if the following holds: Let ( X,F ) , ( X,F ( j ) ) ∈ O D be such that the Hilbert space X is the same for all these objects and let ( X,S 0 ( X ) , ( F V ) V ∈ S 0 ( X ) ) = A ( X,F ) be approximating sequences of ( X,F ) and ( X,S 0 ( X ) , ( F j,V ) V ∈ S 0 ( X ) ) = A ( X,F ( j ) ) be approximating sequences of ( X,F ( j ) ) . Moreover, assume that r > 0 and

Then, for all V ∈ S 0 ( X ) the approximations F ( j ) V of F ( j ) and F V of F satisfy

The theorem below states a negative result, namely that there does not exist continuous approximating functors for diffeomorphisms.

Theorem 2. (No-go theorem for discretization of general diffeomorphisms) There exists no functor D → B that satisfies the property (A) of an approximation functor and is continuous.

The proof is given in Appendix A.4.1, and is quite involved, but we give an overview of some of the steps here. A generalization of Theorem 2, in the case where the norm topology is replaced by the weak topology, is considered in Appendix D.1.

A key fact is that for finite dimensional diffeomorphisms the space of smooth embeddings consists of two connected components, one orientation preserving and the other orientation reversing. This is

Connecting point at 00

dim(V)= dim(H) = 0

1 <dim(V)< ∞

(A)+HP

(A)-H!P

non-diffeomorphisms

dim(V) = 1

not the case in infinite dimensions, see e.g. [32] and [45]. For an illustration of this, see Figure 1. The proof proceeds by contradiction. First, we consider the action of the approximation functor as it operates on an isotopy (path of diffeomorphisms) that connects two diffeomorphisms. The first has only orientation-preserving discretizations, and the second only orientation-reversing discretizations. We then show that the image of the path under the approximation functor yields a disconnected path, as the discretization 'jumps' from the orientation preserving component to the orientation reversing component. This violates continuity. To encode the notions of orientation preserving and orientation reversing that allow for a description of nonlinear discretization theory, we use topological degree theory. This generalizes the familiar notion of orientation that uses the sign of the determinant of the Jacobian matrix.

3.2 Strongly monotone diffeomorphisms and their approximation on finite-dimensional subspaces

In this section and the next two we show that, although Theorem 2 precludes continuous approximation of general diffeomorphisms, stronger constraints on the diffeomorphisms allows one to sidestep the topological obstruction. In this section, in summary, we show that the obstruction to continuous approximation vanishes when the diffeomorphisms in question are assumed to be strongly monotone. Key to our positive result is the following technical result that states that the restriction of the domain and codomain of a strongly monotone diffeomorphism always yields another strongly monotone diffeomorphism.

Lemma 1. Let V ⊂ X be a finite-dimensional subspace of X , and let P V : X → X be orthogonal projection onto V . Let F : X → X be a strongly monotone C 1 -diffeomorphism. Then, P V F | V : V → V is strongly monotone, and a C 1 -diffeomorphism.

The proof is given in Appendix A.5.1. Lemma 1 implies that the discretization functor A lin defined in (2) is a well-defined functor from strongly monotone C 1 -diffeomorphisms of X to C 1 -diffeomorphisms of V . Note that the discretization functor A lin on strongly monotone C 1 diffeomorphisms may not be a continuous approximation functor in the strong sense of Definitions 7 and 8, but it is obviously a continuous approximation functor in the weak sense of Definitions 11 and 12. Therefore, we obtain that :

Proposition 1. Let A lin be the discretization functor that maps F to P V F | V for each finite subspace V ⊂ X . Let D sm and B sm be categories where F : X → X and F V : V → V are strongly monotone C 1 -diffeomorphisms. Then, the functor A lin : D sm → B sm satisfies assumption (A') of a weak approximation functor in Definition 11, and is continuous in the weak sense of Definition 12.

3.3 Strongly monotone Nemytskii operators and linear bounded operators and their continuous approximation on finite-dimensional subspaces

By Proposition 1, strongly monotone maps can be continuously discretized in the weak sense. Thus, we concern under what conditions, the maps, of which generalized neural operator consists, can be

strongly monotone. In this subsection, we focus on bounded linear operators and Nemytskii operators (layers of neural operator will be discussed in the next subsection). The following lemma is obviously given by the definition of a strongly monotone map :

Lemma 2. Let A : X → X be a linear bounded operator and satisfy ⟨ Au,u ⟩ ≥ c 0 ∥ u ∥ 2 X for some c 0 > 0 . Then, A : X → X is strongly monotone.

Next, assuming that X = L 2 ( D ; R ) , we define Nemytskii operator by

where σ : R → R is continuous. In this case, we can show the following lemma by using [50, Corollary 3.3]:

Proposition 2. Assume that σ satisfies that | σ ( s ) | ≤ C 1 | s | + C 2 and the derivative of s → σ ( s ) is defined a.e and satisfies σ ′ ( s ) ≥ α > 0 . Then, F σ : L 2 ( D ; R ) → L 2 ( D ; R ) is strongly monotonous.

3.4 Strongly monotone generalized neural operators and their continuous approximation on finite-dimensional subspaces

As seen in Theorem 3 below, strongly monotone layers of neural operators do not suffer from the same topological obstruction to continuous discretization as general diffeomorphisms. We now give sufficient conditions for the layers of a neural operator to be strongly monotone, and show that these conditions imply that those are diffeomorphisms.

Lemma 3. All strongly monotone layers of neural operators ( F ) defined by (4) are diffeomorphisms.

Also, the following theorem is proven in Appendix A.6.3.

Theorem 3. Let A lin be the discretization functor that maps F to P V F | V for each finite subspace V ⊂ X . Let D smn and B smn be categories where F : X → X and F V : V → V are strongly monotone C 1 -functions of the form (4) . Then, the functor A lin : D smn →B smn satisfies assumption (A), and it is continuous in the sense of Definition 8.

The functor defined in Theorem 3 does not suffer from the same topological obstruction as functors for general diffeomorphisms, shown in the no-go Theorem 2.This is because when F V = P V F | V is strongly monotone, its derivative D | x F V : V → V is a strongly monotone matrix at all points x ∈ V . Therefore it is strictly positive definite (see [47, Prop. 12.3]) and the determinant det ( D | x F V ) is strictly positive. Due to this, the orientation of the finite-dimensional approximations never switch signs, and the key technique used in the proof of the no-go Theorem 2 does not apply.

A straightforward condition to guarantee strong monotonicity of a neural operator layer is given in

Lemma 4. Let F : X → X be a layer of neural operator that is of the form F ( u ) = u + T 2 G ( T 1 u ) , where T j : X → X , j = 1 , 2 are compact operators and G : X → X is a C 1 -smooth map. Assume that Fréchet derivative DG | x of G at x satisfies the following for all x ∈ X ,

Then, F : X → X is strongly monotone.

3.5 Bilipschitz neural operators are conditionally strongly monotone diffeomorphisms

Now we show an analogous result to Theorem 3, but applied to bilipschitz neural operators. Moreover, we will show that all neural operator F : X → X that are bilipschitz admit approximations that can be locally inverted using iteration for each point in their range using an iteration.

Theorem 4. Let X be a Hilbert space. Then there is e ∈ X , ∥ e ∥ X = 1 such that the following is true: Let F : X → X be a layer of a bilipschitz neural operator. Then for all r 1 > 0 and ϵ > 0 there are a linear invertible map A 0 : X → X , that is either the identity map or a reflection operator 3

x → x -2 ⟨ x, e ⟩ X e , and strongly monotone functions H k that are also layers of neural operators such that

where B : X → X is a compact mapping and satisfies Lip ( B ) < ϵ and

Moreover, if F ∈ C ( X,X ) , then J = O ( ϵ ) .

The proof of Theorem 4 is in Appendix A.7.1. Theorem 4 shows that we may always decompose a bilipschitz neural operator into the composition of strongly monotone neural operator layers H j and a reflection operator A 0 . Each H j can be discretized using the continuous functor A lin from Theorem 3. If we consider the discretization (via the construction in Definition 6) using a collection of subsets S 0 ( X ) ⊂ S ( X ) such that all V ∈ S 0 ( X ) satisfy e ∈ V , then the operator A 0 can be discretized by A 0 ,V = P V ◦ A 0 | V . These mean that if we write a bilipschitz neural operator as a sufficiently deep neural operator where each layer is either of the form Id + B j , where Lip ( B j ) < 1 , or a reflection operator A 0 . In either case, we may use linear discretization to approximate each layer. So, we may discretize F in a ball B X (0 , R ) by discretizing each layer H j and A 1 where F = H J ◦· · · ◦ H 1 ◦ A 1 . We observe that the number of layers, J , depends on R .

We have observed that operators of the form identity plus a compact term are critical for continuous discretization. This insight motivates the introduction of residual networks as approximators within the framework of finite-rank neural operators. In what follows, we assume that X is a separable Hilbert space, with an orthonormal basis φ = { φ n } n ∈ N . For N ∈ N , we define E N : X → R N and D N : R N → X by E N u := ( ⟨ u, φ 1 ⟩ X , ..., ⟨ u, φ N ⟩ X ) ∈ R N . D N α := ∑ n ≤ N α n φ n . We note that P V N = D N E N , where P V N : X → X is the projection onto V N := span { φ n } n ≤ N . Using E N , D N , we define the class of residual networks in the separable Hilbert space, with T, N ∈ N and activation function σ , as

The following theorem proves a universality result for each of the layers G , allowing us to obtain a general universality result for the entire network. The statement of the theorem requires the careful construction of a neural operator-representable function Φ . Giving a full description of Φ involves introducing a lot of technical notation, and so the presentation here in the main text leaves the details of the construction of Φ abridged. For the full statement of the theorem and definition of the notation, see Section A.7.2. Intuitively, Φ is the 'wrapping' of a fixed-point process in a neural operator.

Theorem 5. Let R > 0 , and let F : X → X be a layer of a bilipschitz neural operator, as in Defintion 3. Let σ be the Rectified Cubic Unit (ReCU) function defined by σ ( x ) := max { 0 , x } 3 . Then, for any ϵ ∈ (0 , 1) , there are T, N ∈ N and G ∈ R T,N,φ,σ ( X ) that has the form

such that each map ( Id X + D N ◦ NN t ◦ E N ) is strongly monotone C 1 -diffeomorphisms on some ball and

where A : X → X is a linear invertible map that is either the identity map or a reflection operator x → x -2 ⟨ x, e ⟩ X e with some unit vector e ∈ X . Further, G ◦ A : B X (0 , R ) → G ◦ A ( B X (0 , R )) is invertible, and there is some neural operator Φ : G ◦ A ( B X (0 , R )) → A ( B X (0 , R )) so that ( G ◦ A | B X (0 ,R ) ) -1 = A -1 ◦ Φ .

The proof is given in Section A.7.2. Neural operator Φ becomes a better approximation of the inverse operator when it becomes deeper. Theorem 5 means that neural operators are an operator algebra of nonlinear operators that are closed in composition and when the inverse of a neural operator exists, the inverse operator can be locally approximated by neural operators.

In the case when the separable Hilbert space X is the real-valued L 2 -function space L 2 ( D ; R ) , residual networks in the separable Hilbert space can be represented as residual neural operators defined by (179). See Lemma 11 for details. Then, we obtain the following

Corollary 1. Let D ⊂ R d be a bounded domain, and let φ = { φ n } n ∈ N be an orthonormal basis in L 2 ( D ; R ) . Assume that the orthonormal basis φ include the constant function. Let RNO T,N,φ,σ ( L 2 ( D ; R )) be the class of residual neural operators defined in (179). Then, the statement replacing X with L 2 ( D ; R ) and G ∈ R T,N,φ,ReLU ( X ) with G ∈ RNO T,N,φ,ReLU ( L 2 ( D ; R )) in Theorem 5 holds.

The proof is given by a combination of Theorem 5 and Lemma 11. We note that the assumption that the orthonormal basis φ includes the constant function is satisfied if we choose φ to be a Fourier basis, which yields the Fourier neural operator, see e.g., [39, 38].

Remark 1. In this section, we have shown that residual networks in a separable Hilbert space X , defined in (10), are universal approximators for layers of bilipschitz neural operators. Additionally, in the specific case where X = L 2 ( D ; R ) , residual neural operators, defined in Definition 9, also provide universal approximators for layers of bilipschitz neural operators. We note that the residual network we have discussed is locally invertible but not globally. By introducing invertible residual networks on Hilbert space X , defined in (166), we can similarly prove that these networks by employing sort activation functions (see [3, Section 4]) are universal approximators for strongly monotone diffeomorphisms with compact support. Specifically, when X = L 2 ( D ; R ) , invertible residual neural operators, defined in Definition 9, are also universal approximators for strongly monotone diffeomorphisms with compact support. For further details, we refer to Appendix B.

4 Quantitative approximation

Quantitative approximation results for neural networks, see e.g. [57] or [23], can be used to derive quantitative error estimates for discretization operations. Let F : B X (0 , r ) → X be a non-linear function satisfying F ∈ Lip ( B X (0 , r ); X ) , in n = 1 , or F ∈ C n ( B X (0 , r ); X ) , if n ≥ 2 . Then, F can be discretized using neural networks in the following way: Let ε V > 0 be numbers indexed by the linear subspaces V ⊂ X such that ε V → 0 as V → X . When ⃗ ε = ( ε V ) V ∈ S ( X ) , in the sense of Definition 1, an ⃗ ε -approximation operation A NN : F → ( F V ) V ∈ S ( X ) in the ball B X (0 , r ) can be be obtained by defining F V = J -1 V ◦ F V,θ ◦ J V : V → V , where J V : V → R d is an isometric isomorphism, d = dim ( V ) , F V,θ : R d → R d is a feed-forward neural network with ReLU-activation functions with at most C ( d ) log 2 ((1+ r ) /ε V ) layers and C ( d ) ε d V log 2 ((1+ r ) /ε V ) non-zero elements in the weight matrices. Details of this result are given in Proposition 5 in Appendix A.8.

5 Conclusion

In this work, we have studied the problem of discretizing neural operators between Hilbert spaces. Many physical models concern functions R n → R , for example L 2 ( R n ) , the computational methods based on approximations in finite dimensional spaces should become better when the dimension of the model grows and tends to infinity. We have focused on diffeomorphisms in infinite dimensions, which are crucial to understand in generative modeling. We have shown that the approximation of diffeomorphisms leads to computational difficulties. We used tools from category theory to produce a no-go theorem showing that general diffeomorphisms between Hilbert spaces may not admit any continuous approximations by diffeomorphisms on finite spaces, even if the approximations are allowed to be nonlinear. We then proceeded to give several positive results, showing that diffeomorphisms between Hilbert spaces may be continuously approximated if they are further assumed to be strongly monotone. Moreover, we showed that the difficulties can be avoided by considering a restricted but still practically rich class of diffeomorphisms. This includes bilipschitz neural operators, which may be represented in any bounded set as a composition of strongly monotone neural operators and strongly monotone diffeomorphisms. We then showed that such operators may be inverted locally via an iteration scheme. Finally we gave a simple example on how quantitative stability questions can be obtained for discretization functors, inviting other researchers to study related questions using more sophisticated methods.

Acknowledgments

TF was supported by JSPS KAKENHI Grant Number JP24K16949, JST CREST JPMJCR24Q5, JST ASPIRE JPMJAP2329, and Grant for Basic Science Research Projects from The Sumitomo Foundation. MP was supported by CAPITAL Services of Sioux Falls, South Dakota and NSF-DMS under grant 3F5083. MVdH was supported by the Simons Foundation under the MATH + X program, the Department of Energy, BES under grant DE-SC0020345, and the corporate members of the Geo-Mathematical Imaging Group at Rice University. A significant part of the work of MVdH was carried out while he was an invited professor at the Centre Sciences des Données at Ecole Normale Supérieure, Paris. M.L. was partially supported by a AdG project 101097198 of the European Research Council, Centre of Excellence of Research Council of Finland and the FAME flagship. Views and opinions expressed are those of the authors only and do not necessarily reflect those of the funding agencies or the EU.

References

Ahmed Abdeljawad and Philipp Grohs. Approximations with deep neural networks in sobolev time-space. Analysis and Applications , 20(03):499-541, 2022.

Giovanni S. Alberti, Matteo Santacesaria, and Silvia Sciutto. Continuous generative neural networks. arXiv:2205.14627 , 2022.

Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function approximation. In International Conference on Machine Learning , pages 291-301. PMLR, 2019.

Ivo Babuška and Manil Suri. On locking and robustness in the finite element method. SIAM J. Numer. Anal. , 29(5):1261-1293, 1992.

Heinz H. Bauschke and Patrick L. Combettes. Convex analysis and monotone operator theory in Hilbert spaces . CMS Books in Mathematics/Ouvrages de Mathématiques de la SMC. Springer, Cham, second edition, 2017. With a foreword by Hédy Attouch.

Roberto Bentivoglio, Elvin Isufi, Sebastian Nicolaas Jonkman, and Riccardo Taormina. Deep learning methods for flood mapping: a review of existing applications and future research directions. Hydrology and earth system sciences , 26(16):4345-4378, 2022.

Kaushik Bhattacharya, Bamdad Hosseini, Nikola B Kovachki, and Andrew M Stuart. Model reduction and neural networks for parametric pdes. The SMAI journal of computational mathematics , 7:121-157, 2021.

Klaus Böhmer. Numerical methods for nonlinear elliptic differential equations . Numerical Mathematics and Scientific Computation. Oxford University Press, Oxford, 2010. A synopsis.

Klaus Böhmer and Robert Schaback. A nonlinear discretization theory. J. Comput. Appl. Math. , 254:204-219, 2013.

Philippe G. Ciarlet. Linear and Nonlinear Functional Analysis with Applications . Society for Industrial and Applied Mathematics, USA, 2013.

Masoumeh Dashti, Stephen Harris, and Andrew Stuart. Besov priors for Bayesian inverse problems. Inverse Probl. Imaging , 6(2):183-200, 2012.

Maarten De Hoop, Daniel Zhengyu Huang, Elizabeth Qian, and Andrew M Stuart. The costaccuracy trade-off in operator learning with neural networks. arXiv preprint arXiv:2203.13181 , 2022.

Dennis Elbrächter, Dmytro Perekrestenko, Philipp Grohs, and Helmut Bölcskei. Deep neural network approximation theory. IEEE Transactions on Information Theory , 67(5):2581-2623, 2021.

Takashi Furuya, Michael Puthawala, Matti Lassas, and Maarten V de Hoop. Globally injective and bijective neural operators. arXiv preprint arXiv:2306.03982 , 2023.

Jean Gallier. Geometric methods and applications , volume 38 of Texts in Applied Mathematics . Springer-Verlag, New York, 2001. For computer science and engineering.

Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2414-2423, 2016.

Bruno Gavranovi´ c, Paul Lessard, Andrew Dudzik, Tamara von Glehn, João GM Araújo, and Petar Veliˇ ckovi´ c. Categorical deep learning: An algebraic theory of architectures. arXiv preprint arXiv:2402.15332 , 2024.

David Gilbarg and Neil S. Trudinger. Elliptic partial differential equations of second order . Classics in Mathematics. Springer-Verlag, Berlin, 2001. Reprint of the 1998 edition.

Evarist Giné and Richard Nickl. Mathematical foundations of infinite-dimensional statistical models . Cambridge university press, 2021.

Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. Advances in neural information processing systems , 30, 2017.

Somdatta Goswami, Aniruddha Bora, Yue Yu, and George Em Karniadakis. Physics-informed deep neural operator networks. In Machine Learning in Modeling and Simulation: Methods and Applications , pages 219-254. Springer, 2023.

Ingo Gühring, Gitta Kutyniok, and Philipp Petersen. Error bounds for approximations with deep relu neural networks in w s, p norms. Analysis and Applications , 18(05):803-859, 2020.

Ingo Gühring, Gitta Kutyniok, and Philipp Petersen. Error bounds for approximations with deep ReLU neural networks in W s,p norms. Anal. Appl. (Singap.) , 18(5):803-859, 2020.

E. M. Harrell and W. J. Layton. L 2 estimates for Galerkin methods for semilinear elliptic equations. SIAM J. Numer. Anal. , 24(1):52-58, 1987.

Isao Ishikawa, Takeshi Teshima, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Universal approximation property of invertible neural networks. Journal of Machine Learning Research , 24(287):1-68, 2023.

Thomas Jech. Set theory . Perspectives in Mathematical Logic. Springer-Verlag, Berlin, second edition, 1997.

Jari Kaipio and Erkki Somersalo. Statistical and computational inverse problems , volume 160 of Applied Mathematical Sciences . Springer-Verlag, New York, 2005.

R. I. Kaˇ curovski˘ ı. Nonlinear monotone operators in Banach spaces. Uspehi Mat. Nauk , 23(2(140)):121-168, 1968.

Gavin Kerrigan, Justin Ley, and Padhraic Smyth. Diffusion generative models in infinite dimensions. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 9538-9563. PMLR, 25-27 Apr 2023.

Nikola B Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew M Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. J. Mach. Learn. Res. , 24(89):1-97, 2023.

Anastasis Kratsios and Ievgen Bilokopytov. Non-euclidean universal approximation. Advances in Neural Information Processing Systems , 33:10635-10646, 2020.

Nicolaas H Kuiper. The homotopy type of the unitary group of hilbert space. Topology , 3(1):19-30, 1965.

Samuel Lanthaler, Zongyi Li, and Andrew M Stuart. The nonlocal neural operator: Universal approximation. arXiv preprint arXiv:2304.13221 , 2023.

Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis. Error estimates for deeponets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications , 6(1):tnac001, 2022.

Samuel Lanthaler, Andrew M Stuart, and Margaret Trautner. Discretization error of fourier neural operators. arXiv preprint arXiv:2405.02221 , 2024.

Matti Lassas and Samuli Siltanen. Can one use total variation prior for edge-preserving bayesian inversion? Inverse problems , 20(5):1537, 2004.

Markku S Lehtinen, Lassi Paivarinta, and Erkki Somersalo. Linear inverse problems for generalised random variables. Inverse Problems , 5(4):599, 1989.

Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. Journal of Machine Learning Research , 24(388):1-26, 2023.

Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 , 2020.

Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193 , 2019.

Robert J. Martin and Patrizio Neff. Minimal geodesics on gl(n) for left-invariant, right-o(n)invariant riemannian metrics. Journal of Geometric Mechanics , 8(3):323-357, 2016.

Peter W. Michor and David Mumford. A zoo of diffeomorphism groups on R n . Ann. Global Anal. Geom. , 44(4):529-540, 2013.

Donal O'Regan, Yeol Je Cho, and Yu-Qing Chen. Topological degree theory and applications , volume 10 of Series in Mathematical Analysis and Applications . Chapman & Hall/CRC, Boca Raton, FL, 2006.

Michael Puthawala, Konik Kothari, Matti Lassas, Ivan Dokmani´ c, and Maarten de Hoop. Globally injective relu networks. Journal of Machine Learning Research , 23(105):1-55, 2022.

Calvin R. Putnam and Aurel Wintner. The connectedness of the orthogonal group in Hilbert space. Proc. Nat. Acad. Sci. U.S.A. , 37:110-112, 1951.

Calvin R. Putnam and Aurel Wintner. The orthogonal group in Hilbert space. Amer. J. Math. , 74:52-78, 1952.

R. Tyrrell Rockafellar and Roger J.-B. Wets. Variational analysis , volume 317 of Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] . Springer-Verlag, Berlin, 1998.

Matti Lassas Saksman, Samuli Siltanen, et al. Discretization-invariant bayesian inversion and besov space priors. arXiv preprint arXiv:0901.4220 , 2009.

Martin H. Schultz. Error bounds for the Rayleigh-Ritz-Galerkin method. J. Math. Anal. Appl. , 27:524-533, 1969.

R. E. Showalter. Monotone operators in Banach space and nonlinear partial differential equations , volume 49 of Mathematical Surveys and Monographs . American Mathematical Society, Providence, RI, 1997.

Andrew M Stuart. Inverse problems: a bayesian perspective. Acta numerica , 19:451-559, 2010.

Manil Suri, Ivo Babuška, and Christoph Schwab. Locking effects in the finite element approximation of plate models. Math. Comp. , 64(210):461-482, 1995.

Michael E. Taylor. Partial differential equations I. Basic theory , volume 115 of Applied Mathematical Sciences . Springer, New York, second edition, 2011.

Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling-based invertible neural networks are universal diffeomorphism approximators. Advances in Neural Information Processing Systems , 33:3362-3373, 2020.

Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature , 620(7972):47-60, 2023.

Jared Willard, Xiaowei Jia, Shaoming Xu, Michael Steinbach, and Vipin Kumar. Integrating scientific knowledge with machine learning for engineering and environmental systems. ACM Computing Surveys , 55(4):1-37, 2022.

Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks , 94:103-114, 2017.

A Proofs and additional examples

A.1 Non-linear discretization

In this section we consider the non-linear discretization theory, see [9]. Let Ω ⊂ R n be a bounded open set with smooth boundary. Let H 1 (Ω) = { u ∈ L 2 (Ω) | ∇ u ∈ L 2 (Ω) } and H 1 0 (Ω) = { u ∈ H 1 (Ω) | u | ∂ Ω = 0 } be the Sobolev spaces with an integer order of smoothness and H s (Ω) , s ∈ R the Sobolev space with a fractional order of smoothness [53].

For simplicity, we assume in the section that n = 1 , and that Ω ⊂ R is an interval and we point out that in this case the Sobolev embedding theorem implies H 1 (Ω) ⊂ C (Ω) which significantly simplifies the constructions below.

Let F = F ( g ) : H 1 0 (Ω) → H 1 0 (Ω) , F ( g ) : x → u be the solution operator of the non-linear differential equation

where ∆ is the Laplace operator operating in the t variable, where the function x ( t ) is a physical source term. Because the u that solves (12)-(13) is the minimizer of the strictly convex and lower semi-continuous function H : X → R ,

the Weierstrass theorem, see [50], Theorem II.8.1, implies that (12)-(13) has a unique solution. Moreover, by regularity theory for elliptic equations, see [18], Theorem 8.13, we have that the solution u is in H 3 (Ω) . Moreover, as G is convex and satisfies (11), the function H in (14) is coercive, for any R > 0 there is ρ ( R ) > 0 such that when ∥ x ∥ H 1 0 (Ω) ≤ R , then ∥ u ∥ H 3 (Ω) ≤ ρ ( R ) .

The equation (12)-(13) can be written also as an integral equation

where Ψ( t, t ′ ) is the Dirichlet Green's function of the negative Laplacian, that is, -∆Ψ( · , t ′ ) = δ t ′ ( · ) , Ψ( · , t ′ ) | ∂ Ω = 0 , and g ∗ ( u ) = g ( u ) . We can write (15) as

where i H 1 0 → H 3 / 4 : H 1 0 (Ω) → H 3 / 4 (Ω) and i H 2 → H 1 0 : H 2 (Ω) ∩ H 1 0 (Ω) → H 1 0 (Ω) are compact operators and as H 3 / 4 (Ω) ⊂ C (Ω) , the operators g ∗ : H 3 / 4 (Ω) → L 2 (Ω) and Q : L 2 (Ω) → H 2 (Ω) ∩ H 1 0 (Ω) are continuous, we see that

is a layer of the neural operator. As G ∈ C m +1 , m ≥ 2 , we have that F : H 1 0 (Ω) → H 1 0 (Ω) is C 1 -smooth. Moreover, as G is convex, the operator F : H 1 (Ω) → H 1 (Ω) is strongly monotone and hence by Lemma 3, F has an inverse map F -1 : H 1 (Ω) → H 1 (Ω) that is C 1 -smooth. Thus, the solution u of (12)-(13) can be represented as

Let V ⊂ X = H 1 0 (Ω) be a finite dimensional space. The Galerkin methods to obtain an approximation for the solution of the boundary value problem (12)-(13) using Finite Element Method. To do this, let w ∈ V be the solution of the problem

When ϕ 1 , . . . , ϕ d ∈ V are a basis of the finite dimensional vector space V , and we write w = ∑ d k =1 w k ϕ j , the problem (17) is equivalent that ( w 1 , . . . , w d ) ∈ R d satisfies the system equations d equations

where

When x ∈ V , we define the map F ( g ) V : V → V by setting map F ( g ) V ( x ) = w , where w ∈ V is the solution of the problem (17). When

we define the map A FEM : F ( g ) → F ( g ) V for F ( g ) ∈ F PDE . As the the function g in the equation (18) is non-linear, the solution u of the continuum problem (15)-(13) and the solution w of the finite dimensional problem have typically no linear relationship, even if the source x satisfies x ∈ V.

Let R > 0 As the embedding H 2 (Ω) ∩ H 1 0 (Ω) → H 1 0 (Ω) is compact and Q : H 1 0 (Ω) → H 2 (Ω) ∩ H 1 0 (Ω) is continuous, we see that the image of the closed ball B X (0 , R ) in the map Q is a precompact set Q ( B X (0 , R )) ⊂ X . As F -1 : X → X is continuous the set of corresponding solutions,

is precompact in X. Hence,

As G is convex, the Fréchet derivative of the map F = Id + Q ◦ g ∗ : X → X is a strictly positive linear operator at all points x ∈ X . This, the uniform convergence (19), and the convergence results for the Galerkin method for semi-linear equations, [49, Theorem 3.2], see also [24], imply that in the space X = H 1 0 (Ω)

These imply that A FEM : F ( g ) → F ( g ) V is an approximation operation for the function F ( g ) ∈ F PDE .

The map F ( g ) V : V → V is called the Galerkin approximation of the problem (12)-(13) and is an example of the non-linear approximation methods. The properties of the Galerkin approximation is studied in detail in [8], in particular sections 4.4 and 4.5.

A.2 Negative results

On the positive results, in Appendix A.1, we have considered nonlinear discretiation of the operators u → -∆ u + g ( u ) . To exemplify the negative result, we consider the following example on the solution operation of differential equations and the non-existence of approximation by diffeomorphic maps: We consider the elliptic (but not not strongly elliptic) problem

with the Dirichlet and Neumann boundary conditions

Here, 0 ≤ s ≤ 1 is a parameter of the coeffient function and sign ( t -s ) = 1 if t > s and sign ( t -s ) < 0 if t < s . We consider the weak solutions of (21) in the space

We can write where

Then the equation,

parametrized by 0 ≤ s ≤ , are multiplication operations that are invertible operators, A s : L 2 (0 , 1) → L 2 (0 , 1) (this invertibility makes the equation (21) elliptic). Moreover, D (1) t and D (2) t are the operators v → d dt v with the Dirichlet boundary condition v (0) = 0 and v (1) = 0 , respectively. We consider the Hilbert space X = H 1 D,N (0 , 1) ; to generate an invertible operator G s : X → X related to (21), we write the source term using an auxiliary function g ,

defines a continuous and invertible operator,

In fact, G s = B -1 s ◦ Q when the domains of B s and Q are chosen in a suitable way. The Galerkin method (that is, the standard approximation based on the Finite Element Method) to approximate the equation (23) involves introducing a complete basis χ j ( t ) , j = 1 , 2 , . . . of the Hilbert space X , the orthogonal projection

and approximate solutions of (23) through solving

This means that operator B -1 s Q : g → u is approximated by ( P n B s P n ) -1 P n QP n : g n → u n , when P n B s P n : X n → X n is invertible.

The above corresponds to the Finite Element Method where the matrix defined by the operator P n B s P n is b ( s ) = [ b jk ( s )] n j,k =1 ∈ R n × n , where

Since we used the mixed Dirichlet and Neumann boundary conditions in the above boundary value problem, we see that for s = 0 all eigenvalues of the matrix b ( s ) are strictly positive, and when s = 1 all eigenvalues are strictly negative. As the function s → b ( s ) is a continuous matrix-valued function, we see that there exists s ∈ (0 , 1) such that the matrix b ( s ) has a zero eigenvalue and is no invertible. Thus, we have a situation where all operators B -1 s Q : g → u , s ∈ [0 , 1] are invertible (and thus define diffeomorphisms X → X ) but for any basis χ j ( t ) and any n there exists s ∈ (0 , 1) such that the finite dimensional approximation b ( s ) : R n → R n is not invertible. This example shows that there is no FEM-based discretization method for which the finite dimensional approximations of all operators B -1 s Q , s ∈ (0 , 1) , are invertible. The above example also shows a key difference between finite and infinite dimensional spaces. The operator A s : L 2 (0 , 1) → L 2 (0 , 1) has only continuous spectrum and not eigenvalues nor eigenfunctions whereas the finite dimensional matrices have only point spectrum (that is, eigenvalues). The continuous spectrum makes it possible to deform the positive operator A s with s = 0 to a negative operator A s with s = 1 in such a way that all operators A s , 0 ≤ s ≤ 1 , are invertible but this is not possible to do for finite dimensional matrices. We point out that the map s → A s is not continuous in the operator norm topology but only in the strong operator topology and the fact that A 0 can be deformed to A 1 in the norm topology by a path that lies in the set of invertible operators is a deeper result. However, the strong operator topology is enough to make the FEM matrix b ( s ) to depend continuously on s .

A.3 Application of Theorem 2 to Neural Operators

In this section, we give an example of the application of Theorem 2, as applied to the problem of using a neural operator to solve a linear PDE. The guiding idea is to draw inspiration from Figure 1; the space of diffeomorphisms on X is disconnected if X is a finite-dimensional space, but connected if it is a Hilbert space.

Before constructing the path, we first construct a isotopy of diffeomorphisms in L 2 (Ω) in several steps

Let ρ : R → [0 , 1] be a smooth function such that ρ | ( -∞ , 0] = 0 , ρ | [1 , ∞ ) = 1 . Let R ( θ ) ∈ R 2 × 2 be the rotation matrix given by

Then, define the function R i : R → R 2 × 2 by

Note that R i ( t ) ∈ SO (2) , for t ≤ i , R i ( t ) = I , t ≥ i +1 , R i ( t ) = -I .

Next, we consider the 'little ell' space, ℓ 2 := { ( α 1 , α 2 , . . . , ): ∑ ∞ i =1 α 2 i < ∞ } . Consider the linear operator ˆ R : R → ( ℓ 2 → ℓ 2 ) on ℓ 2 given by a pair-wise coordinate representation given by the infinite dimensional 'matrix' with block structure

Finally, define R : [0 , 1] → ( ℓ 2 → ℓ 2 ) by

and define the related operator ˜ R : [0 , 1] → ( ℓ 2 → ℓ 2 ) by

Proposition 3 ( R , ˜ R are isotopies of smooth diffeomorphisms in ℓ 2 ) . The maps H 1 : ℓ 2 × [0 , 1] → ℓ 2 and H 2 : ℓ 2 × [0 , 1] → ℓ 2 defined by

are isotopies of smooth diffeomorphisms in ℓ 2 and agree at t = 1 .

Proof. We prove that H 1 is an isotopy of smooth diffeomorphisms on ℓ 2 , as the proof of the same for H 2 is very similar.

We first show that H 1 ( · , t ) ∈ diff( ℓ 2 ) for each t .

For t < 1 , R operates on vectors in ℓ 2 in only a finite number of indices. In those indices it corresponds to a rotation, and hence is a smooth diffeomorphism. When t ≥ 1 , it corresponds to scalar multiplication by -1 , and so is too a smooth diffeomorphism. Now we show that R ( t ) is continuous on . The only point where continuity may fail is at t = 1 , and there it may only fail in the

limit from the left. Given a v ∈ ℓ 2 , we compute

Let k ∗ ( t ) := ⌊ 1 1 -t ⌋ . Then for any integer i ≤ k , we have that

and likewise when i ≥ k ∗ ( t ) + 1 , then R i ( 1 1 -t ) = I ∈ R 2 × 2 , and so the r.h.s. of Eqn. 31 becomes

As t → 1 -, k ∗ ( t ) → ∞ , and so, by standard estimates, 2 √∑ ∞ i = k ∗ ( t ) v 2 i → 0 . Hence, R ( t ) is continuous on ℓ 2 at t = 1 .

Finally, by definition at t = 1 , we have that H 1 ( v, 1) = R (1) = -I = R (1) = H 2 ( v, 1) .

Finally, we define H : R ∞ × [0 , 1] → R ∞ by gluing the isotopies H 1 and H 2 together by

Proposition 4. The function H given by Eqn. 36 is an isotopy of diffeomorphisms in ℓ 2 .

Proof. Continuity of H follows from continuity of R , ˜ R , and that R (1) = ˜ R (1) . For each t ∈ [0 , 1] , H ( · , t ) ∈ diff( ℓ 2 ) .

A.4 Proofs from Sec. 3.1

A.4.1 Proof of Theorem 2

Proof. Assume that A : D → B is a functor that satisfies the property (A) of an approximation functor and is continuous. Let us consider the case when X = ℓ 2 .

Let e ∈ X be a unit vector and B e : X → X be a linear map Bx = x -2 ⟨ x, e ⟩ X e . In other words, B is a diagonal matrix diag ( -1 , 1 , . . . ) in some orthogonal basis where e is the first basis vector.

By [32, Theorem 2], see also and [45], for an infinite dimensional Hilbert space X the space GL ( X ) of linear invertible maps have only one topological component in the operator norm topology and the set GL ( X ) is contractible and hence path-connected. This implies that there are linear maps A t : X → X , t ∈ [0 , 1] such that A 0 = Id , and A 1 ∈ GL ( X ) is arbitrary invertible linear map and that t → A t ∈ GL ( X ) is continuous. Similarly, in an infinite dimensional real Hilbert space X the space O R ( X ) of the orthogonal operators A : X → X is path-connected in the operator norm topology, see [46, Section 4]. We recall that an orthogonal operator A : X → X is a bounded linear operator satisfying A ∗ A = AA ∗ = I , where A ∗ : X → X is the adjoint (i.e., the transpose) of the operator A : X → X . Observe that in a finite dimensional space R n the set O R ( R n ) has two disjoint topological components, those which determinant is 1 and those which determinant is -1 , and thus the set O R ( R n ) is not connected.

A bounded linear operator B : X → X is called a rotation operator of the form B = e S where S : X → X is a linear, bounded, skew-symmetric operator, S ∗ = -S , and e S = I + S + 1 2! S 2 + 1 3! S 3 + . . . . An orthogonal operator A is called a reflection if it is not a rotation.

One of the fundamental reasons for the surprising property that the space O R ( X ) of orthogonal operators in X is path-connected is that in an infinite dimensional real Hilbert space X every orthogonal operator A : X → X can be represented as a product of two rotation operators (that is no valid in the finite dimensional spaces), that is, A = e S 1 e S 2 where S 1 , S 2 : X → X are skew-symmetric linear operators. Thus any operator A ∈ O R ( X ) can be connected to the identity operator Id : X → X via a path α : t → e tS 1 e tS 2 ∈ O R ( X ) , t ∈ [0 , 1] , so that α (0) = Id and α (1) = A.

As a motivating example on the fact that O R ( X ) is connected in the operator norm topology, is to consider a similar, but easier result in the strong operator topology, that is, topology generated by the evaluation maps A → A ( u ) , u ∈ L 2 (0 , 1) . Observe that the strong operator topology is weaker than the operator norm topology, but in finite dimensional space those are equivalent.. So, let us next show that the operators Id and -Id are in the same topological component of O R ( X ) in the strong operator topology. As all infinite dimensional separable Hilbert spaces are isometric to L 2 (0 , 1) , let us consider the Hilbert space L 2 ( R ) and the orthogonal linear operators A s : L 2 (0 , 1) → L 2 (0 , 1) , s ∈ [0 , 1] that are given by

where u ∈ L 2 (0 , 1) and sign ( t ) is the sign of the real number t . Then, for every u ∈ L 2 (0 , 1) the functions

are continuous functions a u : [0 , 1] → L 2 (0 , 1) . As A 0 = Id and A 0 = -Id , we see that

is a continuous path in the strong operator topology, that connects the operator Id to -Id .

We recall that the continuity of the function s → A s in the strong operator topology, means that for all u ∈ L 2 (0 , 1) the map

is continuous. As for s > s 0

as s → s 0 , we see that s → A s is continuous in the strong operator topology. All maps A s : L 2 (0 , 1) → L 2 (0 , 1) are invertible and A 0 = Id , A 1 = -Id . This has implications e.g. for Finite Element method analysis of partial differential equations (See Appendix A.5.2).

Note that when X is any infinite dimensional separable Hilbert space with real scalars, there is an isometry J : X → L 2 (0 , 1) (e.g., the linear operator mapping an orthogonal basis of X to an orthogonal basis L 2 (0 , 1) . Then, the maps ˜ A s : X → X given by ˜ A s = J -1 ◦ A s ◦ J : X → X , where A s : L 2 (0 , 1) → L 2 (0 , 1) are given above, are continuous paths in O R ( X ) from ˜ A 0 = Id : X → X to ˜ A 1 = -Id : X → X . As discussed above, the deep result that Id and -Id can be connected by a continuous path in the operator norm topology of O R ( X ) is given in [32, Theorem 2], see also [45].

Let us first warm up by proving the claim in the case when an additional assumption (B) is valid:

(B) : When V ⊂ X is finite dimensional discretization maps F V of linear invertible maps F : X → X are linear invertible maps and moreover, the set S 0 ( X ) = S ( X ) consists of all linear subspaces of X .

Under assumptions (A) and (B), we consider the case when A 1 = -Id and denote by F t = A t : X → X a family of linear maps such that A 0 = Id , and that t → A t ∈ GL ( X ) is continuous path of operators connecting Id to A 1 = -Id . Let ( X,S ( X ) , ( F t,V ) V ∈ S ( X ) ) = A ( X,F t ) , that is, F t,V : V → V be the linear isomorphism that is the discretizations of the map F t : X → X . As the functor A is continuous, the map t → F t,V is a continuous map from [0 , 1] to the space of linear operators endowed with the topology of uniform convergence on compact sets, c.f. limit (7). As F t,W are linear, this implies that for all t ′ ∈ [0 , 1] ,

and hence the map t → F t,W is a continuous map [0 , 1] → GL ( W ) .

Let 0 < ϵ 0 < 1 / 2 and r > 1 . Let t 0 = 0 and t 1 = 1 . Using Property (A) for operators F t j , j = 0 , 1 , we see that there are W 0 , W 1 ∈ S 0 ( X ) such that for all V ∈ S 0 ( X ) satisfying V ⊃ W j we have

Let W ∈ S 0 ( X ) be such a finite dimensional space which dimension is an odd integer and that W 0 + W 1 ⊂ W . Then,

Clearly, A 0 = Id satisfies A 0 ( W ) = W and A 0 : W → W is an invertible linear map. Similarly, the map A 1 = -Id satisfies A 1 ( W ) = W and A 1 : W → W is an invertible linear map. These observation and assumptions (A) and (B) imply that F 0 ,W : W → W and F 1 ,W : W → W are linear maps and by inequality (39),

and as the dimension of W is odd, we have

Let B 0 ,s = (1 -s ) A 0 | W + sF 0 ,W : W → W and B 1 ,s = (1 -s ) A 1 | W + sF 0 ,W : W → W . As ∥ ( A 0 | W ) -1 ∥ W → W = ∥ ( A 1 | W ) -1 ∥ W → W = 1 , we see using (40) that the maps B -1 0 ,s : W → W and B -1 1 ,s : W → W are invertible and that the functions s → B 0 ,s ∈ GL ( W ) and s → B 1 ,s ∈ GL ( W ) are continuous matrix-valued functions which determinants does not vanish and

for all s ∈ [0 , 1] . These imply that

However, as t → F t,W is a continuous maps [0 , 1] → GL ( W ) , see (38), we have that

is a continuous function [0 , 1] → R which does not obtain value zero and thus has a constant sign. However, this is in contradiction with formula (41).

Next we return to the main part of the proof where we do not assume that assumption (B) is valid, but we only assume that the assumption (A) is valid. In this case, we use degree theory instead of the determinants.

Next, let F t = A t : X → X be a family of linear maps such that A 0 = Id , but that A 1 ( x ) = B e ( x ) = x -2 e ⟨ x, e ⟩ X is a reflection, where e ∈ X is a unit vector, and t → A t ∈ GL ( X ) is continuous. Again, let ( X,S 0 ( X ) , ( F t,V ) V ∈ S 0 ( X ) ) = A ( X,F t ) , that is, F t,V : V → V be C 1 -diffeomorphisms that are discretizations of the map F : X → X . As the functor A is continuous, the map t → F t,V | B (0 ,R ) ∈ C ( B (0 , R )) is a continuous map for all V ∈ S 0 ( X ) and R > 0 . Moreover, we note that by our assumptions on B , the discretized map F t,V : V → V , is C 1 -diffeomorphism of V .

As V is a finite dimensional vector space, we can use finite dimensional degree theory and consider the degree deg ( F t,V , V ∩ B X (0 , r ) , p ) , that is the degree of the map F t,V : V ∩ B X (0 , r ) → V at the point p . We recall that when Ω ⊂ R d is open and bounded, F : Ω → R d is a C 1 -smooth function and p ∈ R d are such that p ̸∈ f ( ∂ Ω) and for all x ∈ f -1 ( p ) the derivative Df ( x ) is an invertible matrix, the degree is defined to be

where sgn ( r ) is sign of a real number r . Also, the map f → deg ( h, Ω , p ) is defined for a continuous function h : Ω → R d by approximating h by C 1 -smooth function, see [43], Definition 1.2.5 on details. Let us denote B W (0 , r ) = W ∩ B X (0 , r ) .

Let r > 0 . Recall that by assumption (A), for F 0 = Id : X → X and F 1 = B e : X → X there are finite dimensional spaces V 0 ∈ S 0 ( X ) and V 1 ∈ S 0 ( X ) such that e ∈ V 0 and e ∈ V 1 and that when V ∈ S 0 ( X ) satisfies V 0 ⊂ V and V 1 ⊂ V , then

Moreover, let W ∈ S 0 ( X ) be such a finite dimensional linear space that V 0 ⊂ W and V 1 ⊂ W . Observe that as e ∈ W , we can decompose W as

and denote W ′ = { x ∈ W : ⟨ x, e ⟩ X = 0 } . We see that

Next we use the facts that F 0 (0) = Id (0) = 0 and F 1 (0) = B e (0) = 0 . Let us define the maps

that are C 1 -smooth maps B W (0 , r ) → W . Moreover, for j = 0 , 1 , we have f 0 ( ∂B W (0 , r )) = ∂B W (0 , r ) and f 1 ( ∂B W (0 , r )) = ∂B W (0 , r ) . Let p j := f j,W (0) . Then, by (43) and (44) we have

and

and

This implies that

Then, by [43], Definition 1.2.5, the formula (48) implies that

Moreover, as F 1 = A 1 = B e satisfies (46), and p j ∈ B W (0 , r ) , we have

and

Moreover, by our assumptions on B , the discretized maps F j,W : W → W , j = 0 , 1 are C 1 -diffeomorphism. Hence, the inverse image F -1 j,W ( { p j } ) is a set containing only one point that coincides with f -1 j,W ( { p j } ) (that in fact is the set { 0 } ) . This and the above show that for any R > r , we have

Let us now consider the maps F t,W : W → W where t ∈ [0 , 1] , that are C 1 -diffeomorphism f t,W : W → W . As t → F t = A t is a continuous map [0 , 1] → GL ( X ) and A : D → B is a continuous functor, the map t → F t,W is a continuous map [0 , 1] → C ( B W (0 , r ); W ) .

Let us consider ˆ t ∈ [0 , 1] and denote p t = F t,W (0) . As F ˆ t,W : W → W is a C 1 -diffeomorphism, the set F ˆ t,W ( B W (0 , r )) is open and hence there is ϵ > 0 such that

As t → F t,W is a continuous map [0 , 1] → C ( B W (0 , r ); W ) , there is δ > 0 such that when | t -ˆ t | < δ then

In particular, this implies that

This implies that

and thus p t = F t,V (0) and p ˆ t = F ˆ t,V (0) are in the ball B W ( p ˆ t , 5 ϵ ) and hence by (51) these points are in the same topological component of by W \ F ˆ t,W ( ∂B W (0 , r )) . Hence, it follows from [43], Theorem 1.2.6 (5), that for | t -ˆ t | < δ , we have

Next, we observe that by (53), for any t 1 , t 2 ∈ [0 , 1] satisfying | t 1 -ˆ t | < δ and | t 2 -ˆ t | < δ ,

Inequalities (52), (54) and (56) imply that for any t 1 , t 2 ∈ [0 , 1] satisfying | t 1 -ˆ t | < δ and | t 2 -ˆ t | < δ we have

As t → F t,W is a continuous map [0 , 1] → C ( B W (0 , r ); W ) and the formula (57) is valid, it follows from [43], Theorem 1.2.6 (3) that for t 1 and t 2 satisfying | t 1 -ˆ t | < δ and | t 2 -ˆ t | < δ , we have

This and (55) imply that

In particular, when t 1 = t 2 satisfy | t 1 -ˆ t | < δ , this implies

As ˆ t ∈ [0 , 1] is above arbitrary, this implies that the function

is a continuous integer valued function on [0 , 1] and thus it is constant on the interval t ∈ [0 , 1] . However, by (49), g (0) = 1 is not equal to g (1) = -1 and hence g ( t ) can not be constant function on the interval t ∈ [0 , 1] . This contradiction shows that the required functor does not exists.

A.5 Proofs from Sec. 3.2

A.5.1 Proof of Lemma 1

Proof. Let F : X → X be strongly monotone and C 1 -diffeomorphism. It is obvious that strongly monotonicity implies that strictly monotonicity.

We first show that the strongly monotonicity implies that the coercivity. Indeed, by the strongly monotonicity we have

which implies that, as ∥ x ∥ X →∞ ,

Therefore, F : X → X is coercive.

Let us consider the operator

From the definitions, F V : V → V is C 1 and strongly monotones, and the (Fréchet) derivative DF V | v at v ∈ V is given by

which is linear and continuous operator from V to V . Since F V : V → V is C 1 and strongly monotone, it is hemicontinuous, strictly monotones, and coercive. By the Minty-Browder theorem [10, Theorem 9.14-1], F V : V → V is bijective, and then its inverse F -1 V : V → V exists.

Next, let v ∈ V . As F V : V → V is strongly monotones, we estimate that for all h ∈ V ,

Taking ϵ → +0 , we have that

Therefore, DF V | v : V → V is injective for all v ∈ V . Then, it is bijective because V is now finite dimensional. By the inverse function theorem, the inverse F -1 V : V → V is C 1 .

A.5.2 Additional examples

Let us consider the elliptic (but not stongly elliptic) problem

with the boundary conditions

The FEM (Finite Element Method) matrix corresponding to this problem is [ p jk ( s )] n j,k =1 , where

and χ j ( t ) , j = 1 , 2 , . . . are a complete basis of the Hilbert space H 1 D,N (0 , 1) := { v ∈ H 1 (0 , 1) : v (0) = 0 } that correspond to the mixed Dirichlet and Neumann boundary conditions. Note that in the FEM approximation we use only a finite subset of the complete basis of the Hilbert space. Note for u in the canonical domain of the above problem that makes the operator appearing in the above equation selfadjoint (the Friedrichs extension), that is, for

we have

where A s is the multiplication with the function sign ( t -s ) , see (37), and D (1) t and D (2) t are the derivative operators D ( j ) t u = d dt u having the different domains

Observe that D (1) t and D (2) t have the inverse operators

that map ( D ( j ) t ) -1 : L 2 (0 , 1) →D ( D ( j ) t ) . The eigenvalues of the matrix s → [ p jk ( s )] n j,k =1 change from positive values to negative values when s moves from 0 to 1 . Thus, for some value s ∈ (0 , 1) the matrix [ p jk ( s )] n j,k =1 has a zero eigenvalue and is no invertible even though all maps D (1) t , D (2) t , and A s : L 2 (0 , 1) → L 2 (0 , 1) are invertible. In particular, there are no finite FEM basis in where the Galerkin discretizations of all operators P s , s ∈ [0 , 1] are invertible.

Let us consider even simpler example: Similarly to the above, if we consider the Galerkin discretizations of the operator A s : L 2 (0 , 1) → L 2 (0 , 1) , we see that the Galerkin matrix corresponding to A s : L 2 (0 , 1) → L 2 (0 , 1) is [ a jk ( s )] n j,k =1 , where

and ψ j ( t ) , j = 1 , 2 , . . . are a complete basis of the Hilbert space L 2 (0 , 1) . Again, we see that the eigenvalues of the matrix s → [ a jk ( s )] n j,k =1 change from positive values to negative values when s moves from 0 to 1 . Thus, for some value s ∈ (0 , 1) the matrix [ a jk ( s )] n j,k =1 has a zero eigenvalue and is no invertible even though all maps A s : L 2 (0 , 1) → L 2 (0 , 1) are invertible. Thus there are no finite basis in where the Galerkin discretizations of all operators A s , s ∈ [0 , 1] are invertible.

A.6 Proofs from Sec. 3.4

A.6.1 Proof of Lemma 4

Proof. From the assumption, it holds that

Then, by the mean value theorem, there is 0 < t < 1 such that

These imply that F : X → X is strongly monotone.

A.6.2 Proof of Lemma 3

Proof. Observe that strongly monotone neural operators are coercive, that is,

and therefore F : X → X is surjective by Browder-Minty theorem [28, Thm. 2.1], see also [14] considerations for neural operators. Moreover, the derivatives DF | x are linear strongly monotone operators for all x ∈ X , and therefore injective. Observe that the derivative is a linear operator of the form

and as T 1 and T 2 are compact and DG | Tx is bounded we see that DF | x is a Fredholm operator of index zero. Observe that

where h ∈ R and v ∈ X , and hence DF | x : X → X is an injective linear operator. As DF | x is a Fredholm operator of index zero, this implies that the derivative DF | x : X → X is a bijection. As the Hilbert space X can be identified with it dual space and the operator F : X → X is continuous and hence hemi-continuous, it follows from [28], Theorem 3.1, that the map F : X → X is a homeomorphism. As it is C 1 -smooth and its derivative is bijective operator DF | x : X → X at all x ∈ X , it follows that F : X → X is a C 1 -smooth diffeomorphism.

A.6.3 Proof of Theorem 3

Proof. To show the well-definedness of the discretization functor A lin , we need to show that, for each strongly monotone C 1 -function F : X → X that is of the form (4), F V = P V F | V : V → V is still strongly monotone C 1 -smooth of the form (4). This is given by Lemma 1. Moreover, such functions are C 1 -smooth diffeomorphisms.

To verify assumption (A), let r, ϵ > 0 , and let F : X → X be a strongly monotone diffeomorphisms that is of the form F = Id + T 2 ◦ G ◦ T 1 : X → X where T 1 : X → X and T 2 : X → X are compact linear operators, and G : X → X is such that G ∈ C 1 ( X ) . Since T 2 ◦ G ◦ T 1 is a compact mapping, there is a finite-dimensional subspace V ⊂ X , depending on ϵ > 0 such that

which implies that

To prove the continuity in the sense of Definition 8 let r > 0 and let V ∈ S 0 ( X ) . Assume that

where F ( j ) and F are strongly monotone diffeomorphisms F : X → X that are of the form (4). Then, we see that, as j →∞ ,

A.7 Proofs from Sec. 3.5

A.7.1 Proof of Theorem 4

In the proof of Theorem 4, we prove first few auxiliary lemmas

Lemma 5. A layer of a neural operator F : X → X is surjective. In particular, if F : X → X is bilipschitz, then F : X → X is a homeomorphism.

Proof. By formula (4) a layer of neural operator F : X → X is of the form F ( u ) = u + T 2 G ( T 1 u ) where T 1 : X → X and T 2 : X → X are compact linear operators and G : X → X is a function in C 1 ( X ) . Let c 0 , c 1 > 0 be such that ∥ G ∥ L ∞ ( X ) ≤ c 0 and Lip X → X ( G ) ≤ c 1 .

Let p ∈ X and R 0 > ∥ T 2 ∥ X → X c 0 + ∥ p ∥ X . Then K p : X → X , defined by the formula

is a compact non-linear operator, see [43, Definition 2.1.11].

When p satisfies p ̸∈ ( Id -K 0 )( ∂B (0 , R 0 )) , let deg ( Id -K 0 , B (0 , R 0 ) , p ) = deg ( Id -K p , B (0 , R 0 ) , 0) be the infinite dimensional (Leray-Schauder) degree of the operator Id -K 0 : X → X in the set B (0 , R 0 ) with respect to the point p , see [43, Definition 2.2.3]. Let K p ; t : X → X be the non-linear compact operators that depend on the parameter t ∈ [0 , 1] , obtained by multiplying K p ( x ) by the number t , that is,

As ∥ K p ; t ( x ) ∥ X ≤ ∥ T 2 ∥ X → X c 0 + ∥ p ∥ X < R 0 for all t ∈ [0 , 1] and x ∈ X , we see that

̸

Then by the homotopy invariance of the Leray-Schauder degree, see [43, Theorem 2.2.4(3)], the function

is a constant function. Moreover, by [43, Theorem 2.2.4(1)],

By [43, Theorem 2.2.4(2)], this implies that the equation

has a solution x ∈ B X ( R 0 ) . As p ∈ X was above arbitrary, this implies that F = Id + T 2 ◦ G ◦ T 1 : X → X is surjection.

Finally, if F : X → X is bilipschitz, it is a bijection and its inverse function is Lipschitz function, and thus F : X → X is a homeomorphism.

The next lemma shows the existence of a (finite-dimensional) orthogonal subspace so that for each compact operator, projection onto the subspace is a perturbation of the identity under either pre or post composition.

Lemma 6. Let T 1 , T 2 : X → X be compact operators and h > 0 . There is a finite dimensional space W ⊂ X such that for the orthogonal projector P W : X → X it holds that

Proof. We use the singular value decomposition of compact operators: We can write

where ψ ℓ,p and ϕ ℓ,p are orthogonal families in X and ω ℓ,p ≥ 0 satisfy ω ℓ,p +1 ≤ ω ℓ,p and ω ℓ,p → 0 as p →∞ . For all h > 0 we can choose P > 0 such that ω ℓ,p < h when p ≥ P . Then

we see that if W ⊂ X is a linear subspace such that V ⊂ W then

Observe that for w ∈ W and v ∈ W ⊥ we have

This means that F W = Id W ⊥ ⊕ F W | W , where F W | W : W → W is a function which maps the finite dimensional vector space W to itself.

The lemma below shows that given an F and ϵ > 0 , we may perturb it by a Lipschitz term B so it becomes the operator F W .

If and

and thus and moreover,

Let and

Lemma7. For any ϵ > 0 , there is a finite dimensional space W ⊂ X such that Lip X → X ( F -F W ) < ϵ and for any ball B X ( R ) ⊂ X , R > 0 , the maps F : B X ( R ) → X and F W : B X ( R ) → X satisfy ∥ F -F W ∥ L ∞ ( B X ( R )) < 1 2 (1 + R ) ϵ .

Proof. Let us choose a finite dimensional space W ⊂ X so that Lemma 6 is valid with

The right hand side of (72) is chosen so that

and

Lemma 8. For any ϵ > 0 , there are finite dimensional space W ⊂ X and (possibly non-linear) functions B : X → X and ˜ B : X → X such that

Moreover, B : X → X is a compact non-linear operator of the form

where F 1 , F 2 , F 3 , F 4 : X → X are functions in C 1 ( X ) and ˜ B : X → X is of the same form. In addition, the operator Id + B : X → X and Id + ˜ B : X → X are layers of neural operators.

Proof. Below, let C 0 > 0 be such that

and let W be the space in Lemma 6 with

By Lemma 5, F ( x ) = x + T 2 G ( T 1 ( x )) is by an invertible function F : X → X . Thus we can define a non-linear operator

and

Let

Then for all x, y ∈ X

and thus,

This implies that F W : X → X satisfies for all x, y ∈ X

and thus and

Moreover, for all x, y ∈ X

Let c 0 > 0 be such that ∥ G ∥ L ∞ ( X ) ≤ c 0 . Let R 0 > 0 . As ∥ T 2 ◦ G ∥ L ∞ ( X ) ≤ ∥ T 2 ∥ c 0 , finite dimensional degree theory, see using [43, Theorem 1.2.6], as above that

when R 1 > R 0 + ∥ T 2 ∥ c 0 ≥ R 0 + ∥ P W ◦ T 2 ◦ G ◦ T 1 ◦ P 2 ∥ L ∞ ( X ) . As R 0 > 0 above is arbitrary, formula (93) implies that F W : X → X is surjective. Thus, we have shown that F W : X → X is a bijective bilipschitz map.

Similarly to the above, by replacing (85) by (92) and changing the roles of F W and F , we see that for x, y ∈ X

and

The above implies that

where satisfy

Next we use that so that

Observe that by (86)

and as P W : X → X is a finite rank operator and T 2 : W → W is a compact linear operator, we see that B : X → X is a compact non-linear operator of the form

where F 1 , F 2 , F 3 , F 4 : X → X are functions in C 1 ( X ) .

Moreover, similarly to (86), we see that

and hence

and again, as P W : X → X is a finite rank operator and T 2 : W → W is a compact linear operator, we see that ˜ B : X → X is a compact non-linear operator of the form

where ˜ F 1 , ˜ F 2 , ˜ F 3 , ˜ F 4 : X → X are functions in C 1 ( X ) . For an infinite dimensional Hilbert space X there is a linear isomorphism J : X → X × X , as the cardinality of Hilbert basis of the space X is the same as the cardinality of the Hilbert basis of X × X , see [26, Theorem 3.5]. Then, by writing the isomorphism J as J ( x ) = ( J 1 ( x ) , J 2 ( x )) ∈ X × X , we see that

that is a composition of a linear compact operator X → X , a non-linear operator X → X , and a compact operator X → X . Hence we see that Id + ˜ B , and similarly Id + B , are layers of neural operators.

Now we present our Proof of Theorem 4.

Proof. Let F ( x ) = x + T 2 G ( T 1 ( x )) . Without loss of generality, we may assume that 0 < ϵ < min( ∥ T 2 ∥ X → X C 1 ∥ T 1 ∥ X → X , 1 2 ) .

By (94)

and which implies that

is a strongly monotone operator. Similarly, we see that

is a strongly monotone operator.

Recall that F W maps W ⊥ to itself and it is equal to the identity map in W ⊥ , and moreover F W can be decomposed according to the formula (78). Thus we can write

Below, let us identify W with R n to clarify notations. We denote

By (91) and (92) there are c 1 , c 0 > 0 such that

where Df | y : R n → R n is the derivative of the map f at the point y , that is considered as a linear map (or a matrix), and we denote its value at vector v by Df | y ( v ) = Df | y v .

Then f 1 ( x ) = f ( x ) and and as

we have

Below, let and

Define 4 for 0 < t ≤ 1

For t = 0 , we define

where r 1 appears in the claim of Theorem 4. Observe that then f 0 ( B R n (0 , r 1 )) ⊂ B R n (0 , R 1 ) . As f ∈ C 2 ( R n , R n ) , we have by Taylor's series (with the remainder written in the integral form)

Thus,

Observe that for all x ∈ X ,

By (113),

Here, for x, y ∈ B R n (0 , R 1 ) ,

Hence, where

Let us choose t 1 ∈ (0 , 1) such that

Next we consider operators f t with t > 0 .

Let us next consider t 2 , t 3 , . . . , t m +1 ∈ (0 , 1] such that t 2 > t 1 , t m +1 = 1 , and t k +1 > t k . We have

so that

Below, we will denote

and Lip ( f t ) ≤ c 1 . Hence, and

Moreover, for z = f (0) we have f -1 t ( z ) = 0 and | z | = R 0 , and as the Lip ( f -1 t ) ≤ c -1 0 ,

so that

Also, as we can write

we have for t k , t k +1 ∈ (0 , 1] , k ≥ 1 ,

where

To analyze the above, we denote

As ∥ Df | t k + s ( t k +1 -t k ) ( x ) ∥ ≤ c 1 ∥ x ∥ , we have for all R > 0

Moreover, for x 1 , x 2 ∈ B R n (0 , R ) ,

and for all x ∈ R n ,

Moreover,

Let where

For k ≥ 1 we have t k ≥ t 1 , and thus

Hence by (128), for x 1 , x 2 ∈ B R n (0 , R ) ,

By (122) and (129), it hold that of all x ∈ R n

Let R 2 ≥ R 1 and define for k ≥ 1 a function that is a convex combination of the identity map Id and the map f t k +1 ◦ f -1 t k ,

where ϕ ∈ C ∞ 0 ( R n ) is such that ϕ ( x ) = 1 for | x | ≤ R 2 , ϕ ( x ) = 0 for | x | ≤ 2 R 2 , and ∥∇ ϕ ( x ) ∥ ≤ 2 /R 2 . Then

Then, when the condition f t k -1 ( x ) ∈ B R n (0 , R 2 ) it holds, we see that

Observe that when x ∈ B R n (0 , R 0 ) then f t 0 ( x ) ∈ B R n (0 , R 1 ) ⊂ B R n (0 , R 2 ) . Next, we denote t 0 = 0 . Hence, by using induction, we see that for all j = 1 , 2 , . . . , m +1 it holds that

We recall that above we chose t 1 > 0 such that

We now choose m so that there are t k , k = 2 , 3 , . . . , m +1 satisfying | t k -t k -1 | < 1 m and

This means that we can choose

is a continuous map [0 , 1] → C 1 ( R n ; R n ) .

The space W is a finite dimensional Euclidean space, and let GL ( W ) be the set of linear diffeomorphisms of it, that is, invertible linear operators A : W → W . We consider GL ( W ) with the topology given by the operator norm of linear maps. Then GL ( W ) is a topological space with two path connected components - those matrices which have a positive determinant and those having a negative determinant.

The above implies that Df | 0 can be connected in GL ( R n ) either to a matrix B where

with a continuous path β s ∈ GL ( W ) , s ∈ [0 , 1] with Df | 0 = β 1 and B = β 0 .

We need to consider the path β s in GL ( W ) from β 1 = f 0 = Df | 0 to matrix β 0 = B ∈ O ( n ) . There are some explicit formulas in literature with relatively complicated formulas, see [41]. However, let us next consider estimates using a non-optimal but relatively explicit path. To do this we start from the PU-decomposition of the matrix β 1 = Df | 0 , that we denote β 1 = PU where P = diag ( σ 1 , . . . , σ n ) is positive matrix (given in a suitable basis) and U is an orthogonal matrix. Then, we consider the path t → P t U , where t ∈ [0 , 1] and

This is a path from the matrix Df | 0 = PU to the matrix U . Observe that

Then,

We define the map

where

and when | t 1 -t 2 | ≤ 1 , by mean value theorem we have

That is, using the norm of Df | 0 and the norm of inverse matrix ( Df | 0 ) -1 we can bound the length of the path t → P t U , t ∈ [0 , 1] (in the operator norm). After this, we need to consider the path from U to B = Id k × ( -Id n -k ) in O ( W ) . For this, we could use the structure of the Lie group O ( n ) , n = dim( W ) . By [15, Thm. 11.3.3], any matrix U in O ( n ) can be written as a tensor product of elements in O (2) and possibly an operator ± Id : R → R , that is, in a suitable orthogonal basis a matrix U ∈ O ( n ) is a block diagonal matrix of 2 × 2 matrixes in O (2) and one or two 1 × 1 matrices ± Id , that is,

where the matrices R 1 = R 1 ( ϑ 1 ) , ..., R k = R 1 ( ϑ k ) are 2-by-2 rotation matrices in SO (2) ,

Using this, we find a path form U ( s ) , s ∈ [0 , 1] either to a matrix Id k × ( -Id n -k ) ,

Observe that due the block diagonal form of this matrix, the operator norm satisfies

where C ∗ is an absolute constant (that does not depend on the dimension n ). We obtain the path s → β s by concatenating the paths from PU to U in GL ( n ) and from U to B in SO ( n ) . The length of the obtained path β in the operator norm metric can be estimated and it is bounded C (1 + c 1 + c -1 0 ) 2 plus a constant. Moreover, the path s → β s can be decomposed to a product of ( C (1 + c 1 + c -1 0 ) 2 + C ) ε -1 matrices of the form Id + B j where ∥ B j ∥ ≤ ε .

Summarising the above analysis, we see that J can bounded by

We recall that here

radii R 0 and R 1 are given in formulas (110) and (111) and c 0 and c 1 are the bi-Lipschitz constants of f , see (106).

Note that c 0 , c 1 , C > 0 are constants independent of ϵ , while R 0 , R 1 , R 2 > 0 depends on | f (0) | . We see that by f = P W F | W .

thus, the upper bounds of R 0 , R 1 , R 2 are independent of ϵ . By choosing t 1 as t 1 = c 0 c 1 + ∥ f ∥ C 2 ( B R n (0 ,R 1 )) R 1 ε , we furthermore estimate that

Here, if we assume that F ∈ C 2 ( X,X ) , we see that

Thus, we obtain

Now we can finish the proof of Theorem 4. Denote ˜ f t = f t ⊕ Id W ⊥ , ˜ H t 1 ,t 0 = H t 1 ,t 0 ⊕ Id W ⊥ ˜ β s = β s ⊕ Id W ⊥ and A 0 = B ⊕ Id W ⊥ . Using these notations, we can write F ( x ) as

where t j and s j are chosen so that 0 = t 0 < t 1 < . . . t m < t m +1 = 1 and 0 = s 0 < s 1 < . . . s m < s m +1 = 1 and that the Lipschitz constant of the maps ˜ H t j +1 ,t j -Id and ˜ β s j ◦ ˜ β -1 j -1 -Id are less that ϵ .

Moreover, recall that

Observe that by writing x ∈ X in the form x = x 0 + x 1 , where x 0 = ( I -P W ) x and x 1 = P W x , we see that

Hence, ( F W ) -1 : X → X can be written as

and thus ( F W ) -1 is a layer of a neural operator by definition. Similarly, as

and

for 0 < t ≤ 1 , and we see as above that f t and f -1 t are neural operators. Similarly, we see that ˜ β -1 s and ˜ β -1 s are neural operators. Hence, all factors in the product (152) are (strictly monotone) neural operators.

A.7.2 Proof of Theorem 5

Theorem 6 (Theorem 5 in the main text) . Let X be a separable Hilbert space, and let φ = { φ n } n ∈ N be an orthonormal basis in X . Let δ ∈ (0 , 1) , and let R > 0 , and let F : X → X be a layer of a bilipschitz neural operator, as in Def. 3. Then, for any ϵ ∈ (0 , 1) , there are T, N ∈ N and G ∈ R T,N,φ,ReLU ( X ) that has the form

such that

where A : X → X is a linear invertible map that is either the identity map or a reflection operator x → x -2 ⟨ x, e ⟩ X e with some unit vector e ∈ X . Moreover, G ◦ A : B X (0 , R ) → G ◦ A ( B X (0 , R )) is invertible, and there exists R ′ > 0 such that

′

+

δ

)

,

· · ·

Γ

T

:=

B

(0

, R

′

and denoting by

Γ

0

:=

B

X

(0

, R

′

and

we have that for each t = 0 , 1 , ..., T

)

,

Γ

where L t : Γ t → Γ t +1

is defined by

1

:=

B

(0

, R

and the mapping G ◦ A | B X (0 ,R ) : B X (0 , R ) → G ◦ A ( B X (0 , R )) is bijective and its inverse is given by

where Φ : G ◦ A ( B X (0 , R )) → A ( B X (0 , R )) is defined by

where ϕ t : Γ t +1 × Γ t → Γ t +1 × Γ t is defined by

where π 1 ( x, y ) = x and e 1 ( y ) = (0 , y ) .

Remark 2. Note that, in the proof, we have used the approximation result [22, Theorem 4.3] of Sobolev functions by neural networks with ReLU function x ↦→ max { 0 , x } , which is not differentiable at 0 . Alternatively, we can use approximation result [1, Theorem 4.1] by neural networks with ReCU function x ↦→ max { 0 , x } 3 , which are continuously differentiable. Then, each block Id X + D N ◦ NN t ◦ E N in a obtained approximator G is C 1 and strongly monotone on ball Γ t = B (0 , R ′ + tδ ) , that is, it holds that there is an α > 0 such that

which implies that, by the same argument as in Lemma 3,

is a C 1 -diffeomorphism.

Remark 3. From the proof, we can show that, for each t = 1 , ..., T

Then, the mapping Φ defined in (156) is well-defined.

+

Tδ

)

,

Γ

T

+1

:=

B

(0

, R

′

+(

T

+1)

δ

)

,

Proof. Let δ, ϵ ∈ (0 , 1) , and let F : X → X be a layer of a neural operator defined in (4). Assume that F is bilipschitz, that is, it satisfies (85) and

By Theorem 4, there is T ∈ N such that F can represented in the form

where A : X → X is a linear invertible map that is either the identity map or a reflection operator x → x -2 ⟨ x, e ⟩ X e with some unit vector e ∈ X , and H t : X → X are global Lipschitz maps satisfying

for all t . Moreover, the operator H t : X → X is a compact mapping. We choose R ′ > 0 such that

We denote by

We choose ˜ R > 0 such that for all t = 0 , ..., T +1

Since H t : X → X is a compact mapping, for large enough N ∈ N , we have that for all t = 1 , ..., T ,

Note that where ˜ H N,t : R N → R N by

From (158), we have

By approximation by ReLU neural networks in Sobolev spaces (see e.g., [22, Theorem 4.3]) , there is a ReLU neural network NN t : R N → R N , NN t ∈ R 1 ,N,φ,ReLU ( X ) such that

Then, we estimate that by (159) and (160)

Also, we estimate that by (158) and (160)

We denote G : X → X by

which belong to R ( X ) . Then, by (161) and (162), we estimate that for all x ∈ B (0 , R ) ,

T,N,φ,ReLU X ∥ F ( x ) -G ◦ A ( x ) ∥ X

T

∏

=

t

(

+1

T

∏

(1 +

→

R

(0

,

∥

R

)

˜

H

t

(

y

δ

)

1 + Lip

R

B

N

sup

=

1

1

∑

t

T

∑

t

T

h

h

y

∈

K

t

-

1

t

+1

Next, as (160), we see that

and

G | Γ 0 = ( I -D N ◦ NN T ◦ E N ) | Γ T -1 ◦ ( I -D N ◦ NN T -1 ◦ E N ) | Γ T -2 ◦· · ·◦ ( I -D N ◦ NN 1 ◦ E N ) | Γ 0 , which means that G | Γ 0 maps from Γ 0 to Γ T . For t = 1 , ..., T , we define L t : Γ t → Γ t +1 by

where x ∗ ∈ Γ t +1 is a unique fixed point of h t,y : Γ t +1 → Γ t +1 , where

that is, x ∗ ∈ Γ t +1 is a unique solution of

Indeed, there is a unique solution because we have for x 1 , x 2 ∈ Γ t +1 , from (162)

∥ h t,y ( x 1 ) -h t,y ( x 2 ) ∥ X = ∥ D N ◦ NN t ◦ E N ( x 1 ) -D N ◦ NN t ◦ E N ( x 2 ) ∥ X ≤ δ ∥ x 1 -x 2 ∥ X , which implies that h t,y : Γ t +1 → Γ t +1 is a contraction map. Then, we have for x ∈ Γ t -1 , t = 1 , ..., T ,

Here, the solution x ∗ ∈ Γ t +1 is given by

We define φ t : Γ t +1 × Γ t → Γ t +1 × Γ t by

Let π 1 ( x, y ) = x and e 1 ( y ) = (0 , y ) where π 1 : X × X → X and e 1 : X → X × X are linear maps. Then,

We define Φ P : Γ T → Γ 0 by

where P Γ t : X → X is the projection onto the convex set Γ t = B (0 , R ′ + tδ ) . Then, by (163), we have for x ∈ B X (0 , R ′ )

N

)

)

sup

y

t

K

∈

-

NN

t

◦

where

(

NN

-

h

D

N

)

◦

∥

1

E

N

(

I

(

-

y

)

H

t

∥

X

)(

y

)

-

(

I

-

D

N

◦

NN

t

◦

E

N

)(

y

)

∥

X

Therefore, Φ P is the left-inverse of G , that is, G | A ( B X (0 ,R )) : A ( B X (0 , R )) → X is injective, and G | A ( B X (0 ,R )) : A ( B X (0 , R )) → G ◦ A ( B X (0 , R )) is bijective, and the inverse ( G | G ◦ A ( B X (0 ,R )) ) -1 : G ◦ A ( B X (0 , R )) → A ( B X (0 , R )) is given by

where Φ : G ◦ A ( B X (0 , R )) → A ( B X (0 , R )) is defined by Φ := Φ P | G ◦ A ( B X (0 ,R )) and has the form

Therefore, G ◦ A : B X (0 , R ) → G ◦ A ( B X (0 , R )) is also bijective, and the inverse is given by

Note that G ◦ A ( B X (0 , R )) ⊂ Γ T .

A.8 Production of Quantitative Universal Approximation Estimates

Here, we show how our framework may be used 'out of the box' to produce quantitative approximation results for discretization of neural operators.

Let X be a separable Hilbert space. We will consider a Hilbert space X , endowed with its norm topology. Recall that S 0 ( X ) ⊂ S ( X ) is a partially ordered lattice of finite dimensional subspaces of X . Next, we consider quantified discretization of a continuous, possibly non-linear function F : X → X , that is, how the discretizations F V : V → V can be chosen (using e.g. neural networks F V having a given architecture for each subspace V ⊂ X ) so that the obtained discretization operator A F has the explicitly given error bounds ε V .

Using quantitative approximation results for neural networks in R d , see e.g. [57] or [23], one obtains quantitative results for neural operators. An example of such result is given below.

Proposition 5. Let r > 0 and F : B X (0 , r ) → X be a non-linear function satisfying F ∈ Lip ( B X (0 , r ); X ) , in n = 1 , or F ∈ C n ( B X (0 , r ); X ) , if n ≥ 2 . Let ε V > 0 be numbers indexed by the linear subspaces V ⊂ X such that ε V → 0 as V → X . When d = dim ( V ) , the space V is identified with R d using an isometric isomorphism J V : V → R d . Then there is a feed forward neural network F V,θ : R d → R d with ReLU-activation functions with at most C ( n, d ) log 2 ((1+ r ) n /ε V ) layers and C ( n, d ) ε -d/n V log 2 ((1 + r ) n /ε V ) non-zero elements in the weight matrices such that A NN : F → ( F V ) V ∈ S 0 ( X ) , where F V = J -1 V ◦ F V,θ ◦ J V : V → V , is an ⃗ ε -approximation operation in the ball B X (0 , r ) .

Proof. Let Lip ( F : B X (0 , r ) → X ) ≤ M , in n = 1 , or ∥ F ∥ C n ( B X (0 ,r ); X ) ≤ M , if n ≥ 2 . Let V ⊂ X be a linear subspace of dimension d and ε V > 0 . Let us use some orthogonal basis of V to identify V and R d and denote the identifying isomorphism by J : R d → V . The function ˆ F : R d → R d , given by ˆ F = J -1 ◦ P V ◦ F ◦ J satisfies Lip ( ˆ F : B R d (0 , r )) → X ) ≤ M if n = 1 , or ∥ ˆ F ∥ C n ( B R d (0 ,r )) ≤ M if n ≥ 2 . Then, by applying [57, Theorem 1] or [23], we see that there exists a feed forward neural network F V,θ : R d → R d with ReLU-activation functions with at most C ( n, d ) log 2 ((1 + r ) n /ε V ) layers and C ( n, d ) ε -d/n V log 2 ((1 + r ) n /ε V ) non-zero elements in the weight matrices such that

Due to Def. 1, this yields the claim.

B Invertible residual network on separable Hilbert spaces

In this section, we consider the approximation of diffeomorphisms by globally invertible residual networks on Hilbert spaces. From the viewpoint of no-go theorem, as the class of diffeomorphisms is too large, we focus on the class of strongly monotone C 1 -diffeomorphisms with compact support 5 .

̸

Wefirst obverse a following similar result with Theorem 3 that strongly monotone C 1 -diffeomorphism F : X → X with compact support has the property that any linear discretizations P V F | V are still strongly monotone C 1 -diffeomorphism with compact support. The proof can be given by the same argument in Theorem 3 because F has the form of F = Id + B where B := F -Id is a compact mapping.

Proposition 6. Let A lin be the discretization functor that maps F to P V F | V for each finite subspace V ⊂ X . Let D smc and B smc be categories where F : X → X and F V : V → V are strongly monotone C 1 -diffeomorphisms with compact support. Then, the functor A lin : D smc → B smc satisfies assumption (A), and it is continuous in the sense of Definition 8.

Under the same setting and notations in Section 3.5, we define the class of invertible residual networks in the separable Hilbert space by, for T, N ∈ N and δ ∈ (0 , 1) ,

which is a subset of R inv T,N,φ,δ,σ ( X ) defined in (10). The smallness of Lipschitz constants Lip X → X ( D N ◦ NN t ◦ E N ) implies that R inv T,N,φ,δ,σ ( X ) is included in the class of homeomorphisms. The following lemma shows this fact.

Lemma 9. Let δ ∈ (0 , 1) , and let F ∈ R inv T,N,φ,δ,σ ( X ) . If σ : R → R is Lipschitz continuous, then F : X → X is homeomorphism. Moreover, if σ : R → R is C 1 , then, F : X → X is C 1 -diffeomorphism.

Proof. Assume that σ is Lipschitz continuous. Let G ∈ R inv T,N,φ,δ,σ ( X ) , that is,

where Lip X → X ( D N ◦ NN t ◦ E N ) ≤ δ . It is enough to show that for all t = 1 , ..., L

is homeomorphism. Indeed, we have for u, v ∈ X

that is, Id X + D N ◦ NN t ◦ E N : X → X is strongly monotone. By the same argument in Lemma 1, we can show that ( Id X + D N ◦ NN t ◦ E N ) : X → X is coercive. By the Minty-Browder theorem [10, Theorem 9.14-1], ( Id X + D N ◦ NN t ◦ E N ) : X → X is bijective, and then its inverse exists. Denoting by H t := Id X + D N ◦ NN t ◦ E N , we see that by substituting u = H -1 ( u ) and v = H -1 ( v ) for (167)

which means that its inverse is continuous. Therefore, ( Id X + D N ◦ NN t ◦ E N ) : X → X is homeomorphism.

For the second statement, we assume that σ is C 1 . By the same argument in Lemma 1, we can show that the derivative DH t | u : X → X at u ∈ X is given by

and it is injective. Since D N ◦ D ( NN t ) | E N ( u ) ◦ E N : X → X is a finite dimensional linear operator, it is compact operator. Then by the Fredholm theorem, DH t | u : X → X is bijective. By the inverse function theorem, the inverse H -1 t : X → X is C 1 , which implies that H t : X → X is C 1 -diffeomorphism.

In what follow, we employ σ as the GroupSort activation having a group size of 2 (see [3, Section 4]). As sort activation is 1 -Lipschitz, from Lemma 9, R inv L,N,φ,δ,σ ( X ) is a subset of the class of homeomorphisms. We finally show that, in this case, R inv L,N,φ,δ,σ ( X ) is an universal approximator for the class of the strongly monotone diffeomorphisms with compact support.

Theorem 7. Let R > 0 , and let F : X → X be strongly monotone C 1 -diffeomorphism with compact support, and let σ be the GroupSort activation having a group size of 2. Then, for any orthonormal basis { φ n } n ∈ N ⊂ X , δ ∈ (0 , 1) , and ϵ ∈ (0 , 1) , there are T, N ∈ N , and G ∈ R inv T,N,φ,δ,σ ( X ) such that

Moreover, the inverse G -1 : X → X of G ∈ R inv T,N,φ,δ,σ ( X ) is given by

where ˜ L t : X → X is defined by

where ˜ ϕ t : X × X → X × X is defined by

where π 1 ( x, y ) = x and e 1 ( y ) = (0 , y ) .

̸

Proof. Wedenote by Diff 1 ( X ) the class of C 1 -diffeomorphisms between X , and Diff 1 sm ( X ) the class of strongly monotone C 1 -diffeomorphisms between X . We also denote the support of F ∈ Diff 1 ( X ) by supp( F ) := { x ∈ X : F ( x ) = x } . We say that F ∈ Diff 1 sm,c ( X ) if F ∈ Diff 1 sm ( X ) has a compact support.

Let F ∈ Diff 1 c,sm ( X ) . We define F 1 : X → X by F 1 := Id X + P V N ( F -Id X ) P V N , and we see that

We can show that for large N ∈ N

as Id X -F : X → X is a compact mapping. By the same argument in Lemma 1, we can show that F N := E N FD N ∈ Diff 1 sm,c ( R N ) , and DF N | 0 is a positive definite matrix, which is connected to Id R N . By the similar argument in the proof of Theorem 4, see (105)-(152), we can construct continuous paths f : [0 , 1] → Diff 1 sm,c ( R N ) and β : [0 , 1] → GL ( R N ) with f 0 = DF N | 0 , f 1 = F N , β 0 = Id R N , and β 1 = DF N | 0 such that

where t j and s j are chosen so that 0 = t 0 < t 1 < . . . t m < t m +1 = 1 and 0 = s 0 < s 1 < . . . s m < s m +1 = 1 and that the Lipschitz constant of the maps f t j ◦ f -1 t j -1 -Id R N and β s j ◦ β -1 j -1 -Id R N are less that δ .

Then, there is T ∈ N such that F N has the form

where for each t = 1 , ..., T ,

Then, remarking that E N D N = Id R N and D N E N = P V N , we see that

For each t = 1 , ..., T , by using [3, Theorem 3 and Observation], δ -Lipschitz function H t : R N → R N can be approximated by a neural network NN t : R N → R N with GroupSort activation σ having a group size of 2 in L ∞ -norm on any compact set, and NN t : R N → R N is δ -Lipschitz continuous.

We denoting by

Note that Lip X → X ( D N ◦ NN ℓ ◦ E N ) ≤ δ . Then, we can show that, by similar way in the first half of proof of Theorem 5,

With (169) and (171), we obtain that

The representation of inverse G -1 can be given by the same argument in the proof of Theorem 5.

Similarly in Corollary 1, Theorem 7 and Lemmas 10 and 11 have the following corollary.

Corollary 2. Under the same setting and assumptions with Corollary 1, let RNO inv T,N,φ,δ,σ ( L 2 ( D ; R )) be the class of invertible residual neural operators defined in (180). Then, the statement replacing X with L 2 ( D ; R ) and G ∈ R inv T,N,φ,δ,σ ( X ) with G ∈ RNO inv T,N,φ,δ,σ ( L 2 ( D ; R )) in Theorem 7 holds.

C Neural operators

C.1 Examples of generalized neural operators

In the main text, we defined generalized neural operators on Hilbert spaces. Here, we give several examples to show that they are extensions of classical neural operators [33, 30].

Example 1. Let X be a Hilbert space. Let G ℓ : X → X be a classical neural operator having the form

where W ℓ : X → X are linear bounded operators corresponding to the local term and K ℓ : X → X are compact operators corresponding to the non-local term (e.g., integral operators having a smooth kernel or smooth basis). We assume that activation function σ is C 1 . Then, we have G ℓ ∈ C 1 ( X ; X ) . Let T 1 = K lift : X → X and T 2 = K proj : X → X be compact linear operators, which corresponds to lifting and projection, respectively. We denoting by

which is one block of classical neural operators with skip-connection. We also denote by A ℓ = Id and σ = Id . Then, the generalized neural operator H = F L ◦ · · · ◦ F 1 corresponds to classical neural operators with skip-connections. In this paper, the skip-connection (ie., the structure of the identity plus some compact mapping) is so important to preserve bijectivity, discussed in Section 3.

Example 2. We show that classical neural operators , for example

see [30, 33], can be written in the form of the generalized neural operators that we consider of the form

where A j : X → X are linear operators which may not be bijective and F j = Id + T j, 1 ◦ G ◦ T j, 2 : X → X .

We could start with the observations that for an infinite dimensional Hilbert space X there is a linear isomorphism J m,n : X n → X m , where X m = X ×··· × X . The reason for this is that the cardinality of Hilbert basis of the space X is the same as the cardinality of the Hilbert basis of X n , see [26, Theorem 3.5].

First we observe that we can write an operator

where ˜ A j : X n j → X n j +1 are linear operators which may not be bijective and ˜ F j = Id + ˜ T j, 1 ◦ ˜ G ◦ ˜ T j, 2 : X n j → X n j and ˜ σ k : X n k → X n k is ˜ σ ( x 1 , x 2 , . . . , x i k , x i k +1 , . . . , x n k ) = ( x 1 , x 2 , . . . , x i k , σ ( x i k +1 ) , . . . , σ ( x n k )) in the form

where e.g. ( J 1 ,n k +1 ◦ A k ◦ J n k , 1 ) : X → X and ( J 1 ,n k ◦ ˜ σ k ◦ ˜ F k ◦ J n k , 1 ) : X → X . This means that in our formalism we can replace e.g. operators A k by matrices of operators A k .

Next we go to the second step of the construction:

As an example, let us consider the classical neural operator

where W 1 and W 2 are invertible matrices and K 1 and K 2 are compact operators, can be written as an generalized neural operator

where F 0 : X → X is a layer of neural operator

and L 1 : X → X is the invertible linear operator

and ˜ σ ( u ) = σ ( u ) and L 2 : X → X × X is a non-invertible linear operator

and and F 3 : X × X → X × X is a layer neural operator

and L 4 : X × X → X is the non-invertible linear operator

Finally, we point out that if a non-invertible activation function σ : X → X satisfies Lip ( σ ) ≤ λ , it can be written as

Note that equation 2 λu + σ ( u ) = w can be written as a fixed point equation

where Lip ( g w ) ≤ 1 / 2 . Hence, g w : X → X is a contraction and the equation u = g w ( u ) has a unique solution for all w by Banach fixed point theorem. Hence, u → 2 λu + σ ( u ) is invertible.

Example 3. Let D ⊂ R d be a domain, and let L 2 ( D ; R ) be the real-valued L 2 -function space on D , and let φ = { φ n } n ∈ N ⊂ L 2 ( D ; R ) be an orthonormal basis in L 2 ( D ; R ) . We consider the case when X = L 2 ( D ; R h ) = L 2 ( D ; R ) h , and A j = W ( j ) where W ( j ) ∈ R h × h are invertible matrices, and F j = Id + P V h N ◦ K ( j ) N ◦ P V h N W ( j ) -1 ) (corresponding to T 1 = P V h N , T 2 = P V h N W ( j ) -1 , G = K ( j ) N )

where and ˆ σ is an invertible function

and

where V N := span { φ n } n ≤ N , and K ( j ) N : L 2 ( D ; R ) h → L 2 ( D ; R ) h is a finite rank operator defined by

Then, H : L 2 ( D ; R ) h → L 2 ( D ; R ) h can be written by

which coincides with classical neural operators (assuming that all local weight matrices are invertible) used in e.g., [33]. See Definition 9 as well.

Example 4. Let us consider an example of an example of an generalized neural operator which is obtained by composition of non-linear integral operators and smooth activation functions. Let D ⊂ R d be be a bounded domain with a smooth boundary. We consider the case when X = H 1 ( D ; R h ) = H 1 ( D ; R ) h are Sobolev spaces, and A j = W ( j ) where W ( j ) ∈ R h × h are invertible matrices, and F j = Id H 1 + i H 2 → H 1 ◦ ˜ K ( j ) ◦ i H 1 → L 2 W ( j ) -1 (corresponding to T 1 = i H 1 → L 2 , T 2 = i H 2 → H 1 ( W ( j ) ) -1 , G = ˜ K ( j ) ) where ˜ K ( j ) : L 2 ( D ; R ) h → H 1 ( D ; R ) h is non-linear integral operator

where kernel satisfies k ( j ) ∈ C 3 ( D × D × R h ; R h × h ) has uniformly bounded three derivatives. Also, let σ ∈ C 1 ( R ) be an activation function which derivative is uniformly bounded and we denote σ ∗ f = σ ◦ f . Then, H : H 1 ( D ; R ) h → H 1 ( D ; R ) h , defined by

is an generalized neural operator. Here, i H 1 → L 2 and i H 2 → H 1 are compact emending from H 1 ( D ) to L 2 ( D ) and H 2 ( D ) to H 1 ( D ) , respectively. Non-linear integral operator in neural operator has been used in [30].

Example 5. Let us consider an example of a typical neural operator which is a finite composition of layers of neural operators similar to those introduced in [30, 33], F : X → X of the form

where X = H m (Ω) , Ω ⊂ R d is a bounded set with a smooth boundary and W : X → X is a linear operator. In the above, Y = C (Ω) and Z = C m +1 (Ω) , where m>d/ 2 . Moreover, G : Y → Z is a nonlinear integral operator

where k θ , ∂ u k θ ∈ C m +1 (Ω × Ω × R ) is a kernel given by a feed-forward neural network with sufficiently smooth activation functions. Here, we assume that σ j ∈ C ℓ ( R ) , ℓ ≥ m +2 and that the kernel k θ is of the form

Moreover, T 1 : X → Y and T 2 : Z → X are the identical embedding operators

Due to the choice of function spaces X,Y and Z , the maps T 1 and T 2 are compact operators. Summarizing, F = F σ,W,k , where

Furthermore, by choosing k θ ( x, y, u ( y )) = k θ ( x -y ) and Ω = T d as the convolutional kernel and the torus, the map F takes the form of an FNO [39].

Here the activation functions σ and σ j are in different roles as the functions σ j appear inside the integral operator (or between the compact operators T 1 and T 2 ). The function σ is useful in obtaining universal approximation results for neural operators, but as this question is somewhat technical, we postpone this discussion to the end.

The appearance of the compact operators T 1 and T 2 makes the discretization of activation function σ and the activation functions inside G in Definition 4 different, and this is the reason why we have introduced both σ and G . To consider invertible neural operators, we will below assume that σ is an invertible function, for example, the leaky Relu function. In the above operator, the nonlinear function G inside compact operators in the operation

and the compact operators map weakly converging sequences to norm converging sequences. This is essential in the proofs of the positive results for approximation functors as discussed in the paper. However, we do not have general results on how the operation

can be approximated by finite dimensional operators in the norm topology, but only in the weak topology in the sense of Definition 11 of the Weak Approximation Functor. However, one can overcome this difficulty in two ways. The first way is to use in the discretization a suitable finite dimensional space V that satisfies G σ ( V ) ⊂ V . For example, when X = L 2 (Ω) and σ is a leaky relu function, one can choose V to be a space that consists of piecewise constant functions. The second way is choosing a composition of layers of the form

in different finite dimensional spaces V j . For example, we can consider these operations as maps

well defined. The maps, G σ , are clearly invertible and one can use Theorems 4 and 5 to analyze when N j : V j → V j are invertible functions.

Finally, we return to the question whether the activation function σ is useful in universal approximation results. If the activation function σ is removed (that is, it is the identical map σ id : s → s ), the operator F is a sum of a linear operator and a compact nonlinear integral operator,

Moreover, if we compose above operators F j of the above form, the obtained operator, G : X → X is also a sum of a linear operator and a compact operator,

Moreover, the Frechet derivative of G at u 0 , denoted DG | u 0 is the map

and, due to the above assumptions on kernel k θ ( x, y, u ) , the derivative is a linear operator

and

This makes the composition

By the Sobolev embedding theorem it is a compact operator D ˜ K | u 0 : X → X . This means that the Fredholm index of the derivative of DG | u 0 is constant

that is, independent of the point u 0 where the derivative is computed. In particular, this means that one cannot approximate an arbitrary C 1 -function G : X → X in compact subsets of X by neural operators which are compositions of layers (178) . Indeed, for a general C 1 -function G : X → X the Fredholm index may be a varying function of u 0 . Thus, σ appears to be relevant for obtaining universal approximation theorems for neural operators.

C.2 Residual neural operators

Let D ⊂ R d be a domain. In what follows, we consider the real-valued L 2 -function space L 2 ( D ; R ) 6 . Let φ = { φ n } n ∈ N ⊂ L 2 ( D ; R ) be an orthonormal basis in L 2 ( D ; R ) .

Definition 9 (Neural operators [33]) . We define a neural operator G : L 2 ( D ; R ) → L 2 ( D ; R ) by

where u L +1 is give by the following steps:

where σ : R → R is a non-linear activation operating element-wise, and W ( ℓ ) ∈ R d ℓ +1 × d ℓ and b ( ℓ ) ∈ R d ℓ +1 and

where K ( ℓ ) p,q ∈ R d ℓ +1 × d ℓ ( ℓ = 0 , ..., L , p, q = 1 , ..., N , d 0 = d L +2 = 1 ). Here, we use the notation for v = ( v 1 , ..., v d ℓ ) ∈ L 2 ( D ; R ) d ℓ

We denote by NO L,N,φ,σ ( L 2 ( D ; R )) the class of neural operators G : L 2 ( D ; R ) → L 2 ( D ; R ) defined above, with depths L , rank N , orthonormal basis φ , and activation function σ .

Definition 10 (Residual Neural Operator) . Let T, N ∈ N and let δ ∈ (0 , 1) . We define by

Lemma 10. Let δ ∈ (0 , 1) , and let F ∈ RNO inv T,N,φ,δ,σ ( L 2 ( D ; R )) . Let σ : R → R be Lipschitz continuous. If σ : R → R is Lipschitz continuous, then F : X → X is homeomorphism. Moreover, if σ : R → R is C 1 , then, F : X → X is C 1 -diffeomorphism.

The proof is given by the same argument in Lemma 9.

Lemma 11. Assume that the orthonormal basis φ include the constant function. Then, we have the following inclusion:

Proof. Let G ∈ R T,N,φ,σ ( L 2 ( D ; R )) such that

Since φ = { φ n } n ∈ N ⊂ L 2 ( D ; R ) has the constant basis, denoting it by φ 0 := 1 D ( x ) ∥ 1 D ∥ L 2 ( D ; R ) , where 1 D ( x ) = 1 for x ∈ D , we see that for α = ( α 1 , ..., α N ) ∈ R N

We define ˜ D N : L 2 ( D ; R ) N → L 2 ( D ; R ) by for u ∈ L 2 ( D ; R ) N

where ˜ D p,q ∈ R 1 × N is defined by

Then, we have that

Next, we see that

E N ( u ) = ( ⟨ u, φ 1 ⟩ L 2 ( D ; R ) , ..., ⟨ u, φ N ⟩ L 2 ( D ; R ) ) = ∥ 1 ∥ L 2 ( D ; R ) ( ⟨ u, φ 1 ⟩ L 2 ( D ; R ) , ..., ⟨ u, φ N ⟩ L 2 ( D ; R ) ) φ 1 ( x ) ,

We define ˜ E N : L 2 ( D ; R ) → L 2 ( D ; R ) N by

where ˜ E p,q ∈ R N × 1 is defined by

Then we have that

With (183) and (184), we see that

where L t ∈ N is the depth of NN t . Therefore, G has the form

(182) can be proved by the same argument.

D Generalizations

D.1 Generalization of the no-go Theorem 2 using weak topology

We can generalize the no-go Theorem 2 for the case when approximations and continuity of approximations are considered in the weak topology of the Hilbert space X . This can be done when the approximations of the identity map and the minus one times the identity operator satisfy additional assumptions and the partially ordered subset S 0 ( X ) is the set of all all linear subspaces S ( X ) of X .

In the case when S 0 ( X ) = S ( X ) and Definition 7 the condition (A) can generalized as follows:

̸

̸

Definition 11 (Weak Approximation Functor) . When S 0 ( X ) = S ( X ) we define the weak approximation functor , that we denote by A : D → B , as the functor that maps each ( X,F ) ∈ O D to some ( X,S ( X ) , ( F V ) V ∈ S ( X ) ) ∈ O B so that the Hilbert space X stays the same. The approximation functor maps all morphisms a ϕ to A ϕ and morphisms a X 1 ,X 2 to A X 1 ,X 2 , and has the following properties

(A') For all r > 0 , all ( X,F ) ∈ O D and all y ∈ X , it holds that

Moreover, when F : X → X is the operator Id : X → X or -Id : X → X , then F V is the operator Id V : V → V or -Id V : V → V , respectively.

Moreover, Definition 8 can generalized as follows so that it uses the weak topology.

Definition 12. We say that the approximation functor A is continuous in the weak topology if the following holds: Let ( X,F ) , ( X,F ( j ) ) ∈ O D be such that the Hilbert space X is the same for all these objects and let ( X,S ( X ) , ( F V ) V ∈ S ( X ) ) = A ( X,F ) be approximating sequences of ( X,F ) and ( X,S ( X ) , ( F j,V ) V ∈ S ( X ) ) = A ( X,F ( j ) ) be approximating sequences of ( X,F ( j ) ) . Moreover, assume that for r > 0 and all y ∈ X

Then, for all V ∈ S ( X ) the approximations F ( j ) V of F ( j ) and F V of F satisfy for all y ∈ X

The theorem below states a negative result, namely that there does not exist continuous approximating functors for diffeomorphisms.

Theorem 8. (No-go theorem for discretization of general diffeomorphisms) There exists no functor D → B that satisfies the property (A') of a weak approximation functor and is continuous in the weak topology.

The proof of Theorem 8 is analogous to Theorem 2 by replacing A 1 by the map -Id and considering a linear space V ∈ S ( X ) having an odd dimension, in which case deg( A 1 : V → V ) = -1 . Let F 0 = Id : X → X and F 1 = -Id : X → X . Assume that ( X,S ( X ) , ( F t,V ) V ∈ S ( X ) ) = A ( X,F t ) are approximations of the map F : X → X in the weak topology and that A is continuous in the weak topology. Recall that then F t,V : V → V are C 1 -diffeomorphisms that are discretizations of F : X → X . Then, by condition (A'), F 0 ,V = Id V and F 1 ,V = -Id V so that deg( F 0 ,V ) = 1 and deg( F 1 ,V ) = -1 . Observing that when the condition (187) is applied for y 1 , y 2 , . . . , y n ∈ V that form a basis of the space V having the dimension dim( V ) = n , we see that the condition (187) implies the condition (7). Using these observations, Theorem 8 is follows similarly to the proof of Theorem 2.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction accurately summarize and reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [NA]

Justification: Our paper is theoretical. It does not have limitations in the same was as a more applied paper. All theorems and propositions are proved to be true, and so are not limited in their scope.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The statements of all theoretical statements are given in the main body, and all of their proofs are included in the attached appendix. A reference to the proof of each statement is given either immediately before, or immediately after each statement. When appropriate, a proof sketch is given.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: Our paper does not have any numerical experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: Our paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: Our paper does not include experiments

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Our paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Our paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow NeurIPS code of ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is fundamental. Our work is not tied to a particular application. To our knowledge, there are no direct paths to any negative applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work is theoretical, does not release data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Our work is theoretical and does not contain owners of outside assets, like code, data or models.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the question of whether a neural operator (or a general diffeomorphism on an infinite dimensional Hilbert space) can be continuously discretized through the lens of category theory. It first proves that there does not exist a continuous approximation scheme for all diffeomorphisms. Then, it shows that neural operators with strongly monotone layers can be continuously discretized, followed by a proof that a biliptschitz neural operator can be approximated by a deep one with strongly monotone layers. Some further consequences are discussed.\n\nSTRENGTHS:\n1. The paper studies the discretization of a continuous operator, which is a source of error and an important issue in operator learning if one is not careful.\n2. The paper is fairly comprehensive, encompassing both positive and negative results. The study of positive results contains theorems of different flavors.\n3. Although the theory is based on the category theory, the presentation and explanation of the results are relatively clear and accessible for people who are unfamiliar with it.\n\nWEAKNESSES:\n1. Most results presented in the paper are purely theoretical and lack a quantitative or asymptotic estimate. For example,\n   * the notion of continuous approximation functor in Definition 8 does not care about the rate of convergence, and\n   * there is no estimate for the number of layers $J$ in Theorem 4.\n2. No empirical results are there to support the theory. While this is a theoretical paper, some toy experiments that exemplify the theory would be very helpful.\n\nQUESTIONS:\n1. In general, is there any assumption of the Hilbert space studied in this paper? For example, are the Hilbert spaces assumed to be separable?\n2. What is the definition of the convergence of finite-dimensional subspaces used in this paper? Strong convergence of the projection operator? I do not think there is a standard definition for this in elementary functional analysis so it would be helpful to say it explicitly in the paper.\n3. Have you studied the role of the bilipschitz constant in your theorems? For example, how does the number of layers $J$ in Theorem 4 depend on it?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-17"}

comments: SUMMARY:
This paper focuses on the continuous discretization in operator learning. This is a very important question since it involves reducing the infinite-dimensional space to a finite-dimensional space in operator learning. The authors present cases where discretization is continuous and cases where it is not. The results are interesting and can be applied to design methods in operator learning.

STRENGTHS:
The proof is solid, and the paper is well-written and organized. I appreciate the results presented in this paper.

WEAKNESSES:
Since this paper is submitted to NeurIPS and not a mathematical journal, I hope the authors can provide some practical examples, such as solving the Poisson equation \(\Delta u = f\) to learn the operator relationship between \(f\) and \(u\). By using methods like DeepONet and FNO, it would be beneficial to determine whether the discretization in these methods is continuous or not. I believe this could make the paper more accessible to a broader audience.

QUESTIONS:
Mentioned in the Weakness.

LIMITATIONS:
All right.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates theoretical limitations of discretizing neural operators on infinite-dimensional Hilbert spaces. The authors first prove a \"no-go theorem\" (Theorems 1,2) showing that diffeomorphisms between infinite-dimensional Hilbert spaces cannot generally be continuously approximated by finite-dimensional diffeomorphisms. Then, they provide positive results for certain classes of operators such as strongly monotone (Theorem 3) and bilipschitz neural operators (Theorem 4). They finally provide concrete example of approximation by finite residual ReLU networks (Theorem 5).\n\nSTRENGTHS:\n- The universality of neural networks has been demonstrated in various settings. However, research on the approximation abilities of operators is relatively scarce. Particularly, the characterization of classes that cannot be approximated is intriguing. This study is important as it succinctly demonstrates the differences between finite-dimensional and infinite-dimensional properties in the manageable setting of Hilbert spaces.\n\n- Moreover, the novel approach of expressing approximation sequences in terms of category theory is noteworthy.\n\nWEAKNESSES:\n- On the other hand, the proofs are based on conventional analytical arguments rather than category-theoretic arguments. Therefore, the \"category theory\" framework might be somewhat exaggerated. It is expected that with refinement of notation and sentence structure, the description could become more perspicuous in the future.\n\n- There is concern that the categorical description may have obscured the contributions typically seen in __traditional approximation theory__ papers. As the authors likely recognize, various topologies are used in function approximation, and this study focuses __only__ on approximation in the norm topology of Hilbert spaces, and does not negate \"all considerable approximation sequences\". So, the impossibility theorem presented here might simply be due to the norm topology being too strong. While the Hilbert structure sounds natural as a generalization of Euclidean structure, in reality, concepts like L2 convergence of Fourier series are quite technical and not necessarily an inevitable notion of convergence. It seems that in pursuit of an elegant categorical description, the diversity of function approximation may have been compromised.\n\nQUESTIONS:\nIn Definition 3, why $\\sigma$ is imposed besides $G$?\n\nLIMITATIONS:\nThe authors did not discuss the validity of assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of discretizing neural operators, maps between infinite dimensional Hilbert spaces that are trained on finite-dimensional discretizations.  Using tools from category theory, the authors provide a no-go theorem showing that diffeomorfisms between Hilbert spaces may not admit continuous approximations by diffeomorfisms on finite spaces. This highlights the fundamental differences between infinite-dimensional Hilbert spaces and finite-dimensional vector spaces. Despite these challenges, the authors provide positive results, showing that strongly monotone diffeomorphism operators can be approximated in finite dimensions and that bilipschitz neural operators can be decomposed into strongly monotone operators and invertible linear maps. Finally, they observe how such operators can be locally inverted through an iteration scheme.\n\nSTRENGTHS:\n- The paper provides  theoretical results addressing the challenging problem of discretizing inherently infinite-dimensional objects (neural operators)\n\nWEAKNESSES:\n- The text and presentation require significant polishing.  It contains numerous typos, poorly formulated sentences, and instances of missing or repeated words\n- While the paper's theoretical focus is valuable, it lacks  examples of specific neural operator structures that meet the theorems or remarks \n- A more detailed  discussion on the  practical impact of this work, accompanied by examples, would be beneficial for the audience\n\nPlease note that my review should be taken with  caution, as I am not  familiar with category theory and did not thoroughly check the mathematical details. My feedback primarily focuses on the presentation and potential impact of the results rather than a rigorous validation of the theoretical content.\n\nQUESTIONS:\n- Neural operators are typically defined between Banach spaces. Why does your theory focus on maps between Hilbert spaces instead?\n- Comment: The work in [1] might have been relevant to cite as well.\n- The main neural operator paper [2] develops theoretical results on the universal approximation theory of neural operators. How do your results relate to the ones in that paper?\n\n[1] F. Bartolucci, E. de Bézenac, B. Raonić, R. Molinaro, S. Mishra, R. Alaifari, Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning, NeurIPS 2023.\n\n[2] Nikola B. Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. \"Neural operator: Learning maps between function spaces with applications to PDEs,\" J. Mach. Learn. Res., 24(89):1–97, 2023.\n\nLIMITATIONS:\nThe paper lacks  examples of applications of the theorems to specific neural operator structures, and some further discussion on the practical impact of the results with examples", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}