Can neural operators always be continuously discretized?
Authors: Takashi Furuya
Michael Anthony Puthawala
Matti Lassas
Maarten V. de Hoop
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Operators
Invertibility
Category Theory
Primary area: deep_learning_architectures
TL;DR: Discretizing injective neural operators may cause them to lose injectivity, even with very flexible discretization methods.

Abstract: In this work we consider the problem of discretization of neural operators in a general setting. Using category theory, we give a no-go theorem that shows that diffeomorphisms between Hilbert spaces may not admit any continuous approximations by diffeomorphisms on finite spaces, even if the discretization is non-linear. This shows how infinite-dimensional Hilbert spaces and finite-dimensional vector spaces fundamentally differ. A key take-away is that to obtain discretization invariance, considerable effort is needed to ensure that finite-dimensional approximations of neural operator converge not only as sequences of functions, but that their representations converge in a suitable sense as well. With this perspective, we give several positive results. We first show that strongly monotone diffeomorphism operators always admit finite-dimensional strongly monotone diffeomorphisms. Next we show that bilipschitz neural operators may always be written via the repeated alternating composition of strongly monotone neural operators and invertible linear maps. We also show that such operators may be inverted locally via iteration provided that such inverse exists. Finally, we conclude by showing how our framework may be used `out of the box' to prove quantitative approximation results for discretization of neural operators.

Can neural operators always be continuously discretized?
Takashi Furuya
Michael Puthawala
Maarten V. de Hoop"
Matti Lassas
Shimane University, takashi. www.uyaa0110ggaaa..cooom
South Dakota State University, Michael Puthawala@ddsttte. edu Riice University, mdehooprrice. edu
4 University of Helsinki, matti wwwsssaeessslsssiii.i........... .i
* These authors contrihuted equally to this work
Abstract
We consider the problem of discretization neural operators between Hilbert spaces in a general framework including skip connections. We focus on bijec- tive neural operators through the lens of diffeomorphisms Framed using category theory, give no-go theorem that shows that diffeomor- phisms between Hilbert spaces or Hilbert manifolds may not admit any continuous approximations by diffeomorphisms proximations are nonlinear. The natural way out the introduction of strongly monotone diffeomorphisms and layerwise strongly monotone neural operators which have continuous approximations by strongly monotone diffeomorphisms siite-dimensional spaces. For these, one can guarantee discretization invari- ance, while ensuring that finite-dimensional approximations converge not only as sequences of functions, but that their representations converge suitable sense as well. Finally, we show that bilipschitz neural operators may always be written in the form of an alternating composition of strongly monotone neural operators, plus a simple isometry. Thus we realize rigorous platform for discretization of a generalization of a neural operator. We also show that neural operators of this type may be approximated through the composition of finite-rank residual neural operators, where each block strongly monotone, and may be inverted locally via iteration. We conclude by providing quantitative approximation result for the discretization general bilipschitz neural operators.
infinite dimensions.
ditee-dimensioall spaces, even if the ap-
Introduction
Neural operators, first introduced in 330 have become more and more prominent in deep learning on function spaces. As opposed to traditional neural networks that learn maps between mimee-dimensional Euclidean spaces, neural operators learn maps between infiite-dimensionn function spaces yet may be trained and evaluated on finite-dimensional data through rigorous notion of discretization. Neural operators are widely used in the field of scientific machine learning [6] [1 38 (55] [56], among others, principally because of their discretization invariance. In this work, consider the fundamental limits of this discretization. Throughout, we emphasize the importance of continuity under discretization.
A key ingredient in analysis is the identification of properties of diffcomorphisms that may be induced by (bijective) neural operators, which are diffcomorphisms themselves. Diffcomorphisms exist in many contexts, for example, in generative models. These involve mapping one probability distribution or measure, μ., over some measurable space to another, X, via push forward, that is, (U) [(( (U)) for U X. If μ admits density du then, in finite dimensions, may
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

use the change of variables formula to write dp(x) = du(F (=((=(())) where JF is the Jacobian of F. Clearly, F must be bijection with full-rank Jacobian. In other words, F must be a diffeomorphism onto its range. This established diffeomorphisms as natural objects of interest in finite-dimensional machine learning, and helps account for their wide use [20 25 31 44 54], In this work, we consider the extension of these efforts from finite to infinite dimensions implemented via neural operators. Although there is no analogue of the change variables formula in infinite dimensions, we argue that it is, nonetheless, natural to consider the role of diffeomorphisms, and how they may be approximated via diffeomorphisms on finite-dimensional spaces.
The question of when operations between Hilbert spaces may be discretized continuously may be understood through an analogy to computer vision. Consider the task of learning map from one image space to another, for example, a style transfer problem [16], where the mapping learned does not depend much on the resolution of the images provided. It is natural to think of the map as being defined between (infinite-resolutionn) continuum images, and then its application to images of a specific resolution. this analogy, X is function space (over images m [0. 1]2 + R (29) that approximated with a finite- dimensional space E and the transformation approximated by map f : R + R", where each f acts on images of particular resolution. An explicit transformation formula can be obtained when f is diffeomorphism and has a smooth inverse.
X is
We introduce framework based generalized notion of neural operator layers including skip connection and their restrictions to balls rather than compact sets. With bijective neural operators in mind, we give perspective based on diffeomorphisms in infinite dimensions between Hilbert manifolds. We give a no-go theorem, framed with category theory, that shows that diffeommrphissms between Hilbert spaces may not admit any continuous approximations by diffeomorphisms on finite- dimensional spaces, even if the underlying discretization is nonlinear. In this framing the discretization operation is modeled as a functor from the category of Hilbert spaces and C' d-diffommorpiisss on them to their umite-dimensione described by the introduction of strongly monotone diffeomorphisms and layerwise strongly monotone neural operators. We prove that all strongly monotone neural operator layers admit continuous approximations by strongly monotone diffeomorphsisms on finite-dimensional spaces. We then provide various conditions under which a neural operator layer is strongly monotone. Notably, a bilipschitz (and, hence, bijective) neural operator layer can always be represented by a composition of strongly monotone neural operator layers. Hence, such operator may be continuously discretized. More constructively, any bilipschitz neural operator layer can be approximated by residual finite-rank neural operators, each of which are strongly monotone, plus a simple isometry. Moreover, these finite-rank residual neural operators are (locally) bijective and invertible, and their inverses are limits of compositions of finite-rank neural operators. Our framework may be used "out of the box" to prove quantitative approximation results for discretization of neural operators.
approximations. A natural way to mitigate the no-go theorem
1.1 Related work
Neural operators were first introduced in 33 Alternative designs for mappings between function spaces are the DeepONet [3 40). and the PCA-Net [2]. In spite of the multitudinous applications of neural operators, the theory of the natural class of injective or bijective neural operators is comparatively underdeveloped; see, for example
Our work is concerned with the of discretization of neural operators through the lens of diffeomor- phisms. For recent important work on analyzing the effect of discretization error of Fourier Neural Operators (FNOs) arising from aliasing, see 35 Our work has connections to infinite-dimensionall inference, see c.g. [19], and approximation theory, see e.g. 13 while bridging the gap with the theory of neural operators.
We give a no-go theorem that uses category theory framing. This contributes to the use of category theory as an emerging tool in the analysis and understanding of neural networks at large. In this sense, we are in league with the recent work generalizing ideas from geometric machine learning using category theory [7].
Discretization obstructions have been encountered in other contexts. Numerical methods that ap- proximate continuous models are known to sometimes fail in surprising ways. basic example of this is the "locking" phenomenon in the study of the Finite Elements Method (FEM). For example,
2

---

linear elements used to model bending of a curved surface or beam lock in such way that the model exhibits a non-physical stiff response to deformations 4 Understanding this has been instrumental in developing improved numerical methods, such high order FEM [52]. Furthermore, in discretized statistical inverse problems [2 36 51, the introduction of Besov priors 48 has been found to
be essential.
Finally, our work extends prior work (not based on deep learning) discretization of physical or partial differential equations based forward models in inverse problems. The analogous notion of discretization invariance of solution algorithms inverse problems was studied in 33 48 51 and the lack of it (in imaging methods using Bayesian inversion with L priors) in [36] 48) By considering
the neural operator as the physical model, our results state that discretization can be done locally in an appropriate way, together with constructing an inverse.
1.2 Our contributions
The key results in this paper comprise the following:
1. We prove general no-go theorem showing that, under general cireumstances, diffeo- morphisms between Hilbert spaces may not admit continuous approximation by finite- dimensional diffeomorphisms. In particular, neural operators corresponding to diffeomor- phic maps, in general, cannot be approximated by finiteddimensional diffeomorphisms and their associated neural representations.
2. We show that strongly monotone neural operator layers admit continuous approximations by strongly monotone diffeoomrphsisms on finiteddimensional spaces.
3. We show that bilipschitz neural operators can be represented in any bounded set composition of strongly monotone, diffeomorphic neural operator layers, plus a simple isometry. These be approximated by finite-rank diffeomorphic neural operators, where each layer is strongly monotone. For these operators we give a quantitative approximation result.
Definitions and notation
In this section, we give the definitions and notation used throughout the paper. First, we summarize the relevant basic concepts from functional analysis. Then, we introduce generalized neural operators.
2.1 Elements of functional analysis
In this work, all Hilbert spaces, X, are endowed with their norm topology. We denote by Bx (r(r)= = (0, r) the ball in the space X having the center at zero and radius 0. We denote by S(X) ) the set of all fiite-dimensional linear subspaces V C X. The set So(V) C S(X) is a partially ordered lattice. That is, if Vi, V2 (X) then there is V/ E So (X) so that Vi V and V2 V3 l With Y standing for another Hilbert space, we denote by C"(X;; Y) the set of operators, F: +Y having continuous (Fréchet) derivatives, and C" ("(X) = C" XX:
Next, we define what it means that a nonlinear operator or function F: X + on an infinite- dimensional Hilbert space, X, is approximated by operators or functions on finteedimenniooal subspaces V C X. The key is that as V tends to X, the complexity Fy increases and one may hope that the approximation becomes better. We formalize this in the following definition.
Definition 1 (ev approximators and weak approximators). (i) Let r 0, F C ("(X;; X) be family of functions, and = (ev)ves,(X) be a sequence such that ey - V - X. We say that function
Ax: F x--esss)) C(Br(Orr:VV), F F
'Each element of So(X) will come to represent discretization of X. The partially ordered lattice condition will come to represent notion of common refinement of discretization. This makes possible consider "realistic" choices of discretizations. The condition automatically follows for any discretization scheme that has a notion of "common refinement" of two discretizations. Examples include finite diffeeeeefeeeecee schemes, and the finite elements Galerkin discretization that based triangulation of the domain.

---

is an E-approximation operation for functions F in the ball Bx (0. r) taking values in families Fy CC (V:V) Ax maps a function F: - X, where F F. to a sequence of functions (Fr)vES((X) where Fv E Fv, such that the following is valid: For all F: X - X satisfying [1 S M, we have
sup
-V(())- S MEv.
(I)
(0.r)
where pv: X + X is the orthogonal projection onto V, that is, Ran(Pv)=
(ūi) We say that A: (T - XvESa(x C(V:V), F (Fv)vEsu(X) is a weak approxima- tion operation for the family F (";X;)) if any E F and > 0 holds that
lim (0.))
sup
[v(z) - Pv(F(z))llx + 0.
Note that the condition (i) is stronger than the condition (i). An example of an approximation
operation for the family F = C"(X), that is, an Eappprximmaiion operation with all sequences E'= = =Ev (EV)ves)(X) subject to EV > 0, is the linear discretization
(n) = (Fv)v)s(()) Fy= Pv (Flv): V
(2)
Nonlinear discretization methods that do not rely Pv have been used, for example, in the numerical analysis of nonlinear partial differential equations. Here, X becomes an appropriate Sobolev space, and Galerkin approximation is implemented through finite-dimensional subspaces, V. spanned by finite element basis functions. We present an example for the nonlinear equation. Au(t) g(((t)) Ax(t) where g is a smooth convex function, when F + u, in Appendix A.
Below, we will study whether family, F C Dift' (X), of C² diffeomorphisms approximated by C² diffeomorphisms, Fy C (V), finie-dimeensionll subspaces, V. Of course, diffeomorphisms are bijective. Unless stated otherwise, from now on we will omit C'l and implicitly assume that diffeomorphism are C diffeomorphisms, We introduce two more notions that will play a key role in the further analysis.
X can be
Definition (Strongly Monotone). We say that (nonlinear) operator F: X + X on Hilbert space, X, is strongly monotone if there exists constant a >> O so that
(F(z1) - F022 (10)) - x2)x 2 allx1 - x21.3 for all Iı, x2 E X.
Definition 3 (Bilipschitz). We say that F if bilipschitz there exist constants > O and C 00 so that for all vı, D2 E X, e||= - x2l < [(())) ) F(x2))l < C ||m
x2ll-
2.2 A general framework for neural operators
In this paper, we are concerned with the modeling of diffeomorphisms between Hilbert spaces by bijective neural operators. Our working definition of neural operator, which generalizes the traditional notion, is given below. We note the presence of skip connection, which is essential.
Definition 4 (Generalized neural operator layer). For Hilbert space X, layer of neural operator a function F: X + X of the form
F(x) = TTG(Tiz),
where T2: X + X and X are compact linear operators and G: X X is a nonlinear operator in C¹ (X). A generalized neural operator, H X + X, is given by the composition
(5) where each Fe, i L, is of the form (7). the A - X are bounded linear operators and a: X + X is continuous operation (for example, Nemytskii operator defined by composition with a suitable activation funictionnnnntion in function spaces).
H = AL 0000000 AL-1 FL-1 ... o.. o
By using mapping properties of monotone operators [5). we can replace this detinition by using Hilbert spaces Y and Z that are isometric to X, Ti X + Y and Ta Z + X as compact linear operators. and G: Y Z as C nonlinear operator.

---

The generalized neural operators can represent the classical neural operators [30] 33]. For an explicit construction, we refer to Appendix C.I In the next section, we will study, under what conditions, bounded linear operators Ae. Nemytskii operators and neural operator layers F2, for which the generalized neural operator consists, can be continuously discretized.
We note that because G E C* ) in Definition 4 it follows that G E (X) and Lipx-xx (G) < co. Given a Hilbert space, X, a layer of a strongly monotone neural operator (respectively, layer of a bilipschitz neural operator,) is a function F: X + X that is strongly monotone (respectively, bilipschitz). Furthermore, a strongly monotone neural operator (respectively, a bilipschitz neural operator), is generalized neural operator with strongly monotone (respectively, bilipschitz), layers.
3 Category theoretic framework for discretization
Hilbert spaces may have dramatically dif- ferent behaviors than corresponding "well-behaved" maps between finite-dimensional Euclidean spaces. This observation applies to discretization and neural operators versus neural networks. In this section we explore this. We first present a no-go theorem, that there are procedures that continuously discretize an isotopy of diffcomorphisms. Next, we introduce strongly monotone neural operator layers, which are strongly monotone diffeoomorpisms,, and then prove that these allow a continuous approximation functor, that is, continuous approximations by strongly monotone diffeomorphisms finite dimensional spaces. We finally show that bilipschitz neural operator layers admit a representation via strongly monotone operators and linear maps, allowing for their continuous approximation.
WWell-behaved" operators between
3.1 No-go theorem for discretization of diffeomorphisms on Hilbert spaces
In this section, we present our no-go theorem. To formulate the 'impossibilityy' of something, we must define what is meant by discretization and approximation. Before this, we give informal statement of the no-go theorem.
Theorem (No-go Theorem, Informal). Let A be an approximation scheme that maps diffeomor- phisms F' on a Hilbert to a sequence of finite-ape fimmite-pproximattion Fv that are themselves diffeomorphisms. If F'y converges I' as V X,, then A is not continuous, that is, there are maps 123) that converge to F as > xo, but all F(J) are far from Fv.
We want to emphasize that most practical numerical algorithms continuous so that the output depends (in some suitable sense) continuously on the input. This shows that there are no such numerical schemes that approximate infinie-diimensional diffeomorphisms with finiteddimensional ones. In order to prove our no-go theorem in the most general setting, we phrase it in terms of category theory. Namely, we formulate A (which will denote the approximation scheme) as functor from the category of Hilbert spaces and diffcomorphisms thereon, to their finite-rank approximations.
Definition (Category of Hilbert Space Diffcomorphisms). We denote by D the category of Hilbert diffeomorphisms with objects OD that are pairs (X, F) of Hilbert space X and (possibly non- linear) C¹ diffeomorphism F: X - X and the set of morphisms (or arrows that 'map' objects to other objects) A that are either
1. (induced isomorphisms) Maps og that are defined for a linear isomorphism & X1 + X2 of Hilbert spaces Xi and X2 that maps the objects (X1, Fi) E OD to the object (o(Xi), F1 o 1 OD. or
2. (induced restrictions) Maps ax,,X, that are defined for a Hilbert space X1, its closed subspace X₂ C X2, and an object (X1, Fi)) E OD such that Fú =x Xx, Then ax,,x, maps to the object (X1,F) E OD to the object (X2, Filx,) E Op.
Definition 6 (Category of Approximation Sequences). We denote by B the category of approximation sequences, that has objects Og that are of the form SolX, (Sa))) where X is Hilbert space,
So(X) C S(X) = (V IV C is finte dimensional linear subspuce): are partially ordered lattices, UvES,(X) V = X, and Fv: V + V are - -ddiifeomorphhssss of spaces V E So(X).
5

---

The set of morphisms Ag consists of either
1. Maps Ag that are defined for linear isomorphism ø X1 + X2 of Hilbert spaces X1 and X2, and lattices So(X,) and So(X2) = (o(V) IV V (x())) } that maps the objects ((1))(X)), (Fv) (Fv)ves(x)) to X2, ( P Po--(w) o @0 (>)=(\)))))es(xx)))))) or
2. Maps Ax,.x, that are defined for a Hilbert space X1, its closed subspace X2 X1, and an object (X1, So(X)), (Fv)ves;(x))) such that F(X2)
X2 and So(X2) =
(V E (1) V C 32) is partially ordered lattice. Then AxT.x, maps the object SoX1), (Fv) to the object (X2, So (Fv))ess)x)))
Next, we define the notion of an approximation or discretization functor. In practice, an approximation functor an operator which maps function F in an infinite dimensional space X a function Fv that operate in finite dimensional subspaces V of X in such a way that functions Fv are close (in suitable sense) to the function F.
Definition 7 (Approximation Functor). We define the approximation functor, denoted by A: D B. as the functor that maps each (X, F) & OD to some (X, So(),, (Fv) (Fv)veso(x) E OB so that the Hilbert space X stays the same. The approximation functor maps all morphisms as to Ag and morphisms 02 to Xx, and has the following the properties
(A) For all r > 0 and all (X, EOD.
lim V-X
sup (0.r))
(1) F(x)llx = 0.
In separable Hilbert spaces this means that when the finite dimensional subspaces V X grow to fill the whole Hilbert space X, then the approximations Fv converge uniformly in all bounded subsets F.
We recall the notation limy used above: We consider (Sp(X). 5) partially ordered set and say that real numbers y/v converge to the limit as V X, and denote
lim
yv y,
if for all 0 there is Vo So(X) such that for V E (x( satisfying V Vo it holds that lyv yl <.
Definition 8. We say that the approximation functor A is continuous if the following holds: Let (X, (X, F(i) E OD be such that the Hilbert space is the same for all these ob- F) be approximating sequences of FX and (X, So(),, (Fj,) (vEs())))))) = A(X,F()) be approximating sequences of (l,).))). Moreover, assume that > 0 and
jects and let (X, (FV
lim
sup
(x) -x((xxx== =0
(6)
1-000 (U.))
Then, for all V E (S( the approximations F(1) of Fli) and Fv of satisfy
lim
sup 2V.)
- =
(7)
joo
The theorem below states a negative result, namely that there does not exist continuous approximating functors for diffeomorphisms.
Theorem 2. (No-go theorem for discretization of general diffeomorphisms) There exists functor D + B that satisfies the property (A) of an approximation functor and is continuous.
The proof is given in Appendix A.4.1 and is quite involved, but give overview of some of the steps here. A generalization of Theorem [ in the case where the norm topology is replaced by the weak topology, is considered in Appendix D.I
key fact is that finite dimensional diffeomorphisms the space of smooth embeddings consists of two connected components, one orientation preserving and the other orientation reversing. This is

---

Connecting point oc
din(V) = dim(
< din(V) c0
diff (V)
diff (V)
non-diffoonorphissms
dim(V)-
Figure 1: A figure illustrating the proof ideas for Theorem It represents the disconnected com- ponents of diffeomorphisms that preserve orientation, notated by diff't, and reverse orientation, notated, diff" The horizontal axis abstractly represents the two disconnected components of diff for finite-dimensional vector space V. The vertical axis represents the dimension of V. Observe how the two components of diff connect dim(V) - oc, and V becomes Hilbert space H.
not the case in infinite dimensions, see e.g. 332 and 45 For an illustration of this, see Figurel 1 The proof proceeds by contradiction. First, we consider the action of the approximation functor operates on an isotopy (path of diffcomorphisms) that connects two diffeomorphisms. The first has only eeventaaion-preserviin We then show that the image of the path under the approximation functor yields disconnected path, as the discretization 'jumps' from the orientation preserving component to the orientation reversing component. This violates continuity. To encode the notions of orientation preserving and orientation reversing that allow for description of nonlinear discretization theory, we use topological degree theory. This generalizes the familiar notion of orientation that uses the sign of the determinant of the Jacobian matrix.
discretizations, and the second only grientation-reeersin discretizations,
3.2 Strongly monotone diffcomorphisms and their approximation on finiteddimensional subspaces
In this section and the next two we show that, although Theorem precludes continuous approxima- tion of general diffeomorphisms, stronger constraints the diffcomorphisms allows one to sidestep the topological obstruction. In this section, in summary, we show that the obstruction to continuous approximation vanishes when the diffeomorphisms in question Key to our positive result is the following technical result that states that the restriction of the domain and codomain of strongly monotone diffeomorphism always yields another strongly monotone diffeomorphism.
assumed be strongly monotone.
Let V C. X be a minite-dimessoonll subspace of X, and let Py :X + X be orthogonal projection onto V. Let F X X be strongly monotone Ciddiffeomorphhssm Then, Py Fly - is strongly monotone, and a C -iddiffeomorphism.
Lemma 1.
The proof is given in Appendix A.5.I Lemma I implies that the discretization functor Ajin de- fined in 2 is well-defined functor from strongly monotone ("-diffcomorphisms of X to C²_ diffeomorphisms of V. Note that the discretization functor Atı on strongly monotone C' diffeomor- phisms may not be continuous approximation functor in the strong sense of Definitions/ 7 and g but it is obviously continuous approximation functor in the weak sense of Definitions TI and Therefore, we obtain that
Proposition 1. Let Auu be the discretization functor that maps F to Fly for each finite subspace V C X. Let Dam. and Bam be categories where F: X X and Fy V are strongly monotone C d--iffeomorphisms. Then, the functor Aynn D.m + Bem satisfies assumption (A') ) of weak approximation functoo Definition 77 and continuous in the weak sense of Definition 12
3.3 Strongly monotone Nemytskii operators and linear bounded operators and their continuous approximation on finite-dimensional subspaces
By Proposition] strongly monotone maps can be continuously discretized the weak sense. Thus, concern under what conditions, the maps, of which generalized neural operator consists, can be
7

---

strongly monotone. In this subsection, we focus on bounded linear operators and Nemytski operators (layers of neural operator will be discussed in the next subsection). The following lemma is obviously given by the definition of strongly monotone map:
Lemma 2. Let A X + X be linear bounded operator and satisfy (Au, u) 2 co /u/lll for some Co Then, A
X is strongly monotone.
Next, assuming that X (D;;),, we define Nemytskii operator by
Fd (u) u,
(8)
where 1R + R is continuous. In this case, we can show the following lemma by using (50
Corollary 3.3]:
Proposition Assume that a satisfies that [a(()] S Cj + C2 and the derivative of - (s) is defined a.e and satisfies o' (s) > 0. Then, F LD; + L2(D; R) is strongly monotonous.
3.4 Strongly monotone generalized neural operators and their continuous approximation finite-dimensional subspaces
As seen in Theorem below, strongly monotone layers of neural operators do not suffer from the same topological obstruction to continuous discretization as general diffeomorphisms. We now give sufficient conditions for the layers of a neural operator to be strongly monotone, and show that these conditions imply that those are diffeomorphisms.
Lemma 3. All strongly monotone layers of neural operators (F) defined by (4) are diffeomorphisms.
Also, the following theorem proven in Appendix A.6633
Theorem 3. Let Asin be the discretization functor that maps F Py Fly for each finite subspace V X. Let Dmmn and B smn be categories where F: X + X and Fy: - V are strongly monotone C¹ -functions of the form (4). Then, the functor Alin DDmmn Bemn satisfies assumption (A), and it is continuous in the sense of Definition g
The functor defined in Theorem 3 does not suffer from the same topological obstruction as functors for general diffeomorphisms, shown in the no-go Theorem This is because when Fy Pv F y is strongly monotone, its derivative D Fv V + is a strongly monotone matrix at all points x E V. Therefore it is strictly positive definite (see [47 Prop. 12.3]) and the determinant det Fv) is strictly positive. Due to this, the orientation of the finttediimensional approximations never switch does not apply.
signs, and the key technique used in the proof of the no-go Theorem 2
straightforward condition to guarantee strong monotonicity of a neural operator layer is given in
Lemma 4. Let F X >> X be a layer of neural operator that of the form F(u) = u TzG(Tuu), where T₃ - X. = 1.2 are compact operators and G X + X is C¹ -mooohh map. Assume that Fréchel derivative D of G at x satisfies the following for all x E X,
|DG|=llx-x S 1
(T
Then, F; X + X is strongly monotone.
See Appendixes A.6.1 and A.6.2 for the proofs.
3.5 Bilipschitz neural operators are conditionally strongly monotone diffeomorphisms
Now we show analogous result to Theorem 3. but applied to bilipschitz neural operators. Moreover, we will show that all neural operator F: - that are bilipschitz admit approximations that can be locally inverted using iteration for each point in their range using an iteration.
Theorem 4. Let X be Hilbert space. Then there is X, ||e||x such that the following is true: Let F: X + X be layer of bilipschitz neural operator. Then for all ri > 0 and e > there are linear invertible map Ao X + X, that either the identity map or a reflection operator
NNote that we can write the reflection operator across the hyperplane {e)+, that is. the operator x 2(2x, xe diagonal operator diag(- 1, in suitable orthogonal basis of X.
8

---

- x - 2(2, x), and strongly monotone functions H2 that are also layers of neural operators
such that
H₂ X - x, Hu(z) = Bx(x), k = .,,, where Bg X + X is a compact mapping and satisfies Lip(Bx) < c and
F(x)= H. .... H2 H1 Ao(x), for all x E Bx(0,r1).
(9)
Moreover, F E C (X, then J = 3)
The proof of Theorem 4 is in Appendix A.7.1 Theorem shows that we may always decompose a bilipschitz neural operator into the composition of strongly monotone neural operator layers H, and reflection operator Ag. Each H can be discretized using the continuous functor Ala from Theorem B If we consider the discretization (via the construction in Definition 6 using collection of subsets So C S(X) such that all V E (x) satisty e E V. then the operator Ao can be discretized by Ao,y Pv Aolv. These mean that if we write a bilipschitz neural operator neural operator where each layer is either of the form Id + Bj, where Lip( B2) < 1, or a reflection operator Ag. In either case, we may use linear discretization to approximate each layer. So, we may discretize F' in ball Bx (0, Rt) by discretizing each layer Hj and A where F = Hy o ... H1 o A1. We observe that the number of layers, J. depends on R.
sufficiently deep
We have observed that operators of the form identity plus compact term critical for continuous discretization. This insight motivates the introduction of residual networks as approximators within the framework of finite-rank neural operators. In what follows, we assume that X separable Hilbert space, with an orthonormal basis = JnEN. For N E N. we define EN X + R and EnSn Os Pn We note that PVN = Dx Ex, where Pvv X + X is the projection onto VN == span (ean(Pn)re (nningN Using Ex. DN, we define the class of residual networks in the separable Hilbert space, with T, N E N and activation function 0, as
DN : R + X by ENu ((u, (pi) x ..... (u, (x))) E RRN. DNa
R (X) x x: : (Ixx Dy o NN, Fin);
NN: : RN N are neural networks with activation function = T).
(10)
The following theorem proves universality result for each of the layers G, allowing to obtain general universality result for the entire network. The statement of the theorem requires the careful construction of neural pperator-representaae introducing a lot of technical notation, and so the presentation here in the main text leaves the details of the construction of & abridged. For the full statement of the theorem and definition of the notation, see Section A.7.2 Intuitively, is the 'wrapping' of a fxxedpooint process in a neural operator.
function &. Giving full description of & involves
Theorem 5.
Let R > O, and let F: X Defintion Let a be the Rectified Cubic Unit (ReCU) function defined by o axx == 10, Then, for any c (0, there are T, N E N and G NR.N. (X. that has the form
X be layer of bilipschitz neural operator, in
G= (ldx + DN N NN Ex) o... (Idx + DD N Ex), such that each map (Idx + Dx o NN, Ex) is strongly monotone Cidfffoomorssssss on some ball and
sup
[1(()) G A(x)lxx SC
(0.
where A: X X a linear invertible map that is either the identity map or reflection operator x + x - -2(x,e) e)xe with some unit vector e E X. Further, Go A: (0 R) + G o A(Bx (0, R)) invertible, and there is some neural operator 4 : G (0, R)) A( (0, R)) so that
(G o A)
= A $
The proof is given in Section A.7.2 Neural operator & becomes a better approximation of the inverse operator when it becomes deeper. Theorem means that neural operators are an operator algebra of nonlinear operators that are closed in composition and when the inverse of neural operator exists, the inverse operator can be locally approximated by neural operators.
In the case when the separable Hilbert space X is the real-valued T,2-function space T22 R). residual networks in the separable Hilbert space can be represented as residual neural operators defined by [19). See Lemma I for details. Then, obtain the following

---

Corollary 1. Let D C B be a bounded domain, and let = {gnInEN be an orthonor- mal basis in RD) Assume that the orthonormal basis yp include the constant func- tion. Let .N. (T2 R)) be the class of residual neural operators defined in 179 Then, the statement replacing X with (R) and G E R. with G e RNOO.NN (L, R)) in Theorem S holds.
(L
The proof is given by combination of Theorem 5 and Lemma 11] We note that the assumption that the orthonormal basis y includes the constant function is satisfied if we choose yp to be a Fourier basis, which yields the Fourier neural operator, see e.g.. 39 38).
Remark 1. In this section, we have shown that residual networks in separable Hilbert space X, defined in (10g, are universal approximators for layers of bilipschitz neural operators. Additionally, in the specific case where X = = (D; R), residual neural operators, defined in Definition 9 also provide universal approximators for layers of bilipschitz neural operators. We note that the residual network we have discussed is locally invertible but not globally. By introducing invertible residual networks on Hilbert space X, defined (166. we can similarly prove that these networks by employing sort activation functions (see [3 Seccion 4]) are universal approximators for strongly monotone diffeomorphisms with compuct support. Specifically, when X = L2 RR), invertible residual neural operators, defined in Definition 2 are also universal approximators for strongly monotone diffeomorphisms with compact support. For further details, we refer to Appendix
4 Quantitative approximation
Quantitative approximation results for neural networks, see e.g. [57 or 23 can be used to derive
quantitative error estimates for discretization operations. Let F: Bx (0,r) + X be a non-linear function satisfying F E Lip (0, r)) in n = 1, or F E C" (B r); if 2 2. Then, F can be discretized using neural networks in the following way: Let Ey > 0 be numbers indexed by the linear subspaces V C X such that EY + 0 as V + X. When E = (EV)vES(X). in the sense of Definition| [ an E-approximation operation ANN : F - (Fv)ves(x) in the ball Bx (x,00, r) can be be obtained by defining Fv Jyy o V + VV isomorphism, d = dim(V), Fv.:: Ed + Rd is feed-forward neural network with ReLU-activation functions with at most ((d) logz ((++)/sv) layers and C(d)s log,( ((+r)/Ev) non-zero elements in the weight matrices. Details of this result are given in Proposition 5 Appendix A.8
V, where Jv: - Rd is an isometric
Conclusion
In this work, we have studied the problem of discretizing neural operators between Hilbert spaces. Many physical models concern functions RR + R, for example L (R*), the computational methods based on approximations in finite dimensional spaces should become better when the dimension of the model grows and tends to infinity. We have focused on diffeomorphisms in infinite dimensions, which are crucial to understand in generative modeling. We have shown that the approximation of diffeomorphisms leads to computational difficulties. We used tools from category theory to produce a no-go theorem showing that general diffeomorphisms between Hilbert spaces may not admit any continuous approximations by diffcomorphisms on finite spaces, even if the approximations are allowed to be nonlinear. We then proceeded to give several positive results, showing that diffeomorphisms between Hilbert spaces may be continuously approximated if they are further assumed to be strongly monotone. Moreover, we showed that the difficulties can be avoided by considering a restricted but still practically rich class of diffeomorphisms. This includes bilipschitz neural operators, which may be represented in any bounded set as a composition of strongly monotone neural operators and strongly monotone diffeommrphisms. We then showed that such operators may be inverted locally via an iteration scheme. Finally we gave a simple example on how quantitative stability questions can obtained for discretization functors, inviting other researchers to study related questions using more sophisticated methods.
10

---

Acknowledgments
TF was supported by JSPS KAKENHI Grant Number 1P24K169443 JST CREST JPMICR24QS, JST ASPIRE JPMJAP2329, and Grant for Basic Science Research Projects from The Sumitomo Foundation. MP was supported by CAPITAL Services of Sioux Falls, South Dakota and NSF-DMS under grant 3F5083. MVdH was supported by the Simons Foundation under the MATH X program, the Department Energy, BES under grant DE-SC0020345, and the corporate members of the Geo-Mathematical Imaging Group at Rice University. A significant part of the work of MVdH was carried out while he was an invited professor at the Centre Sciences des Données at Ecole Normale Supérieure, Paris. M.L. was partially supported by AdG project 101097199 198 of the European Research Council, Centre of Excellence of Research Council of Finland and the FAME flagship. Views and opinions expressed are those of the authors only and do not necessarily reflect those of the funding agencies or the EU.
References
[1] Ahmed Abdeljawad and Philipp Grohs. Approximations with deep neural networks in soboley time-space. Analysis and Applications, 20(03):499955) 541, 2022.
[2] Giovanni S. Alberti, Matteo Santacesaria, and Silvia Sciutto. Continuous generative neural networks. arXiv:2205.14003 2022.
[3] Cem Anil, James Lucas, and Roger Grosse. International Conference on Machine Learning, pages 291-301. PMLR, 2019.
Sorting out lipschitz function approximation. In
[4] Ivo Babuška and Manil Suri. On locking and robustness in the finite element method. SIAM J. Numer. Anal., 2965):1261-1293, 1992.
[5] Heinz H. Bauschke and Patrick L. Combettes. Convex analysis and monotone operator theory in Hilbert spaces. CMS Books in Mathemattcs/Ouvrrges de Mathématiques de la SMC. Springer, Cham, second edition, 2017. With foreword by Hédy Attouch.
[6] Roberto Bentivoglio, Elvin Isufi, Sebastian Nicolaas Jonkman, and Riccardo Taormina. Deep learning methods for flood mapping: a review of existing applications and future research directions. Hydrology and earth system sciences, 26(16):4355-4378 2022.
[7] Kaushik Bhattacharya, Bamdad Hosseini, Nikola Kovachki, and Andrew M Stuart. Model reduction and neural networks for parametric pdes. The SMAI journal of computational mathematics, 7:121-157, 2021.
[8] Klaus Böhmer. Numerical methods for nonlinear elliptic differential equations. Numerical Mathematics and Scientific Computation. Oxford University Press, Oxford, 2010. A synopsis.
[9] Klaus Böhmer and Robert Schaback. nonlinear discretization theory. Comput. Appl. Math., 254:204-219, 2013.
[10] Philippe Ciarlet. Linear and Nonlinear Functional Analysis with Applications. Society for Industrial and Applied Mathematics, USA, 2013.
[11] Masoumeh Dashti, Stephen Harris, and Andrew Stuart. Besov priors for Bayesian inverse problems. Inverse Probl. Imaging, 6(2):183-200) 2012.
[12] Maarten De Hoop, Daniel Zhengyu Huang, Elizabeth Qian, and Andrew M Stuart. The cost- accuracy trade-off operator learning with neural networks. arXiv preprint arXiv:2203. /3/81, 2022.
[13] Dennis Elbrächter, Dmytro Perekrestenko, Philipp Grohs, and Helmut Böleskei. Deep neural network approximation theory. IEEE Transactions on Information Theory, 67(5):2581 2021.
[14] Takashi Furuya, Michael Puthawala, Matti Lassas, and Maarten de Hoop. Globally injective and bijective neural operators. arXiv preprint arXiv:2306.0330 039822 2023.

---

[15] Jean Gallier. Geometric methods and applications, volume 38 of Texts in Applied Mathematics. Springer- Verla,, New York, 2001, For computer science and engineering.
[16] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2-22,, 2016.
[17] Bruno Gavranović, Paul Lessard, Andrew Dudzik, Tamara von Glehn, João GM Araújo, and Petar Veličković. Categorical deep learning: An algebraic theory of architectures, arXiv preprint arXiv:2402.15333 15333, 2024.
[18] David Gilbarg and Neil S. Trudinger. Elliptic partial differential equations of second order. Classics in Mathematics. Peringer-Verllge Berlin, 2001. Reprint of the 1998 edition.
[19] Evarist Giné and Richard Nickl. Mathematical foundations of infiniteddimensional statistical models. Cambridge university press, 2021.
[20] Aidan Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible resid- ual network: Backpropagation without storing activations. Advances in neural information processing systems, 30, 2017.
[21] Somdatta Goswami, Aniruddha Bora, Yue Yu, and George Em Karniadakis. Physics-informed deep neural operator networks. In Machine Learning in Modeling and Simulation: Methods and Applications, pages 219-254. Springer, 2023.
[22] Ingo Gühring, Gitta Kutyniok, and Philipp Petersen. Error bounds for approximations with deep relu neural networks in w s. p norms. Analysis and Applications, 18(05):803 803-859, 2020.
[23] Ingo Gühring, Gitta Kutyniok, and Philipp Petersen. Error bounds for approximations with deep ReLU neural networks in W* norms. Anal. Appl. (Singap.), ((5):803-859, 2020.
[24] E. M. Harrell and W. J. Layton. L estimates for Galerkin methods for semilinear elliptic equations. SIAM J. Numer. Anal., 24(1):52-58, 1987.
[25] Isao Ishikawa, Takeshi Teshima, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Universal approximation property of invertible neural networks. Journal of Machine Learning Research, 24(287):1-66 2023.
[26] Thomas Jech. Set theory. Perspectives in Mathematical Logic. Beringer-Verlage Berlin, second edition, 1997.
[27] Jari Kaipio and Erkki Somersalo. Statistical and computational inverse problems, volume 160 of Applied Mathematical Sciences. Springer-Verte Verlag, New York, 2005.
[28] R. I. Kačurovskiï. Nonlinear monotone operators Banach spaces. Uspehi Mat. Nauk, 23(2(140)::21-118, 1968.
[29] Gavin Kerrigan, Justin Ley, and Padhraic Smyth. Diffusion generative models in infinite dimensions. In Francisco Ruiz, Jennifer Dy. and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 9538-9563. PMLR, 25-27 Apr 2023.
[30] Nikola B Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew M Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. J. Mach. Learn. Res., 24(89):1-97 2023.
[31] Anastasis Kratsios and levgen Bilokopytov. Non-euclidean universal approximation. Advances in Neural Information Processing Systems, 33:10635-10646, 2020.
[32] Nicolaas H Kuiper. The homotopy type of the unitary group of hilbert space. Topology. 3(1):19-30, 1965.
[33] Samuel Lanthaler, Zongyi Li, and Andrew Stuart. The nonlocal neural operator: Universal approximation. arXiv preprint 1rX21 2023.
12

---

[34] Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis. Error estimates for deeponets: deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6(1 (()::nac001, 2022.
[35] Samuel Lanthaler, Andrew M Stuart, and Margaret Trautner. Discretization error of fourier neural operators. arXiv preprint arXiv:2405.000 02221, 2024.
[36] Matti Lassas and Samuli Siltanen. Can one use total variation prior for edge-preserving bayesian inversion? Inverse problems, 20(5):1537, 2004.
[37] Markku S Lehtinen, Lassi Paivarinta, and Erkki Somersalo. Linear inverse problems for generalised random variables. Inverse Problems, 5(4):599, 1989.
[38] Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes general geometries. Journal of Machine Learning Research, 24(388):1-26, 2023.
[39] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations. arXiv preprint arXiv:2010.0885 2020.
[40] Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint 19100011,,,,,93, 2019.
[41] Robert J. Martin and Patrizio Neff. Minimal geodesics on gl(n) for left-invariant, right-o(n)- invariant riemannian metrics. Journal of Geometric Mechanics, 8(3):323-35)) 2016.
[42] Peter W. Michor and David Mumford. A zoo of diffcomorphism groups on R". Ann. Global Anal. Geom., (144))5595540 2013.
[43] Donal O' O'Regan, Yeol Je Cho, and Yu-Qing Chen. Topological degree theory and applications, volume of Series in Mathematical Analysis and Applications. Chapman Hall/CRC, Boca Raton, FL, 2006.
[44] Michael Puthawala, Konik Kothari, Matti Lassas, Ivan Dokmanic, and Maarten de Hoop. Globally injective relu networks. Journal of Machine Learning Research, 23(105):1-55, 2022.
[45] Calvin R. Putnam and Aurel Wintner. The connectedness of the orthogonal group in Hilbert space. Proc. Nat. Acad. Sci. U.S.A., 37:110-112, 1951.
[46] Calvin R. Putnam and Aurel Wintner. The orthogonal group in Hilbert space. Amer. J. Math., 74:52-78, 1952.
[47] R. Tyrrell Rockafellar and Roger J.-B. Wets. Variational analysis, volume 317 of Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. Springer-Verlag, Berlin, 1998.
[48] Matti Lassas Saksman, Samuli Siltanen, et al. Discretizationninvaaiin bayesian inversion and besov space priors. arXiv preprint arXiv:0901.4220, 2009.
[49] Martin H. Schultz. Error bounds for the Boylligh-Ritz-Galeeee method. J. Math. Anal. Appl.. 27:524-533, 1969.
[50] R. E. Showalter. Monotone operators in Banach space and nonlinear partial differential equations, volume 49 of Mathematical Surveys and Monographs. American Mathematical Society, Providence, RI, 1997.
[51] Andrew M Stuart. Inverse problems: bayesian perspective. Acta numerica, 19:451-555 2010.
[52] Manil Suri, Ivo Babuška, and Christoph Schwab. Locking effects in the finite element approxi- mation of plate models. Math. Comp., 64(210):461 482, 1995.
[53] Michael F. Taylor. Partial differential equations I. Busic theory, volume 115 of Applied Mathematical Sciences. Springer, New York, second edition, 201
13

---

[54] Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling- based invertible neural networks are universal diffeomorphism approxima- tors. Advances in Neural Information Processing Systems, 33:3362-3373, 2020.
[55] Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk. Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47- 2023.
[56] Jared Willard, Xiaowei Jia, Shaoming Xu, Michael Steinbach, and Vipin Kumar. Integrating scientific knowledge with machine learning for engineering and environmental systems. ACM Computing Surveys, 55(4):1-37, 2022.
[57] Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103-114, 2017.
14

---

A Proofs and additional examples
A.1 Non-linear discretization
In this section we consider the non-linear discretization theory, see [91. Let S2 C. I be bounded open set with smooth boundary. Let H4 (22) = {u E (2) Vu E (2)) and H. (12 {u (12) | w/an = O} be the Sobolev spaces with an integer order of smoothness and H (S2), E R the Sobolev space with a fractional order of smoothness [53].
For simplicity, we assume in the section that n = 1, and that Ω C R is an interval and we point out that in this case the Sobolev embedding theorem implies (2) c C(2) which significantly simplifies the constructions below.
Let g E C (R; R), m 2 1, be the derivative of convex function G C (R; I) satisfying
-co - G(r) - ci (1+r)), r E R, cu, P1 > 0.
(11)
Let F Flo): (2) + (2), F(O) : x > u be the solution operator of the non-linear differential equation
(! g(u(t)) = x(1, LE
(12)
ulan 0,
(13)
where A is the Laplace operator operating in the / variable, where the function x(t) is a physical source term. Because the that solves 12 is the minimizer of the strictly convex and lower semi-continuous function H X + R.
(v)) Vell + / (G(((t))=(((((())) x(t)v(t))dt
(14)
the Weierstrass theorem, see 150], Theorem II.8.1, implies that 12-(33 has a unique solution.
Moreover, by regularity theory for elliptic equations, see [18), Theorem 8.13, we have that the solution u is in (S2). Moreover, as G is convex and satisfies (II) the function in (14) is
coercive, for any R >> 0 there is p(R) > 0 such that when ||=||| (0) R. then || (2)))) S p(R).
The equation
12
13 can be written also as an integral equation
u(() + L.- VVI, ((),)))(((()d'===== = =- V (()()))) I LE 2
(15)
where (t,, f') is the Dirichlet Green's 's function of the negative Laplacian, that is, - AAV((.)) = 5-). (,,t')aa 0, and gz = g(u). We can write (15) as
(Td + Q g. = Q(x), Qu(() = 1 ((L,t))x(()d'''
(16)
where 1Hz--H9/4 (92) + H3)+(S2) and i HH)(S) n H6 (s)) + H.O (S2) are compact operators and as H²))((2) C C()), the operators g- H3) + (2( and Q L²( + H²( (12) are continuous, we see that
F= + (Q (...)== -1344 Hb (0)
H (2))
is a layer of the neural operator. As G E Cm+11 m 2 2, we have that F HG( + (22) is C'-smooth. Moreover, as G is convex, the operator F H (22) + H²(() is strongly monotone and hence by Lemma 3 F has an inverse map F-1, H (S2) + H²( that is C--smooth Thus, the solution of 12) can be represented as
u = F=FF)(Q(())
Let V C X (2)) be finite dimensional space. The Galerkin methods obtain an approxima- tion for the solution of the boundary value problem (12 13) using Finite Element Method. To do this, let w E V be the solution of the problem
/ (Vw(t)- Vo(t) + (lw(()) 2
/ x(t) (()ddt, for all & E V.
(17)
15

---

When ,1....,00 A E V are basis of the finite dimensional vector space V, and we write w = Ei=1 w the problem T7 is equivalent that (WI,..., ........ wa) E Ea satisfies the system equations d equations
Σ by --
(5(()gg
Σ wadr(t) de -o((ddt) for 1,2, ......., (18)
12
where
/ Voy(t) V
byk =
1
When I E V. we define the map
V by setting map F (x) w. where E V the
solution of the problem (17). When
dG
FPDE (F X
X ((8) ds
(8) G 2
(R) is convex and satisfies I
we define the map AFEM: F(s) + P. for F(a) E FPDE. As the the function g in the equation (88) non-linear, the solution u of the continuum problem (15)-T3 and the solution of the finite
dimensional problem have typically no linear relationship, even if the source x satisfies x E V.
Let R > 0 As the embedding ( ( + H (22 is compact and Q ( - (22) Ho) (92) is continuous,
that the image of the closed ball Bx (0, R) in the map Q is precompact set Q (0, R)) C X. As F
X X is continuous the set of corresponding solutions,
ZR fu E X (Q(x)), x Bx (0, R)},
precompact in X. Hence,
lim sup I| - Pv) = O. ucZR
(19)
As G' is convex, the Fréchet derivative of the map F Id + Q gu X - X is strictly positive linear operator at all points x e X. This, the uniform convergence (19). and the convergence results for the Galerkin method for semi-linear equations, [49 Theorem 3.2), see also (24), imply that in the space X (2)
lim V-X | SR
sup (| (x) (x) (x)(x = 0.
(20)
These imply that AFEM: F(9) F()) an approximation operation for the function F(9) FPDE
The map F(9) + V is called the Galerkin approximation of the problem 12
13 13 and is an
example of the non-linear approximation methods. The properties of the Galerkin approximation is studied in detail in [8, in particular sections 4.4 and 4.5.
A.2 Negative results
On the positive results, in Appendix A. we have considered nonlinear discretiation of the operators + Au + g(u). To exemplify the negative result, we consider the following example on the solution operation of differential equations and the non-existence of approximation by diffeomorphic maps: We consider the elliptic (but not not strongly elliptic) problem
B,u == dt (1)+ - s) s) u(())
d
d
f(t), te [0,
(21)
with the Dirichlet and Neumann boundary conditions
(()) 0. di m(()
= 0.
(22)
Here, 0 < 8 < is a parameter of the coeffient function and sign(t - 5) = if > and sign(/ - 8) < if < 8. We consider the weak solutions of (21) in the space
u E (0, 1) {0 H (0, 1): v()) 0}.
16

---

We can write
Bau -D,?? Ag A.D
where
A,u(t) = ()signtt - s)u(t),
parametrized by 0 S S. are multiplication operations that are invertible operators, A, (22 1) + (0, (this invertibility makes the equation (2I) elliptic). Moreover, D. and D.a²) are the operators $ with the Dirichlet boundary condition v(0) and v(1) 0, respectively. We consider the Hilbert space X b, (0, 1); to generate an invertible operator G, X X related to (21). we write the source term using an auxiliary function 9.
dc f(t) = Qg = dr dr + gt)..
Then the equation,
B.u = Q9,
(23)
defines a continuous and invertible operator,
G. G
G
In fact, G. B Q when the domains of B, and Q are chosen in suitable way. The Galerkin method (that is, the standard approximation based on the Finite Element Method) to approximate the equation 23 involves introducing complete basis (t), 1,2.... of the Hilbert space X, the orthogonal projection
Pa X Xn span (x; j 1, ..... n},
and approximate solutions of 23 through solving
Un Pr.P Un Xn, In -Png-
(24)
This means that operator Q= :g u approximated by (PoB,P.)--PPQPP + the when Pa P.B.P, X, X, is invertible.
The above corresponds to the Finite Element Method where the matrix defined by the operator P,B.Pr is b(s) = [6,x (s)]iAm E R2x0 where
(8)(8) f (1+ -
dt x²()) dt (x()ddt j. =-k1 1,..,....,
Since used the mixed Dirichlet and Neumann boundary conditions in the above boundary value problem, we see that for = 0 all eigenvalues of the matrix b(s) are strictly positive, and when strictly negative. As the function
= 1 all eigenvalues function, we see that there exists E (0, 1) such that the matrix b(s) has a zero eigenvalue and is no invertible. Thus, we have situation where all operators B-
b(s) is continuous matrix-valued
invertible (and thus define diffeomorphisms X - X) but for any basis X; (1) and any n there exists s E (0, 1) such that the finite dimensional approximation b(s) R R is not invertible. This example shows that there no FEM-based discretization method for which the finite dimensional approximations 1). are invertible. The above example also shows a key difference
- u, [0,
of all operators B- between finite and infinite dimensional spaces. The operator A, L 1) + ( 1) has only continuous spectrum and not eigenvalues nor eigenfunctions whereas the finite dimensional matrices have only point spectrum (that is, eigenvalues). The continuous spectrum makes it possible to deform the positive operator A, with s operators A,, 0 < S 1, are invertible but this is not possible to do for finite dimensional matrices. We point out that the map s - A, is not continuous the operator norm topology but only in the strong operator topology and the fact that An can be deformed to A₁ in the norm topology by path that lies the set of invertible operators is deeper result. However, the strong operator topology enough to make the FEM matrix b(s) to depend continuously
negative operator A, with
1 in such a way that all
17

---

A.3 Application of Theorem 2 to Neural Operators
In this section, give example of the application of Theorem 2 applied to the problem of using a neural operator to solve a linear PDE. The guiding idea is to draw inspiration from Figure I the space of diffeomorphisms X is disconnected if X is finite-dimensional space, but connected it is Hilbert space.
Before constructing the path, we first construct a isotopy of diffeomorphisms in (2( in several steps
Let p: R + [0, 1] be a smooth function such that pl (- = 0, pl11,00) = 1. Let R(0) E R2x2 be
the rotation matrix given by
R(0)= = cos(() sin
sin(t) (+)
(25)
cos(O) (
Then, define the function Rg: + 12x2 by
(t( = R(p(t - i)m).
(26)
Note that (t(t) (2),), for 1 i, R,()) I, 2 2 (t)
Next, we consider the 'little ell' space, e = {01 .2.....): Ei < c0). Consider the linear
operator R: (e2 - ( on f given by a pair-wise coordinate representation given by the infinite dimensional 'matrix' with block structure
(1(1)
R(t)=
R2(t)
(27)
Finally, define R: [0, 1] (6 + by
R
1
EE (0.1)
R(t)
1-t
(28)
-1
t21
and define the related operator R. [0, + (ft + P2) by
Re()= (-1
(-1
R(1)
(29)
Proposition (R, R isotopies of smooth diffeomorphisms in (2), The maps H1: 6 (0, 1] + 62 and H2: 82 x [0, + l2 defined by
H (v, R(t)e, H2(v, 1) R(()),
(30)
are isotopies of smooth diffeomorphisms in l and agree at L 1.
Proof. We prove that Hi is isotopy of smooth diffeomorphisms on l², as the proof of the same for H2 is very similar.
We first show that H,(;,t) (if) for each
For < 1, R operates on vectors in 2 in only a finite number indices. those indices corresponds to rotation, and hence is smooth diffeomorphism. When /. > 1, it corresponds to scalar multiplication by 1, and so too smooth diffeomorphism. Now we show that R()) is continuous The only point where continuity may fail is at = 1, and there it may only fail in the
18

---

limit from the left. Given V E e,, we compute
R1 R1( T-
U2
02
(2(1-1)
U3
U4
0/
lim (R(t)c-R(I)el)) lim ---
(31)
2-1-
R& (
U2k-1
12k-1
D2kk
U2K
02k+1
126+1
R 1--1
U2k+2
026+2
Let (()) = 1 Then for any integer k. we have that
R 1 = R
P
R((T) = = 7 E R222
(32)
1 - t
21
and likewise when 2 (t) + 1, then Ra
R2x and so the r.h.s. of Eqn. 31 becomes
1-7
U1
V1
U2
U2
U3
M/
U1
2/
:
lim 1-1-
(33)
U2x U2x (0)-1
Ri (
(1)--
(2)=
U2 (1)
(2)+1))+
(1)+1))++1
U2k (4)++2
U2 (1)++2
lim
(34)
(2)-
(2)-
(1) [-
U2k- (t)
(2)+2))+2
2012 (1)+1
20)x-(0)+2
<2
(35)
(1)
1-, + oc, and so, by standard estimates, 2 Ei=k((1) 2s + 0. Hence, R(t) is
000
continuous on L at / = 1. Finally, by definition Finally, we define H: 2 x [0, 1] + 8e00 by gluing the isotopies Hi and H2 together by
1, we have that H1 H1 1) R(1) - =R(() H2 1).
(v, if
H(v,t) 22( ift
(36)
19

---

Proposition 4. The function H given by Equ. 36 is an isotopy of diffeomorphisms in e.
Proof. Continuity of H. follows from continuity of R. R, and that R(1) = RR(1). For each / E [0. 1]. ((,t) E ((*)
A.4 Proofs from Sec. 3.1
A.4.1 Proof of Theorem 2
Proof. Assume that A: D - B is functor that satisfies the property (A) of approximation functor and is continuous. Let us consider the case when = 1²t
Let E X be a unit vector and B. X + X be a linear map Ba (zx e)xe. In other words, B is diagonal matrix di 1,.....) in some orthogonal basis where e is the first basis vector.
By [32] Theorem 2], see also and [45]. for an infinite dimensional Hilbert space X the space GL( of linear invertible maps have only one topological component in the operator norm topology and the set GL(X) is contractible and hence path-connected. This implies that there are linear maps A, X > X, l E [0. 1] such that Ao Id, and A E GL(() is arbitrary invertible linear map and that t + A, E GL( is continuous. Similarly. in an infinite dimensional real Hilbert space X the space Og (X) of the orthogonal operators A X + X path-connected in the operator norm topology, see [46] Section 4J. We recall that an orthogonal operator A X + X is a bounded linear operator satisfying A AA I, where A' X + X is the adjoint (i.e., the transpose) of the operator A : X - X. Observe that in finite dimensional space I the set Og (IR)) has two disjoint topological components, those which determinant and those which determinant is 11, and thus the set Op (R") is not connected.
A bounded linear operator B X X is called rotation operator of the form B 6 where S: X + X is a linear, bounded, skew-symmetric operator, S SS and e + S + 21 ... An orthogonal operator A is called reflection if it is not rotation.
3: 5 +....
One of the fundamental reasons for the surprising property that the space (x(X) of orthogonal operators in X is path-connected is that in an infinite dimensional real Hilbert space X every orthogonal operator A X + X can be represented is no valid in the finite dimensional spaces), that is, A = e where S1,S2 X + X are skew-symmetric linear operators. Thus any operator A OR(X) can be connected the identity operator Id: X + X via a path : + e E (x))), t E [0, 1], so that a(0) = Id and (1) A.
product of two rotation operators (that
As a motivating example on the fact that (x( is connected in the operator norm topology, is to consider similar, but easier result in the strong operator topology, that is, topology generated by the evaluation maps A > A(u), u E (0, 1). Observe that the strong operator topology is weaker than the operator norm topology, but in finite dimensional space those are equivalent.. So, let next show that the operators 7d and - Id are in the same topological component of Ox(X) in the strong operator topology. As all infinite dimensional separable Hilbert spaces are isometric to T,2 (0. 1), let us consider the Hilbert space 2 (R) and the orthogonal linear operators A, : (0, 1) + L 1). E [0, 1] that are given by
A,((t) sign(t s) uu().
(37)
where u E (,, 1) and sign(f) is the sign of the real number f. Then, for every E (,2 1) the functions
au : As(u),
continuous functions a. [0, 1] + (0) As Ao Id and Ao -Td, that
+ A, E (z( 1)), s [0, 1].
continuous path in the strong operator topology, that connects the operator Id to - -d.. We recall that the continuity of the function s + A, in the strong operator topology, means that for all E (2 1) the map
A&u L² 1).
20

---

is continuous. As for s >
A ANo u = | [2u(x)] 0,
as 8 - so. we see that 8 + Ag is continuous in the strong operator topology. All maps A, (0. - (0, 1) are invertible and Ag = Id, A = -d. This has implications e.g. for Finite Element method analysis of partial differential equations (See Appendix/A.5.22
Note that when is any infinite dimensional separable Hilbert space with real scalars, there is an isometry J X + 1,2 1) (e-g., the linear operator mapping an orthogonal basis of X to an orthogonal basis 2 (0, 1). Then, the maps A. X + X given by Å, J_1 A, J X X, where A, ()) - (0, 1) are given above, are continuous paths in Og (X) from Ag Id X + to A = -Id X + X. As discussed above, the deep result that Id and -Id can be connected by continuous path in the operator norm topology of (x(X) ( is given in 32 Theorem 2), see also 451
Let us first warm up by proving the claim in the case when an additional assumption (B) is valid:
(B): When V C. X is finite dimensional discretization maps Fy of linear invertible maps F: X - X are linear invertible maps and moreover, the So = SX) consists of all linear subspaces of X.
Under assumptions (A) and (B), we consider the case when A1 -dd and denote by F; = A, X > X a family of linear maps such that Ap Id, and that f + A, E GL(X) is continuous path of operators connecting Id to A = -dd. Let (X, S(X). (FLv)VEs(X)) = A(X, Ft), that is, F.v: > V be the linear isomorphism that is the discretizations the map F. : functor A is continuous, the map / Fi,v is continuous map from [0, 1] to the space of linear operators endowed with the topology of uniform convergence on compact sets, c.f. limit (7). As P.,w are linear, this implies that for all 1' E 1).
X. As the
lim Fe, Fe,wll De
lim sup 8-1
(F) Fe, =
(38)
=EVn (0,1)
and hence the map t + Fe, is continuous map [0, 1] + GL(W).
Let 0 < 60 1/2 and > 1. Let to and t 1. Using Property (A) for operators Fe,, 0, 1, we see that there are Wo. Wi E So(X) such that for all V E So(X) satisfying V o W, we have
sup # xx,r))
[|Fi,, (x) (x))lxx Eo. for (0,
Let W E So(X) be such a finite dimensional space which dimension an odd integer and that Wo + Wi C W. Then,
sup <0
(F),, Fe, (x) S co. for E (0,
(39)
Clearly, Ag Id satisfies Ao(W) W and Ag W - W is an invertible linear map. Similarly, the map A = - Id satisfies A₁ (W) = W and A W + W is invertible linear map. These observation and assumptions (A) and (B) imply that Fo,w W + and Fi.w : + W are linear maps and by inequality (39).
1 |F Aiww < < 2
|oo. Wlwwlw <0 < <
(40)
and as the dimension of W is odd, we have
det(App w) = det(Idlw) = 1, det(AA/w)=dett = det(-Id] -
Let Ba.. = (1-1)) Anl w + + Fo (| w- = 1, we see using (40) that the maps B W + W and (aol:/)wlllllo. =
W + W and BI., = (1 ()= A1 w + s sFo. W + W. As
BT. W + W are invertible and that the functions s + Bo E GL(W) and + BI,, e GL(W) are continuous matrix-valued functions which determinants does not vanish and
dct(Bo..) det(Aolw) > 0, det( det(Arlw) < 0,
21

---

for all 8 E [0. These imply that
det(Fo,w) det(Bo,1) > =0, det(Fi.w) - det(Bo,1) O,
(41)
However, as t + Fi,w is continuous maps [0, 1] + GL(W), see (38) we have that
+ det(Fe,,v))
(42)
is a continuous function [0, 1] - IR which does not obtain value zero and thus has a constant sign. However, this is in contradiction with formula (41).
Next we return to the main part of the proof where we do not assume that assumption (B) is valid, but only assume that the assumption (A) is valid. In this case, we use degree theory instead of the determinants.
Next, let F. = A. X + X be family of linear maps such that Ao = Id, but that A1 (x = 2e(x, x)x reflection, where X unit vector, and -> A, E GL(X) is continuous. Again, let (X, So(),, (Fiv)y diffeomorphisms that are discretizations the map F: X - X. As the functor A is continuous, the map Fi, B( E C(B(O, R)) continuous map for all V E So(X) and R 0. Moreover, we note that by our assumptions on B, the discretized map Fi,v V - V, is -'diiffcomopphism of V.
- A(X, Ft), that is, FL, V + V be C
As V is a finite dimensional vector space, we can use finite dimensional degree theory and consider the degree deg(4',v V Bx (0, p). that is the degree of the map Fr,v Bx(,r) - V at the point p. We recall that when Q C F is open and bounded, F: 52 + R a C -smsmooth function and p Fd are such that p & f(22) and for all x E (p) the derivative D (x) is invertible matrix, the degree defined to
(
Σ (p)
sgn(de((Df(x)))
where sgn(r) is sign of real number r. Also, the map / + deg (h, S2, defined for a continuous
function : 52 + Rdd by approximating h by -mmoohhth function, see [43. Definition 1.2.5 on details. Let us denote Bw r0, W n 3x (0,
Let r > 0. Recall that by assumption (A), for Fu = Id: + X and Fi = B X + X there are finite dimensional spaces Vo E (X) and Vi (x( such that E Vo and E V1 and that when V E (X) satisfies Vo C Vnd Vi C V, then
sup = xx (V.))
[F Fo(x))lx <r/4,
(43)
and
sup REB (V.))nV
- <//4.
(44)
Moreover, let W E So) be such finite dimensional linear space that Vo C W and Vi C W. Observe that as e E W, we can decompose W as
W span(e) @ fø W (z, = 0}.
(45)
and denote W' = fa E W: (x, e)x = 0}. We see that
A B. span(e) - span(e), Belspan(e) = -Id,
(46)
A B W W", Belw Id.
Next we use the facts that (0) Id(0) 0 and (0) (0) = 0. Let us define the maps
fo Fil Fol (0.r):
fi = Fi FilT (0.r)" fo.w = Fo, www.wl/wwo.. (O.r)) fL. = Fi,w (0.r))
that are -m-sohh maps Bw (0. - W. Moreover, for j 0, 1, we have fo (0,r)) @B (0,r) and (a (0,r)) = r0, Let p. = (O). Then, by (43) and we have
1
1p Il.f,w (0)l (lf f,(0)ll < 7
(47)
22

---

and
1
dist(S),(())
2 2".
for all @Bw(Orr).
This implies that
1 1 (is(()/w(0BBr.)))))...)
sup (0.r)
4 2
(48)
Then, by 43 Definition 1.2.5, the formula
48
implies that
deg(ff,w, Bww (0, p5) deg(ff, Bw (),r).p))
Moreover, as F1 A1 Be satisfies (46 and Pj E Bw (0, we have
deg(Jo: Bw (0, r). deg(Td, Bw (0, r), po) sgn(det(Td)) = 1,
and
Bw pi) = d (w, = sgn(det( Bg [l)) = -1
Moreover, by our assumptions on B, the discretized maps F,w: W W. 0, are C²- diffeomorphism. Hence, the inverse image (,w((1p))) coincides with 1,, (that in fact is the set {0}). This and the above show that for any R > we have
containing only one point that
deg(F,,,, R),p,)
deg(F,,w, Bw (w, r), deg(Saw, Bw (0, r),p)) deg(S,, Bw (0, r).
(49)
(50)
= (-1)².
Let now consider the maps Fi,w: W W where t E 00, 1]. that are C¹ dififccoooooi
few W + W. As + F; = A, a continuous map [0, 1] + GL(X) and A: + is
continuous functor, the map t + Fr,w w is continuous map [0, 1] + (OBw(Or));W)
Let us consider e E [0, 1] and denote = Fe, (0). As F/w : W - W a C³ diffecoomrphissm the set Fi, Bw (0, r)) is open and hence there is e > such that
Bw (pi. 5E) C Fi.((Bu r).
(51)
This implies that
distw (py, (3Bw(O (0) > 5e.
(52)
As 1 Few is continuous map [0. 1] + C(Bw (0, r); W), there is 8 > 0 such that when
8 < then
[[)vv(z) - (xx <e.
(53)
zeVn (0.r)
In particular, this implies that
F (0) (0) <c,
(54)
and thus = Ft,v (0) (0) and p; = Fr, (0) are in the ball Bw (p, 5c) and hence by (1) these points are the same topological component of by W \ Theorem 1.2.6 (5), that for t -- 8, we have
Hence, it follows from [3
deg( (Fg, Bw (0. r), pt) deg(F Fl. (w(ro)).pp)
(55)
Next, we observe that by (53 for any &,, t2 E [0, 1] satisfying It - i| < 8 and Its - < 8,
sup [F Fig. (x))lx < 2e.
(56)
(0..)
23

---

Inequalities (2)
54
and (56)
imply that for any t1, t2 E [0, 1] satisfying |t1 - < 5 and Itg - <
we have
distw (pt, W (dB (aBw(O,r)) (0)
inf wEaB (0.r)
distw (Prs (P (())
(57)
inf distw (Pty Fi, (v))
(0.
inf distw (p:. Ft, (y)) 22 -
vE8Bw
distw (pp, F2 (0Bw (0, r])) - 3e e 5e - 3e = 2e.
As t - Ft,w is a continuous map [0, 1] + C(B (0, r): W) and the formula (77 is valid, it follows
from 43. Theorem 1.2.6 (3) that for 1 and l2 satisfying 1 i < ō and 1/2 t < 6, we have
deg(Fi, W. (0, pt,) deg(Fi, Fi.: Bw (0, r), pta)
(58)
This and 55 imply that
F,,, (w(,,r).pu = deg(Fi,w: Bw (0, r), Pta)
(59)
deg(Fi,w Bw (0, r), Pi),
In particular, when i 12 satisfy |t1 8 < 6, this implies deg(Fr,,w, Bw (0, r), Pt.) As f E [0, 1] is above arbitrary, this implies that the funetion
deg(Fi.w (0,r,)),
(60)
g:t - deg( Fr F. Bw (0,(0,) r), Fo,
(61)
is continuous integer valued function on [0. and thus it is constant on the interval [0. 1]. However, by (49), g(0) not equal to g(1) -1 and hence g(l) can not constant function on the interval E [0, 1J. This contradiction shows that the required functor does not exists.
A.5 Proofs from Sec. 3.2
A.5.1 Proof of Lemma 1
Proof. Let F: X + X be strongly monotone and C'-diffeomorphism is obvious that strongly monotonicity implies that strictly monotonicity.
We first show that the strongly monotonicity implies that the coercivity. Indeed, by the strongly monotonicity we have
(F(x) F(0), 0)x 2 |
which implies that, as
||x||x - 00,
(F(),
x 2 (F(0),
4 | x > +a||=lxx + a0. ao.
|
Therefore, F + X coercive.
Let us consider the operator
Fy = PvFlv -V
From the definitions, Fy + V C and strongly monotones, and the (Fréchet) derivative DFvl. at V is given by
DFv = (DFl)o))
which is linear and continuous operator from V to V. Since Py V + V C² and strongly monotone, it is hemicontinuous, strictly monotones, and coercive. By the Minty-Browar theorem
10 Theorem 9.14-1], Fy V + is bijective, and then its inverse F7" V V exists.
Next, let v E As Fv V - V is strongly monotones, we estimate that for all h E EV,
Fv(u+ + eh) - F(x) (x + eh) -2 )x - (F(x + eh) - F(x), (x + eh) x)x
[1 + ch) 211 =
24

---

Taking E - +0, we have that
(DFv (.))).. (b), 2 |1113
Therefore, DFvle V + V is injective for all V. Then, it bijective because V is now finite dimensional. By the inverse function theorem, the inverse FV V V is C¹.
A.5.2 Additional examples
Let us consider the elliptic (but not stongly elliptic) problem
d
P.u dt
sign(t -) u(t) =(t), te [o,
(62)
with the boundary conditions
(0)) 0, dt u(t) =0
(63)
f=1
The FEM (Finite Element Method) matrix corresponding to this problem is [p (+)]i where
(s)(s) = J. sign(t -) at x²(1)
dt xx j. 1.......,m
and X, ( are a complete basis of the Hilbert space x., = {o H (0, 1): v(0) 0} that correspond to the mixed Dirichlet and Neumann boundary conditions. Note that in the FEM approximation we use only a finite subset of the complete basis of the Hilbert space. Note for u in the canonical domain of the above problem that makes the operator appearing in the above equation selfadjoint (the Friedrichs extension), that is, for
d u e (0, sign(t s) vt) E 10, v(0) = 0. d dt
-
u()))==1 =0}
dt
we have
P.u Pu==-Dy -.. u.
where A, is the multiplication with the function sign(t - s). see 37 and D and D are the derivative operators D. 4u having the different domains
D(D(2)) (1)) HD, (0, 1) == (u H (0, 1): v(0) 0},
(64)
D(D(2) (2) H ( (0, 1) fve H (0. v(1) = 0).
(65)
Observe that D and 2(20 have the inverse operators
(D!")~+0(1) = S o(t')rt
(66)
= S: o((')dt'
(67)
that map (D()))1 D(D The eigenvalues of the matrix s -+ (s) change from positive values to negative values when s moves from to 1. Thus, for some value a E (0. 1)
the matrix [s)a(s)]; has a zero eigenvalue and is invertible even though all maps D( (1)))
and A, : (0, 1) + (0, 1) are invertible. In particular, there are no finite FEM basis in where the Galerkin discretizations of all operators P., s E [0, 1] are invertible.
Let us consider tions of the operator A, (0, 1) - (02 1), we see that the Galerkin matrix corresponding to A, (.2 1) 1,2 (0, is (aza (8) (5)] A= where
simpler example: Similarly the above, if we consider the Galerkin discretiza-
asx(s) (s) = / sign(t - (t)) (t(d)t j.k = 1,..., m,
and v (t). = 1,2, are a complete basis of the Hilbert space (0, 1). Again, we see that the eigenvalues of the matrix + [s), change from positive values to negative values when moves from 0 to 1. Thus, for some value (0, the matrix [a)ı<())]
has and is no invertible even though all maps A, T,2(0, 1) + T.2 1) are invertible. Thus there are no finite basis in where the Galerkin discretizations of all operators A., [0. are invertible.
eigenvalue
25

---

A.6
Proofs from Sec. 3.4
A.6.1 Proof of Lemma
Proof. From the assumption, holds that
1
DF12 I xxx-x | 1D S
Then, by the mean value theorem, there is 0 < such that
0
(F(x1) - (1 - x22)x
( + t(02 - -1))==1))--)))))))))) (x22) x22)x
01
8 DD
(22-1),a1 11-)),11-0 -x2)x 2 || - x21
ot
2
These imply that F; P X is strongly monotone.
A.6.2 Proof of Lemma 3
Proof. Observe that strongly monotone neural operators coercive, that is,
lim (F(u), [Jl]ix 700
u xx lim
||u|x (F(u) F(0), 0)x 2o.
(68)
|
||
and therefore F: X X is surjective by Browder-Minty theorem 28 Thm. 2.1], also [4 considerations for neural operators. Moreover, the derivatives DF are linear strongly monotone operators for all X, and therefore injective. Observe that the derivative linear operator of the form
DF I T2 DG T1 x X,
and as Tj and T2 are compact and DG is bounded we see that D is a Fredholm operator of index zero. Observe that
F(r + hv) F(x) (x + hv) x)x x 2 al|01123
lim (DF nU o)x T 4-+0
where h E R and v X, and hence DF + is an injective linear operator. As DF a is Fredholm operator of index zero, this implies that the derivative DF X - X is a bijection. As Hilbert space X can be identified with dual space and the operator F: X - X continuous and hence hemi-continuous, it follows from (28), Theorem 3.1, that the map F: X X is homeomorphism. As it is C -smooth and its derivative is bijective operator DF X, it follows that F; X + X is -moshh diffcomorphism.
+ at all
A.6.3 Proof of Theorem
Proof. To show the well-definedness of the discretization functor Aii. we need to show that, for each strongly monotone C¹ -u-nnccioon F: X - X that is of the form (4) Fv Pv Fly V V still strongly monotone C' -mmooth of the form This given by Lemmal Moreover, such functions are -moooh diffeomorphisms.
To verify assumption (A), let r, e 0, and let F: X + X where Ti: X X and T2: X X are
be a strongly monotone diffeomorphisms
that is of the form F = Id + T2 G compact linear operators, and G: X + X such that G E C¹ (X). Since T2 o G o T1 is a compact such that
mapping, there dinte-dimensional subspace V c X, depending
[1(Td Pv)T:G((iz))lx Se,
sup " ==F
which implies that
sup Ze (V
- F(x))l
sup =)
[[ - (V)d-)))))( Se.
To prove the continuity in the sense of Definition
8 let > and let V E (x))). Assume that
lim 10000
sup (0.r)
|| F(x)]x O,
26

---

where F() and F are strongly monotone diffeomorphisms F; X - X that are of the form 4 Then, see that, as + oc,
FFv))lx
[IPvF()(x)) Pv Fr)((x)
sup "EVnBx (0.r)
sup =EVn
||Pr|lop|| F( v sup (0,)
| - F(x)]lx + 0.
sup =EVn
A.7 Proofs from Sec. 3.5
A.7.1 Proof of Theorem
In the proof of Theorem we prove first few auxiliary lemmas
Lemma 5. A layer of a neural operator F: - is surjective. In particular, Ff: X - X is
bilipschitz. then F: X - X is homeomorphism.
Proof. By formula (4) a layer of neural operator F; X + X is the form F(u) = u (Ti(TTu)
where Ti X X and T2 X X compact linear operators and G X - X is function in Let
C² (X). Let cu, c1 > 0 be such that [|]]| S c and Lipx-x((OC) S C1-
Letp E X and Ro | + l|pllx Then Kp X X, defined by the formula
Ky(x)= = GoTT( + +P, x xEX,
a compact non-linear operator, see 43 Definition 2.1.11].
When satisfies E (Td (OB(O, Ro)), let deg(Td - Kn, B(0, Ro), = deg(Td Kp. B(0, Ro), 0) be the infinite dimensional (Leray-Schauder) degree of the operator Td Kg X + X in the set B(, Ro) with respect to the point p. see [43 Definition 2.2.3]. Let Kpt : X - X be the non-linear compact operators that depend on the parameter E [0, 1]. obtained by multiplying (z) by the number i, that is,
Kxt(x) tKp(a), I X.X.
(69)
As | [xa(())lx & [ + 1lllx < Ro for all f E [0, 1] and x E X, we see that
(x)( + for z E 0Bx (Ro).
(70)
Then by the homotopy invariance of the Leray-Schauder degree, see 43 Theorem 2.2.4(3)), the function
d()) = deg(Id Kp,t, ((), LE [0,
(71)
a constant function. Moreover, by [43 Theorem 2.2.4(1)),
deg(Id [d - K2. B 00, Ro), = deg(1 Kp.1 ((0, Ro). d(1)
d(0) deg (0, Ro), 0) 1.
By 43 Theorem 2.2.4(2)), this implies that the equation
x= ==)) equivalently, T2 G
has solution (Ro) As p X above arbitrary, this implies that F= Id I X + X is surjection.
Finally, if F: X X is bilipschitz, it is bijection and its inverse function is Lipschitz function, and thus F: +
homeomorphism.
The next lemma shows the existence of a (fiite-ddimensional) orthogonal subspace so that for each compact operator, projection onto the subspace is perturbation of the identity under either pre or post composition.
27

---

Lemma 6. Let T1. T2 X + X be compact operators and h >> 0. There is a finite dimensional space W X such that for the orthogonal projector Pw X + it holds that
[Ti
< <h,
(72)
[[ Pw Pw))[2llx-+ < h;
(73)
I[(Id Pw Pw) Pw)llx-x < h,
(74)
[I(d 1old-ww'h (W)dddd) Pww)lx(dd-Pwlllx-xx < h.
(75)
Proof. We use the singular value decomposition of compact operators: We can write
T& Σ we, (z, 2h 12..)
where we.p and Φe.p are orthogonal families in X and we,p 2 0 satisfy we,p+1 S We.p and Wepp + 0 as p - oc. For all we can choose such that we,p < h when p 2 P. Then
P
[Tg
Σ p=1
wept we vt kh
If
V Vp = spantvi.,p Dep; E = 1,2,, P),
(76)
we see that W C X is linear subspace such that V W then
(t.))))) Pw Σ
wiplys Vepd)mm
p=1
p=1
P Pw o
Σ
weplis vep))111
p=1
p=1
and
P Pw o
P Pw =
Σ
Σ we V1
and thus
Th To o Pw 1x < <h, [Te - Pw Tellx-xx <h,
and moreover,
[ - Pw o T/ Pwllx-x< chh
Let
FW Id Pw Ta GoTioH, X + X.
(77)
Observe that for W and v - we have
W (w + = F.W (w) [W(w) E W,
(78)
and
FW (WW W, FW W + W FW Iw Idw
(79)
(80)
This means that FW = Idwa ① FW lw where FW ww W + W is a function which maps the finite dimensional vector space W to itself.
The lemma below shows that given an F and 0, we may perturb by a Lipschitz term B so becomes the operator W.
28

---

Lemma 7. For any e >> 0, tthre is f.nite dimensional space W C X such that Lip x- x FFFFFFF <
e and for any ball Bx [R) C X, R > 0, the maps F: Bx ( Rt) - X and FW Bx (R) - X satisfy
||F -F'--WW 1 (Bx (f) < [1 R)e.
Proof. Let us choose a finite dimensional space W c X so that Lemma is valid with
4(1 + +
[1
The right hand side (72 is chosen so that
Lipx-x pW, Lip(Pw o T2 o GoTi
- -
Lip(Pw To G Pw Pw o T2 o
+Lip(Pw o oTo0GoTi6 GoTi
x
Pw) + II(I -ww.T//k--nLLpp/Xxx>>>//.Xl...-.
8
and
|F 11" LL (B(ORR)
sup zEB(O,R)
|Pw o G T - o oo (z)lxx
< sup B),
|Pw o T. + Go7. (2) - Pw o T2 G Ti (x)llx
sup zE
|P T2 T2 o (z))lx
Tilx x - R
+ +|[(I Pw hwwp
1 RR 1)c.
Lemma 8. For any > O, there are finite dimensional space W C X and (possibly non-linear)
functions B : X + X and B X + X such that
Lip(B) <,, Lip(B) <e.
(81)
and
FW = o F = (Id F : + X. Moreover, B: X + X is compact non -in-iinerr operator of the form
(82)
(83)
B = Pw o Fi Pw + T2 o F2 Pw + Pw o F3 o Ti + T2 F4 Ti,
(84)
where F1, F2, F3, F4 X X are functions in (x) and B X - X is of the same form. In addition, the operator 1d + B X X and 1d + B X X are layers of neural operators.
Proof. Below, let Co be such that
[[(()) Fly 2 Colla vllx, 2,
(85)
and let W be the space in Lemma with
4(1 + [|]+))))))))))) + 1 + ||
By Lemma 5. F(x) (2G(Ti is by an invertible function F X + X. Thus we can define
a non-linear operator
FW o (F)
(86)
=(Id Pw o T2 o G (1C
=Id + (Pw o Ta c G T Pw - T, o ( o
29

---

Let
R
Pw o - -2oGo]
(87)
= PwychoGoTiohx-
(88)
P
(89)
Then for all xe, y E X
[R(x)) R(v)lx
< II(ld
yllx
(IT -w)pxx--l--xxxx----->-->-------- vlxx
<
2C, c||= y||x.
and thus,
R o (Td+T:oGoT)) 72 (
2 _ yl1x-
This implies that FW, X + X satisfies for all y E X
II(EW (F)-1-Id)(a)) [x)(x) (FW o (F)-1- [1) zellz= vllx.
1
and thus
1 20]11== yllx.
[I (F) (FW o (F)- I 5(1+
(90)
and
Lip(FW, : - x) $ (1+ 1 ( Lip(F: x x).
(91)
Moreover, for all x. y E
1
|| - Fw (u)llx 2 (Co e)(lx= -ylx 2 マ Collor ylix
(92)
Let co O be such that [|G]|z=(x) Co. Let Ro 0. As 7 G) [2|||o, finite dimensional degree theory, see using 43 Theorem 1.2.6], as above that
(w, Ro) C F (Bw R1)).
(93)
when R1 > Ro [22lcco > Ro + |PW Ta G o Ti Px)l As Ro 0 above is arbitrary, formula (93) implies that FW X + X is surjective. Thus, we have shown that FW X X is
bijective bilipschitz map.
Similarly to the above, by replacing 88 for x. y E X
by 922 and changing the roles of FW and F, we that
|R (I + Pw o T2 o G o T Ro B4
< <||x - vllx:
and
[[F o - Id)(z) - (F ( Id)(x)lxx S Sellx
vllx.
(94)
The above implies that
FW = (Id+ (ol+B)oP B,
(95)
F= (Id + B) o FW
(96)
30

---

where
B (F o - X + X,
B (F o (FWW)-1 -d) X X,
satisfy
Lip(B) < 2 Lip(B) <c.
(97)
Next we use that
(Id + H)-1 = (Id H) +
= Id- (Id+ o H,
so that
+ [2d+720GoT]
Id 0
Observe that by (6
B= o (F)-1 Id
(98)
=Pw o To o Go Pw - Go (2
=(P T2 o o Go Pw To,
(Pw T2 G Ti Pw
Ti (Id T2 G G TT) (T2 o
and as Pw X + X a finite rank operator and T2 : W + is a compact linear operator, we see
that B: X - X is compact non -inear operator of the form
B Pw o F1 Pw + Ta F2 Pw + Pw F3 oT + T2 F4 Ti,
where Fi, P2, F3, F.: X X are functions in (X).
Moreover, similarly to 86 we that
Fo (FFW- =(Id+ T2-+T2200 G o T (Id+ Pw T2 Pw)
(99)
=ld+ (I2oGo Pw - o o
Pw o GoT Pw) Pw))
and hence
B = Id
(100)
-T2 o
Pw o T2 Ti P (Id Pw o
=(220Go Ti Pw o
T2 o
Pw)
G of Pw o T2 G o o
(Id+ Pw
o Pw) (Pw o o Pw),
and again, as Pw: X + X is a finite rank operator and T2 W + W is a compact linear operator, we see that B X + X a compact non-linear operator of the form
B Pw a o Pw + To o F2 Pw + Pw Fg Ti + F4 TT.
where Fi, F2, Fg, F4 - X are functions in C² (X). For an infinite dimensional Hilbert space X there is linear isomorphism J X + X x X, as the cardinality of Hilbert basis of the space X is the same as the cardinality of the Hilbert basis of X x X, see [26) Theorem 3.5J. Then, by writing the isomorphism J as (xx (2),, E X x X, we see that
A F
( Pw Ti
))O),
B(x) PW T2 (J-)
o oJ)
F
that a composition of a linear compact operator X, non-linear operator X - X, and compact operator X X. Hence we see that Td + B, and similarly Id + B, are layers of neural operators.
31

---

Now we present our Proof of Theorem
Proof. Let = = x
(T)) Without loss of generality, we may assume that <6<
min( X
By 94)
[B(x) B())llx & c|| yllx,
(101)
and
((I++ + B)xx) (Id B)()), v)x 2 1K v113 =||xc v13
- -)( yllx.
(102)
which implies that
B
(103)
a strongly monotone operator. Similarly, we see that
Id+ B x x
(104)
strongly monotone operator.
Recall that Fw maps W to itself and is equal to the identity map in W and moreover FW be decomposed according to the formula 78). Thus we can write
can
Fw Idwa @ (Pw Fw Pw): W a W BW-W W ① W.
Below, let us identify with R2 to clarify notations. We denote
= Fw ww Pw Fw Pw W - W.
(105)
By 91 and 922
there are c1, co > 0 such that
ca |r yl $
- -
s c y/-
(106)
Define for <t<ı
Se(x) = (((())) -(0)) + (1(0)-
(107)
For = 0, define
fo(x) = Dflax.
(108)
where Dfly R" + R" is the derivative of the map f the point y, that is considered as linear map (or a matrix), and we denote its value at vector by Df Dflyy Then fi (1r (x) and
(f(()) - f((v)l = 1 (is(e)) - ((y)))
(109)
and as
7 coltx tyl 1 (1s(tz) f(ty)) 1
-or |tx tty.
we have
co | vl s 1f(xx) - Sc(w)) S c wl.
Below, let
Rg = lf(00).
(110)
and
R = ciri + Ro
(111)
'The above used homotopy can be replace by more explicit flow in the Lie group of diffeomorphism, see e.g.
4p
32

---

where appears in the claim of Theorem Observe that then fo(B (0,r1)) C. BR= R1). As E C IR"), we have by Taylor's series (with the remainder written in the integral form)
2 L (t s)Dflez (t-s)Dflae ds
1
f(tx) = f(0) + Dflo(tx) +
f(0) + 1Dfl0 + 2
[-² (1-s) s)Dflesste ds.
(112)
2
Thus,
f((x) = D/ a + 1, 1 [( (1 8) txn -s Lf(0).
(113)
2
Observe that all x E X.
1Dflll cr, and [(Df))) 1 "o
By 113
(((D5))))*') (i((((())))))))))
- /'
1
=r-yf
+
(1 - s)
Dfflca (Dflo) - (Dflo ds
2 00
= = - y + -
(1 (D -w) -)) ds
(Dfl))
2 0
+ bタ / (1 (Dflee - (Df (Dfcc) ds
(114)
Here, for x, y E Br (0, R2),
|D - bngll I [1 [D/|sssl|R S 61,
[IDffl) 2 S c 11311
Hence,
fi o (Dflu) Id + Q1, R" +R"
(115)
where
1
Lip (0,8.) (Qro)) S (Q..))
(c1 + (o..))))
(116)
200
Let us choose ly (0, 1) such that
2co
t
Ci + |f (0 E,
(117)
so that
(OP <E.
(118)
Below, we will denote
H (x))
(119)
Next we consider operators fe with t >> 0.
Let us next consider t2, tg t,....,, tm+1 (0, 1] such that to t1, tm+1 1, and tx+1 ty.
We have
[f((0) +lf((0) t.o.
33

---

and Lip(ft) C1. Hence,
[f(()) 1 | + Ro-
(120)
Moreover, for 2 = f(0) we have (z) = 0 and |z| Ro. and as the Lip(Se s %
(x)) S (x) - +1) s ح =0 co |x|| + (11,
(121)
so that
1x)) s 5 + c Ro.
(122)
Also, as we can write
f((x) = 1 ===(((((t (f((t)) -(0)) (0))+++(0000 = (((x)) + (t -t --))00
(123)
we have for tke tx+1 (0,1), 21,
- f(txx) - ((tt1 | }5(0) + (tu+1 fx)
1
(fen (xx)())
1x+1 f(tx+12)
1x
1
1
1 (f(++11)) -f(t2x)
( 1x+1
)f((x+1a) +
1
1
) +(4+11 f(0)
1x+1
4
tk tkt1
1
)
f(x + (f(tx+1x) - -((t2))
tk
tot1
tx
1
1
)f(()= (tut1 tx))(()
tx+1
tx
( 1k lkt11 1
1 _ 1 (f(1kt1z)- [(L==)
(11+111) + (l&+1 1k+1
t
tk+1
ts
ty tk+1 [tx+1
1
-( -1- 1 ) + -tt++1 - tx)f(00)
)f(0)
to
tu+1
'ht+1 to
tk
tk+1
1
feces
(x) (f(tx+1x)= f(tex)) + Ck.
bx
where
tx+1 tk
1
L ) + (tx+1 - tx)f(0) (124)
Ck
(tx+1
)f(0) - 1x41
tk+1
tx
tx+1 ty
1
)f(0) + fx+1
to
(tx++
f(0) + (tx41 tx)f(()
(125)
1x+11
1,4411
(tx41 txt ( tx
+1) f(().
(126)
To analyze the above, we denote
1 (f(ta41a) f(1x))
k( lx, (k+1)=
tx
( glatt Dflevt
to J.8
to [( (lx+1 - 1c) lee)Dlaetttttrrtteee (x)
As |IDDllttstter s ei [2/l// we have for all R >>> 0
1x+1 lx R.
Lip(k((ss4 44, 4+1) Bg (0, R) - Rn) S cl
tt
and
C1 |zl- tv
1kx;; te, tx+1)l S |t tel
(127)
34

---

Moreover, for 21. E (0, R),
la 1x+1
1a+1
Lx
1ffu-1 - (12) - (Jean (aa) S
1002
x1
+
c1
R|
-21l
tk
tx
(128)
and for all IR""
It tetal Sirm (x)) + (x+1)l + +Ckl
1frr- - fxx)(z)) S
(129)
is
Itu
tk+1l
1
(ci [c1120| + Ro) + |6x+1 - tul To |z| + |lx+1 - tul
+
Ro
L
4x
(130)
te++1 (201 |ar| + 3RR)-
(131)
ts
Moreover,
Sta (FT = ft + 1 - ((e) (JT
(132)
=x (Sev11 S(,) s
(133)
Let
Quaniia (x) (un ((==(()))) (Sten1 Sea) S (x).
(134)
(135)
Hence by 128 for 21, x2 E Bg (0, RO, R);
to txt1 + C1
t&t1 th I8) xr:2 x11-
Quann (122) Quan. (21))
(c1
(136)
C0
1/
By 122 and 129
hold that of all x E R"
tb+1
201
x
Ro) +3Ro)
(137)
1/
60
c0
1/ 1x+11 ||
201
2c1
xr|
+ 3) Ro).
(138)
Cu
co
Let R2 R2 and define for k function that is convex combination of the identity map Id and the map Jezti St
Ht de (x) = (1 δ(x)x ((t) (z))
(139)
= x - (f (x)) x))
(140)
x - ((x)Q(eli
(141)
where & E C8s((")) is such that d(x) = 1 for |z|| < R2 b(x) = 0 for |x| S 2Rg and [[ < 2/Rg. Then
H Id + Pietilees
where
Lipg- (Prsndd))
S [ Lip (onn22nn)(Quiii (Que)dd + 11Q (00
(c) t to
tx+1 tk
+1
R2))
(142)
60
4x
1/ 1x+1
4c1 R2 +
201
2
+
+ +3)Ro))
(143)
1x
Bo
For we have t2 t1, and thus
1t t+l
c1 (10
6
Lipgo (Pratsite) I
(10 + R2) +
1)
(144)
35

---

Then, when the condition fiz-1 (x) E Bz (0, R2) it holds, we see that
Hi o ffu- (x)
fu--l + Of (F)) (fu-1 (x))) sui- (x))
= ficms (2) + (frx (f7 (f(,, fix-1
fee f
(x) E Biz((, R2)-
Observe that when Be- (0, Ro) then (x) Bg (0, R1) BR* (0, R2). Next, denote lo = Hence, by using induction, we see that for all j = 2, ................................ m + 1 holds that
Heg,ts-1 Hez-1 Ho Sta (2) ft,(x) (r), and
H H Ho
fto (2) B (0, R2).
We recall that above we chose 1 > 0 such that
2co
(145)
a 1 Ben (0.R.)) Ri
We now choose m so that there are tg. 2.3......... .... m + satisfying Itx - -t-1l < and
1 m
41
9 (10
+ R2)
R2 ( + 1)
This means that we can choose
2
C1 (10 + R2) +
6 c1 + 1)
m >
(146)
Et
R2 cu
Then,
Lipga (P $ e.
(147)
We define the map
{ ftlk. for <tsl, Dflo, for t = 0,
a(t)
continuous map [0, 1] C (R";
The space W is a finite dimensional Euclidean space, and let GL(W) be the set of linear diffeo- morphisms of it, that is, invertible linear operators A W - W. We consider GL(W) with the topology given by the operator norm of linear maps. Then GL(W) a topological space with two path connected components those matrices which have positive determinant and those having negative determinant.
The above implies that Df can be connected in GL(R") either to a matrix B where
is either the identity map, or the diagonal matrix diag(- - 1, 1,..,,,..1 1),
(148)
with continuous path 3, GL(W), [0, with DSlo 31 and B 30-
We need to consider the path B, in GL(WW) from 31 = So = Dflo lo to matrix 30 B E O(n). There some explicit formulas in literature with relatively complicated formulas, see (41). However, let us next consider estimates using a oon-mpti but relatively explicit path. To do this we start from the PU PU-decomposition of the matrix 31 Dflo. that denote 31 PU where P diag (01,.... ..) positive matrix (given in a suitable basis) and U is an orthogonal matrix. Then, we consider the path
P,U, where E [0, 1] and
P pl-t_ exp((1 1) log( P))-
This is path from the matrix Dfoo PU to the matrix U. Observe that
P.B pti-taaappp P.
36

---

where
ph - = diag (01 1
on t2)
and when It1 - tzl I 1, by mean value theorem we have
1| x max (o)1-t2 - -1) S max j=l....
(0),
max j=l,...ff
|log |t1 t2l
jl
< (1 (1+a + 12 12)- /2/-
That is, using the norm Dflo and the norm of inverse matrix (Dflo) we can bound the length of the path P,U,E E [0, 1] (in the operator norm). After this, need consider the path from U to B = Idg x (-Id,-)) in O(W). For this, we could use the structure of the Lie group Onn). (W). By [15 Thm. 11.3.3], any matrix U in (n) can be written
tensor product of elements in (2) and possibly an operator =/d IR + R, that is, in suitable orthogonal basis matrix U E (n) is block diagonal matrix of x matrixes in ( and one two 1 1 matrices + Id, that is,
R4
+1
+1
where the matrices R1 (1(0)), Rs = -Ri are 2-by-2 rotation matrices in SO(2),
Using this, we find path form U(s), E [0, 1] either to matrix Idg x (-Idn-k),
[s(10))
0
(s
U(s)=
#1
#1
Observe that due the block diagonal form of this matrix, the operator norm satisfies
[[U(s2) - (s1)) I
[[ Ry j=1.2..,
C.ls2 - 511.
where Cs is an absolute constant (that does not depend on the dimension n)). We obtain the path + 3, by concatenating the paths from PU to in GL(n) and from U to in SO(n). The length of the obtained path 3 in the operator norm metric can be estimated and is bounded C(1 + C1 + 2.1y plus a constant. Moreover, the path s - 3g can be decomposed to a product of (C(1 + ci + c 12 + C) matrices the form Id + B, where |B S.
Summarising the above analysis, we see that J can bounded by
1 2 2 (10 (10- R2) + J
6 a + 1)Ro) + CC1 ta + -11---1 + Ce-l.
(149)
1 co
R2 co
We recall that here
20o
t
(150)
(10) and and co and are the bi-Lipschitz constants of ci
radii Ro and Ri are given in formulas f. see (166).
Note that C1, C' > 0 are constants independent of while Ro. R1, R2 > 0 depends on [f))) We
see that by f = Pw Flw.
[f(0]) < [[F()]lxx
37

---

thus, the upper bounds of Ro. R1, R2 are independent of c. By choosing t as t 01 17 (2 R, E, we furthermore estimate that
J S (l (o.)) Here, if we assume that F E (X, we see that
(151)
(1 & [Fllca(w:x) S (ect:wx)xFFFFFllalFa |Fllc((xxx) <0,
Thus, we obtain
J 0
Now we can finish the proof of Theorem
Denote ft fe ) Idw He, He @ Idw
3, B, @ Idy- and Ao B + Idw-. Using these notations, write F(x)
O... F (F (EW)-1) Him HHmmmm-
H, o (3r 3-m11 o (3am B-1 00... (3e, o 3/ Ag:
(152)
where ty and 8, are chosen so that 0 = to < ti <... Im < sm+1 = and 0 = 80 < 81 < ... 8m
8m = 1 and that the Lipschitz constant of the maps Histids Id and Bag 9 3-1 that c.
9
Td are less
Moreover, recall that
FW Id Pw T2 Go -
Observe that by writing x E X in the form x so II, where To = (1 - Pw)x and 01 = Pwæ, we see that
F (wo + xI)
+I+ Pw o T2 GoT Pw) +x1)
To + (T Pw T2 G o Pw)
(I Pw)x Pw (T+ Pw T2 G TT (w)))
(I Pw)x + Pw (FW (Pwx)).
(153)
Hence, (FW)-1; > X can be written
(F I- Pw + Pw (F Pw
I+ Pw (-I+ (FW)-1) Pw,
(154)
and thus (1 is a layer of a neural operator by definition. Similarly, as
f=
F F=
Idx + Pw (Fw Idw) Pw X + X.
and
f((x) 1 (f(tz)) ((()))))(())+t))) +t((0)
Id Pw (w + (f(tPw))- f(0)+
(155)
for 0 < < 1, and we see as above that ft and f. are neural operators. Similarly, we see that 3 and 1 are neural operators. Hence, all factors in the product [522 are (strictly monotone) neural operators.
38

---

A.7.2 Proof of Theorem 5
Theorem 6 (Theorem 5 in the main text). Let X be a separable Hilbert space, and let 4 ffntnet be an orthonormal basis in X. Let ó E (0.1). 1). and let > O, and let F: X + X be a layer
of a bilipschitz neural operator, as in Def. 3 Then, for any E (0,1), there are T,N E N and
G E RT,,Ns,,R... (X) that has the form
G =Idxxx + DN o NNT Ex) ... (Idx DN o N Ex),
such that
Go se
sup zx (R.
where A X + X is linear invertible map that is either the identity map reflection operator I + x 2xx x)xe with some unit vector e E X. Moreover, GoA Bx (0, R. - G (O, invertible, and there exists R' such that
A(Bx (0, R)) C (0,(0,
and denoting by
To R'), :-: B(0,R'+00) TT B(OTT) T7+1:= := B(0, (T(++)))))
and
Ko (O, Ki = (ldx (ddx=DnoNNiiENN
K2 = (Idx + D Ngo
Kr (Idx + DN NNr o RT Go A( (0, R)),
we have that for each 0.I,.
k.
and the mapping Go Al Bx R) + Go A(Bx(0, R)) is bijective and its inverse is given by
(Go A - = =A-
where Φ G. A( R)) + A(Bx (0, R)) is defined by
I,kR, I--1lR TTlR
(156)
where L, Tí + T141
is defined by
Le(v) = lim कi (0) +
where the Pe+1 X P. + T4+1 T, defined by
(1 (y ONONN Ex(x),)).
where a (x. x and e, (1) (0,y)-
Remark 2. Note that, the proof, we have used the approximation result [22] Theorem 4.3) of Sobolev functions by neural networks with ReLU function z + ma x), which is not differentiable at O. Alternatively we can use approximation result [1] Theorem 4.1] by neural networks with ReCU function x max 33 which are continuously differentiable. Then, each block Idx + Dx N N, EN in a obtained approximator G is C and strongly monotone on ball I. = B(0, R' + that is, it holds that there is an 0 such that
((Tdx + DN NN, o Ex)x,,), 2 all vll',, k, E Tr
which implies that, by the same argument as Lemmal
(Idx + Dx NN, Ex) r, - (Idx + DN o NN, ox).),
a C -ddiffeomorphiss.
Remark 3. From the proof. we can show that, for each I,.,,
L,(K,) C Kr-1-
Then, the mapping & defined in (156) is well-defined.
39

---

Proof. 6, E (0, 1). and let F: X + X be a layer of a neural operator defined in (4. Assume that F bilipschitz, that is, satisfies (55 and
[F(x) - F Fly)lx I C ||
yllx: for all x, E X.
(157)
By Theorem there is T E N such that F can represented in the form
F(x) (Idx Hr) (Tdx Hr-1) (Tdx H1) A for (x R),
where A: + X is a linear invertible map that is either the identity map or reflection operator - - 2(z xe with some unit vector X, and H, X X are global Lipschitz maps satisfying
Lipx + (H)) - 8/2.
(158)
for I. Moreover, the operator H, X - X a compact mapping. We choose R' such that A(Bx (0, Rt) c (0, R²).
We denote by
Ko A(Bx(O, R)),
K1:= (ldx (K)(
Kr = (Idx (TT)(KT)).. PT+1 = B Rt' + (T+ 1)6).
To (0, R²),
:= B(0, R' + 6),
We choose R such that for all t T +1
Ew C. Bgx R).
Since H, X + X is a compact mapping, for large enough N E N, we have that for all t = 1.... T.
sup (R)
[[H((x) Pvv 11, PVw (x) 2T + 8)
(159)
Note that
PVx o H4 o Pvx DN HN, EN,
where HN. : N + N by
HN == EN H. DN-
From 158 we have
HN E WL. (0. R); R
By approximation by ReLU neural networks in Sobolev spaces (see e.g., 122 Theorem 4.3)). there is a ReLU neural network N R + RN, NN, E RI, RaLU (X) such that
|HN NN (B,x (0,3) I min
(160)
2T 8)
Then, we estimate that by 159 and 160
sup (x) - D' NN. Ex(x)]xx S
(161)
T(1 00T
ac K
Also, we estimate that by (18 and 160
Lip naN (0, (0. (o. (NN)
(HN, < Lip (0. (0. (HN,)) + Lip (0, (NN - NN)
< Lipx (He) + ||Ā - NN S ó. (162)
We denote G X + X by
G - (T-Dy o NN FN) (T- DN NNT FN) c o... (T- D2 NN, Ex),
40

---

which belong to Rr.N.ppReLU (X). Then, by 161 and (162) we estimate that for all x E (0 R),
[F((z) - Go Az)(xx
Σ (1 ONONN, EN)) (O - o A(x)
1S1ST
- -OT - ONONN, o EN)) (Oh Ha)) A(x)lxx
Σ 1S1ST hettl
II
1+ LiP (O.R)
sup yEK.-1
II(I ( -I-DN NN, EN)(u)lxx
T
Σ II (1+8)
sup
[H+(v) DN o NN, o Ex(U)lxx
ISIST h-tt1
yEKr-1
Σ (L+
T( S.
ISiST
Next, 160 we see that
DN NN1 EN)lr. : To +1, (I-DN NN₂ Ex)lr, :T T2,
DN o NNT o Ex)le -1--1- +Tr.
and
Glr.- = -b- bbrb.
which means that Glr, maps from To to TT. For l,.., T, we define L, P + Pe+1 by
(v) = x", VET.
where Te+1 a unique fixed point of h4,y [441 - Ti41, where
(x) := + DN NN, o Ex(x), that x² E Te+1 is a unique solution of he (z) =
y= (1 DN o NN, o (x)()), rE P
Indeed, there is unique solution because we have for 22 E Pe41, from (162)
(011) (x2))l ||D o NN, o Ex(x1) - Dx o NN, o Ex(x2)lxx s 8||m: x2 x2lxx,
which implies that he, T2+1 T441 is contraction map. Then, have for r
t= 1, 1,...,,7 T,
I (I DN NNr (x))
(163)
Here, the solution x" T(+1 is given by
lim Ens n0000
where
To = 0, En+1 = + DN o NN, En(x.).
We define sp T141 T, T++1 T by
(1 y) = (y + DN o NN, (x) y).
Let Le and ei (y) (0, y) where x x X + X and Then.
X X X are linear maps.
(, = x² lim (0
We define &p 1' + To by
t p :p:= Pro Li Pr, La .... Pry LT,
where Pr. X + X is the projection onto the convex set P, B(0, R' + 18). Then, by 1633), we have for x. Bx (0, R')
Dp G(x)
(164)
41

---

Therefore, &p is the left-inverse of G, that is, GlA(Bx(O. A(Bx(0, + X is injective, and (0. A( (Bx (0, R)) - Go (0, R)) is bijective, and the inverse (xx,
Go A(Bx R)) + A(Bx (0, R)) is given by
(G) (Glco((Bx((0m)))
where + Go A(Bx (0, R)) A(Bx (0, R)) is defined by := (Bx,0.((x(00R1)) and has the form
Φ [1 L
LoT Aa (0,
(1Bxx10 (0, R)) also bijective, and the inverse is given by Therefore, Go A: (0, R) - o A(xx
(G A (x()))) = A o
Note that G AO, R)) C CT.
A.8 Production of Quantitative Universal Approximation Estimates
Here, we show how framework may be used 'out of the box' to produce quantitative approximation results for discrctization of neural operators.
Let X be separable Hilbert space. We will consider a Hilbert space X, endowed with its norm topology. Recall that (x(X) C S(X) a partially ordered lattice of finite dimensional subspaces of X. Next, we consider quantified discretization of continuous, possibly non-linear function F: X + X, that is, how the discretizations Fv: V - V can chosen (using e.g. neural networks Fv having given architecture for each subspace V C X) so that the obtained discretization operator A has the explicitly given error bounds εv
Using quantitative approximation results for neural networks in Rd see e.g. 57 or (23 one obtains quantitative results for neural operators. An example of such result is given below.
Proposition 5. Let r
and F: (0, + X be non-linear function satisfying F' Let ev > be numbers in-
r); Lip(B x r), X), in n = or or E C' (B,()) dexed by the linear subspaces V C X such that εv + 0 as V - X. When d = dim(V), the space V is identified with Rd using an isometric isomorphism Jv: V - Rd. Then there is a forwardd forward neu- functions with at most C(n, d) logz((1+r)) /Ev) layers and C(n, d)) logg((1 + r)" /εv) non-zero elements in the weight matrices such that ANN FF (Ev)))s.(X)) where Fv = J .v. Jv: V + V, is an E-approximation operation in the ball Bx (0,
if n
ral network Fv.0 : [ + Rd with
Proof. Let Lip(F: (0,) - X) S in 1, or [IF(l=((x((00)))) S M, if n 2 2. Let V X be a linear subspace of dimension and Ev O. Let us some orthogonal basis of V to identify V and Rd and denote the identifying isomorphism by Rd + V. The function
F: Idd + Rd, given by F J-1 1 o Py a F o satisfies Lip(F: B (0, r)) - x) S M if 1, or [IFlIc-(B, S M if 2 2. Then, by applying [57] Theorem 1] or [23] we see that there exists a feed forward neural network Fv. : Rdd + Rd with Rel.U-activation functions with at most C(n,d) d) logz(( (11 + r)" /ev) layers and C(n,d)ey ddnn logg weight matrices such that
non-zero elements in the
[F)o(()) -F(x)lee= S MEv.
(165)
Due to Def. this yields the claim.
B Invertible residual network on separable Hilbert spaces
In this section, we consider the approximation of diffcomorphisms by globally invertible residual networks Hilbert spaces. From the viewpoint of no-go theorem, as the class of diffcomorphisms is too large, we focus on the class of strongly monotone C² ddiffecmmorphisss with compact support)
We denote the support of F' - by supp(F) {x e F(x) # z).
42

---

We first obverse a following similar result with Theorem that strongly monotone C' d'fiifcomooppiis F: X + X with compact support has the property that any linear discretizations Py.Fvv are still strongly monotone C² diiffeeomorphism with compact support. The proof can be given by the same mapping.
argument in Theorem 3 because F has the form of F = Id + B where B == F Id is a compact
Proposition 6. Let Atia be the discretization functor that maps F to Fv for each finite subspace
V X. Let Deme and Bsme be categories where F X + X and Fv V - V are strongly monotone C d-diffeomorphism with compact support. Then, the functor Alin Demc + Bemo satisfies assumption (A), and it is continuous in the sense of Definition K
Under the same setting and notations in Section 3.5 we define the class of invertible residual networks the separable Hilbert space by, for T, E N and E (0,
Riny T.N. 5. (X) = {GE GE RT,N G= -T- + DN o NN, o Ex),
Lipx-x o N EN) 5 (6(=
(166)
which is subset of R T. (X) defined in
10). The smallness of Lipschitz constants
Lipx-x( (DN (DN NN, EN) implies that Ring &, (X) is included in the class of homeomor-
phisms. The following lemma shows this fact.
Lemma 9. Let ò E 10, and les F E Rin TNN. then F X X is homeomorphism. Moreover, ifa R is C22 then, F X + X is C ("-diffeomorphism.
(X). Ifa R R is Lipschitz continuous,
Proof. Assume that Lipschitz continuous. Let G Rine p. (X), that is,
= (Idx + DN o NNT o Ex) o...o (Idx + DN o NN1 o Ex), where Lipx-, (DN o NN₂ EN) S It enough to show that for 1....
Idx + DN NN: Fxx X X,
is homeomorphism. Indeed, we have for u, X
((Idx - DN o NN, o EN)(u) (Idx + DN o NN. Ex)(v), u- v)x = =Ilu-oll + (DN o NN. Ex(u) D NN, Ex(e), u u) 2 (1 ( 8) u
(167)
that is, Idx + DN o NN. N, EN X X strongly monotone. By the same argument in Lemma we can show that (Idx + DN o NN, o Ex): :x - X is coercive. By the Minty-Browder theorem 10 Theorem 9. 1.11-,], (Idx + DN o NN, Ex) X + X bijective, and then its inverse exists. Denoting by He = Idx + DN o NN, o EN. we see that by substituting u = H-*(w) and H (v) for 167
1
(w) - H, (w)l - 1- (H H, (u) H, H (()))...'00 - (x)x
(w)
1 11 lu - x||x| ( H. (v)llx
< 1-8
which means that its inverse is continuous. Therefore, (Idx + DN NN, Ex) X X homeomorphism.
For the second statement, we assume that is C². By the same argument in Lemma we can show that the derivative DHt X + X at E X is given by
DHr|u = Ix + Dx D(N EN,
and it is injective. Since Dx o D(NN,)) (x) o Ex x X is a finite dimensional linear operator, compact operator. Then by the Fredholm theorem, DHr + X bijective. By the inverse function theorem, the inverse H, + X C² which implies that H. X - X is di-ifffcooorphism.
43

---

In what follow, we employ d as the GroupSort activation having a group size of (see B Section 4]). As sort activation I-Lipschitz, from Lemma 9 Ring 1.,66 (X) subset of the class of homeomorphisms. We finally show that, in this case, Riny N.,. (X) is an universal approximator for the class of the strongly monotone diffeomorphisms with compact support.
Theorem 7. Let R O, and let F: X - X be strongly monotone C' difffeommorphis with compact support, and let a be the GroupSort activation having group size 2. 2. Then, for any orthonormal basis (Pn)neN 6X,0E E (0,1), and (0, 1), there are T, N E N, and G E Rip p.N. (X) such that
sup
[[F(u) - G(u)llx c.
Moreover, the inverse G-1 X + X of G E Riny &e (X) is given by
G in ...... ---1 LT.
(168)
where Tu + X is defined by
(Oi=i²)) oe)(y).
L,y := lim n
where कe x Xx X is defined by
é1 y) (y DN o NN Ex(x),
where Ti xI(xz)) = x and (1 = (0,
Proof. We denote by Diff' (X) the class of C² c'-diffeomorphiisss between X, and Diff em (X) the class of strongly monotone C -'ddiffcomorphhisms between X. We also denote the support of F Diff by supp(F) : {xr E X F(x)= + æ). We say that E Diff (.. if E Diff sm (X) has compact support.
Let F E Diff that
c. (X). We define Fi X + by Fi == Idx + Pvx - Idx) and we see
F = Idx + Pvw Idxx Pv + PVv = Pv D2 EN.
We show that for large N E
sup B (0,
|I - (i)llxx S
sup T R))
PP (Idx ( Se,
(169)
as Idx- - F: X + X is compact mapping. By the same argument in Lemma | we can show that FN == FDx E Diff (R. and DF is positive definite matrix, which is connected Idns. By the similar argument in the proof of Theorem 4 see (105) (152). we can construct
continuous paths f [0, 1] + Diff sm.a (RN) and B [0, 1] GL(R with fo DFN la Fx. 30 Idga, and B₁ DF lo such that
Fv = (fi o sT (Jem o f
(Fz, t) (Bi 3-ml) (Bam 0 o... (3,, 0,05 )
(170)
where ty and 8y are chosen so that 0 = to < t1 <... tm < sm+11 = 1 and 0 = < < 8m <
8m+1 = 1 and that the Lipschitz constant of the maps fe, f - IdgN and Brg a 3 j-1 are less that 8.
Idgx
Then, there is T E N such that Fx has the form
Fx = (Idax Hr) o (IdgN HT o (IdgN H1).
where for each t = 1....,, T, Then, remarking that F.y D Idga and D2 F.y- = Pvy. we see that
LipRN -RN (Hi) S 5.
F = Pvs + (Jdg - -HL) .... (Idgx - HN) = (Idx DN o HT Ex) o .... (Idx DN H o Ex).
44

---

For each = 1.... T. by using B Theorem and Observation), 8- 5-Lipschitz function H, R - R can be approximated by neural network NN, IR + IR with GroupSort activation having group size of 2 in I -nom on any compact set, and NN, R - TR is ò-L.ipschizz continuous.
We denoting by
G -=dxIdx~ DN NNT Ex) o ... (Idx DN NN1 EN) RT, N. (X).
Note that Lipx ( NN Ex) 8. Then, show that, by similar way in the first half of proof of Theorem
sup [u) G(u)llx sc a (),
(171)
With (169) and
we obtain that
sup x (D.
[((u) - G &
sup (0.
sup Bx
$ 2c.
The representation of inverse G-1 can be given by the same argument in the proof of Theorem
Similarly in Corollary [I Theorem 7 and Lemmas 10 and 11 have the following corollary.
Corollary Under the same setting and assumptions with Corollary let RN N... (L2 be the class of invertible residual neural operators defined Then, the statement replacing with L²²D; and G E R.,.N.pss (X) with G' E RNOONN (.M. in Theorem holds.
in 180
C Neural operators
C.1 Examples of generalized neural operators
In the main text, defined generalized neural operators Hilbert spaces. Here, give several examples to show that they are extensions of classical neural operators [33 30).
Example 1. Let X be a Hilbert space. Lel Ge X > X classical neural operator having the form
G= GWT, + KT,) ( + o... oWi
where We X - X are linear bounded operators corresponding to the local term and Ke X + X
are compact operators corresponding to the non-local term (e.g., integral operators having smooth kernel smooth basis). We assume that activation function C Then, we have Gt C² (X;;'' X). Let Tj = Küfft XX + X and Tr = Kproj X + X be compact linear operators, which corresponds lifting and projection, respectively. We denoting by
:; 1 Ti G T2,
which is one block of classical neural operators with skip-comnections We also denote by At = Id Fr, o.. Fi corresponds classical
and a Id. Then, the generalized neural operator neural operators with skip-connections. In this paper, the skip-connection (ie., the structure of the identity plus some compact mapping) so important to preserve bijectivity, discussed in Section Example 2.
We show that classical neural operators for example
Felus:
(W2 K2) (o(Wiu (u)))
see 30 33 can be written in the form of the generalized neural operators that we consider of the form
H(x) = A a a FŞ Ag-1 o FR-10... A o F1 Au.
where Ag : X + X are linear operators which may not be bijective and F; = Id + T3,1 o T3.2 :
X X.
We could start with the observations that for an infinite dimensional Hilbert space there linear isomorphism Jmn. X - X where X = X .... .. X. The reason for this is that the cardinality of Hilbert basis of the space X the same as the cardinality of the Hilbert basis of x (26) Theorem 3.5].
45

---

First we observe that we can write an operator
H(x) As o oōg o Fs As- oōk- Fs-10... An o F As.
where Â5 X + X are linear operators which may not be bijective and Fj Id T₃ Go GoOT. T32 x + x and ok = + X is 0 D2.......... ,ay F4tt1..... Dmz) = (#1, 12....,. o (Ii++1), ...,, ((nn,) in the form
H(x) = (Jime oAgoJan. Jo(JI, oōoRroJan. (o A 1.1 oo...oooorra AgoJmo..))
where e.g. (J1, Aş In X + X and (Jim, o oā FR Jng.1) X - X. This means that our formalism we can replace e.g. operators Aş by matrices of operators Ax-
Next we go to the second step of the construction:
As example, let us consider the classical neural operator
Feios u (W2 + K2) o (Wiu Ki (i))),
where W1 and W2 are invertible matrices and K₃ and K2 are compact operators, can be wrillen as an generalized neural operator
u u L4 P3 L2 o oo L1 Folu),
where Fo X + X is layer of neural operator
Fo(v) = v + k,
and L1 :X - X is the invertible linear operator
Lι(u) Win,
and (u))= = =(u) and Lg x + X X is a non non-invertible linear operator
Lz(w) = (w, w),
and and F3 X X X X is a layer neural operator
Fs(wi, W2) (wi, 12 W5¹ K2(W2)),
and LA X x X + X is the non-invertible linear operator
La(ui, (2) = W2(u2 u1).
Finally, we point out that non-invertible activation function o X + X satisfies Lip(o) S X can be written as
- L Li(u).
where
L u + (u, u),
and â is an invertible function
ā u2) (wi, 22u2 + a (12)).
and
Li : (wi, w2) + w2 - 2X
Note that equation 2Au (u) w can be written as a fixed point equation
== gw = 12 (22)
where Lip(gw) <. 1/2. Hence, 9o : X + X is contraction and the equation = (w) has unique solution for all w by Banach fixed point theorem. Hence, u - 2xa + (u)) is invertible.
Example 3. Let D C. IRd be domain, and let L²(D;RR RR) be the real-valued L 22-function space on D. and yp = {pm) T.: R) be an orthonormal basis T.2 T,2 TR). We consider the case when X (D: (D) and A₁- W() where W() E hRhxh are invertible matrices, and F, = Id + Pvy KV Pvg 11) (corresponding to T Pvg. T2 Pvy W()-1, G kK
46

---

where VN = span(Pnlng and K LD2(D: I2(;)R)* is finite runk operator defined by
(z(w)(z) E (u (olca))))))))) x D,
PASN
Then, H 1,2 (2; Rh can be written by
= (W + K ((k) o (W(k-1) + k(b-))) .00.)o............. (W + ooo (W + k(
which coincides with classical neural operators (assuming that all local weight matrices are invertible) used in e.g. 133). See Definition Das well.
Example 4. Let consider an example of an example of an generalized neural operator which obtained by composition of non-linear integral operators and smooth activation functions. Let D C Rd be be bounded domain with smooth boundary, We consider the case when X HDD = (D; Ryye are Sobolev spuces, and A; = W where W E hRg are invertible
matrices, and F3 Td + iH o KtB) 1 = W()-1 (corresponding to Ti = ig
T2 = i (WG) G = K()) where K() L (D; 1))* - H (D) is non-linear integral operator
(1)())))) = f 800 v, U, u(v))u(y)dy,
D,
where kernel satisfies k0) E CD x D R": : h has uniformly bounded three derivatives. Also, let C (R) be activation function which derivative is uniformly bounded and we denote o. = =a Then, H (D; Rhth + (D;;; RR), defined by
H = (W(K) + + Rt) +1) o o. (W(k-)) + igg-H
RDO 111+12)00 00.
o (w()) + + o a (W + +i KD) (in-112)
an generalized neural operator. Here, i and -H² are compact emending from H (D) to (D) and (D) to H (D), respectively. Non-linear integral operator in neural operator has been used in 30
Example Let consider an example of a typical neural operator which is a finite composition of layers of neural operators similar to those introduced in (30 33 F:X:
X + X of the form
F u - a o (Wu u + (T)G))
(172)
where = H Ω Rd is a bounded set with smooth boundary and W :X - X is a linear operator. In the above, Y == C(?) and Z 1 (32), where m d/2. Moreover, G: Y + Z is nonlinear integral operator
G(u)(x) (1, y, u(y))u(y)dd.
where kn Jykig E (m Dl IR) is a kernel given by feed-forward neural network with sufficiently smooth activation functions. Here, we assume that oj E C² (R), E m 2 and that the kernel ko is of the form
(2 y.t)
j=1
Moreover, Ti X + and T2 + X are the identical embedding operators
Ty(u)= u.
Due the choice of function spaces X,Y Y and Z, the maps T₁ and T2 compact operators. Summarizing, F = W, where
Fo..A(u)(2)) = ((WW)(()) + ko(a, y, u)y u(y)))(y)dd)
(173)
47

---

Furthermore, by choosing ko (z. y. u(y)) = kg( y) and Q = T as the convolutional kernel and the torus, the map F takes the form of an FNO [39]
Here the activation functions a and J are in different roles as the functions Jj appear inside the integral operator (or between the compact operators T1 and T2). The function is useful in obtaining universal approximation results for neural operators, but as this question is somewhat technical, we postpone this discussion to the end.
The appearance of the compact operators Ti and T2 makes the discretization of activation function a and the activation fanctions inside G in Definition different, and this is the reason why have introduced both a and G. consider invertible neural operators, we will below assume that a is an invertible function, for example, the leaky Relu function. In the above operator, the nonlinear function G inside compact operators in the operation
: + ()G((T))),
and the compact operators map weakly converging sequences to converging sequences. This is essential in the proofs of the positive results for approximation functors as discussed in the paper. However, we do not have general results how the operation
G u -
can be approximated by finite dimensional operators in the norm topology, but only in the weak topology in the sense Definition of the Weak Approximation Functor. However, this difficulty two ways. The first way is to use in the discretization a suitable finite dimensional space V that satisfies GV) V. For example, when X I. (22) and is a leaky relu function, one can choose V to be space that consists of piecewise constant functions. The second way is choosing composition of layers of the form
overcome
N₂ u Wyu + Ti,s(G
and
Go u +
in different finite dimensional spaces Vj. For example, we can consider these operations as maps
Ni Vi + V1,
(174)
G. V1 + V/2 := (V)),,
(175)
N2 V/2 + V/2.
(176)
G V2 V/3 = GV2))).
(177)
This makes the composition
G N2 o Go o N1 V1 + Vis,
well defined. The maps, Go. clearly invertible and when N5- V, + V, are invertible functions.
Theorems 4 And 5
to analyze
Finally, return to the question whether the activation function o is useful in universal approxima- tion results. If the activation innivaaiiofuntttiii function a removed (that is, the identical map osa - 8), the operator F is sum of linear operator and compact nonlinear integral operator,
Fw.x())(x) = (W)) + ko(s, y, uul))u(yyddy
(178)
Moreover, if compose above operators F, of the above form, the obtained operator, G: X - X is also sum of a linear operator and compact operator,
G(u) We K()).
Moreover, the Frechet derivative of G at uo, denoted DGlus, is the map
DG mg
We + DK lug v,
and, due to the above assumptions on kernel kig (x. y, u), the derivative a linear operator
DK uc H1 Th (5) H T +
48

---

By the Sobolev embedding theorem it is a compact operator DK uuo X - X. This means that the Fredholm index of the derivative of DG Iuo is constant
Ind(DGluan u.)) = In)(
that is, independent of the point un where the derivative computed. In particular, this means that one cannot approximate an arbitrary C³ -function G X + X in compact subsets of X by neural operators which are compositions of layers [178. Indeed, for a general C -functionn G X X the Fredholm index may be varying function of uo. Thus, o appears to be relevant for obtaining universal approximation theorems for neural operators.
C.2 Residual neural operators
Let D C IR be a domain. In what follows, we consider the real-valued I?2-function space (2; R) Let x (pn (pnhneN (D; R) be orthonormal basis in L (D; RR).
Definition 9 (Neural operators 33
We define neural operator G ()) + L))
G: un + up+h
where uL+1 is give by the following steps:
(x+1(x) = a ( (w() ue()) + (K (z) + b()), ZED, 0
u (x() + (K u1)(x) + 6t0) DE
where IR - I is non-linear activation operating element-wise, and W(t) E Rdd-1xde and ((0) E IRdi+1 and
(v) Σ
K.g ( (20)))9)))))(())))((()))))))) IED,
PASSN
where Ki9 Rdd (8= 0, ...,. . L. P.q I, N, do dl+2 = 1). Here, we use the notation for u (Vi,.,, Ud.) E RR)
(u, (A(:)))
(va, (p) (2(3)))))) E Rdc
We denote by NOL.N (D: the class of neural operators G (2) - L2( defined above, with depths L, rank N, orthonormal basis s, and activation function a.
Definition 10 (Residual Neural Operator). Let T,N N e N and let (0,1). We define by
R.OON,, (L) = (G: L L3 (IdLA(:)) GT) o... (IdL=(DR) Gi), G. N. La EN,, t= II.,,
-
(179)
RNO T.N.p.d (L: = -
G (IdI( + GT) o... (Id + GG), G. E N. Lip1S(D)) (G.) 6, La L=
No (L)(D:))),
(180)
Lemma 10. Let 1), and let F RNOin T.N... (D).D Let : be Lipschitz continuous. If : R + IR is Lipschitz continuous, then F F:X- + X is homeomorphism. Moreover, if a: + -RR. C²,, then, F X X is C+ C-ddiffeomorphism.
The proof is given by the same argument in Lemma 9
Lemma 11. Assume that the orthonormal basis & include the constant function. Then, we have the following inclusion:
R (L2(D; (R)) RNOO,N (D.
(181)
Rine
(T2; RR) C RNO 5.5 (T;
(182)
'We will discuss the function space (D,; IR) for easier reading, but all arguments can be replaced with real-valued function space L(D;; IR) that is separable Hilbert space.
49

---

Proof. Let G E ( such that
G G (Id(L<)D);) + Dx NNr Ex) o...o (IdLa(D.R) + Dx NN1 Ex).
Since & (PnJneN ()) has the constant basis, denoting by 4o TTp (22(2)) (x(z) = for x E D, we see that for a = (a1, aN) E EN
1p(x) where
Dva(z)= = Σ (n))(())= = Σ an (1111191)
RSN
ASN
Σ
(an 1D. 10.01))
We define Dy (D; 1 (D; by for (D(D
D
E POSNN
where Dp. E _gxx
N is defined by
(0,... O,
p=1 0, 0.....0.
| 1
D Dp. =
9-th
O,
p p#L 1.
Then, we have that
DN O
DN NN Ex(u) 1p).
(183)
Next, we see that
Ex(u) = ((u, (u. = [122)()))
p1 p1(),
We define EN : (D) + L2(;:R) by
Ex(u): = E
PASN
where Ep. E Nx1 is defined by
(0. (o..... (1..).1).))))
==1
Ep.
p-th
O,
##1.
Then we have that
Dy (NN, o Ex(u) 1p) DN N Ex((
(184)
With
183) and 184
we see that
o N EN DN NN, EN E ENOL, (LL (L²)))
where L, E N the depth of NN, Therefore, G has the form
-((dzxrnaatDoooNNodi Cu e RNOT,, (L"(...
G
(L)
182
can be proved by the same argument.
D Generalizations
D.I
Generalization of the no-go Theorem Saarremmusing using weak topology
We generalize the no-go Theorem 2 for the case when approximations and continuity of approx- imations are considered in the weak topology of the Hilbert space X. This can be done when the times the identity operator satisfy additional
approximations of the identity map and the minus assumptions and the partially ordered subset So(X) is the set of all all linear subspaces S(X) of X.
In the case when So() S(X) and Definition 7 the condition (A) can generalized follows:
50

---

Definition 11 (Weak Approximation Functor), When So(X) S(X) we define the weak approx- imation functor, that we denote by A: D - as the functor that maps each (X, F) OD to some (X, S(X), (Fv) E Os so that the Hilbert space X stays the same. The appraximation functor maps all morphisms ag As and morphisms 0x to Ax,,N and has the following properties
(A') For all r > O, all FX E OD and all y E X, it holds that
lim V-X Bx x
sup (Fv(z) F(x).
(185)
Moreover, when F; : + is the operator 1d X X or pd: :X + X, then Fv is the operator Idy V + V or IIy V V, respectively.
Moreover, Definition 8 can generalized as follows so that it uses the weak topology.
Definition 12. We say that the approximation functor A is continuous in the weak topology if the E OD be such that the Hilbert space is the same for all these objects and let (X,5(X) (Fv) (Fv)ves(X) = A(X, F) be approximating sequences of (X, F) A(X, F()) be approximating sequences of (X, F())). Moreover, assume that for r >> 0 and all y E X
following holds: Let F).
(X, FW)) FX,,,X
and (X, S(X), (Fg, (vEs()))
(x)) - F(x),,)x|= = + (0.r)
lim
sup
(186)
Then, for all V S(X) the approximations F of F() and Fv of F satisfy for all y E X
lim aEVnBy
sup
KFF (z) Fv(a), 0.
(187)
The theorem below states a negative result, namely that there does not exist continuous approximating functors for diffeomorphisms.
Theorem 8. (No-go theorem for discretization of general diffeomorphisms) There exists no functor D + B that satisfies the property (A') of weak approximation functor and is continuous in the weak topology.
The proof of Theorem 8 is analogous to Theorem by replacing A1 by the map -Id and considering linear space V E S(X) having an odd dimension, in which case deg(A V - V) -1. Let (VES)()))))
A(X, Fi) approximations of the map F: X - X in the weak topology and that A continuous in the weak topology. Recall that then Fr.v : V > V are C'-diffeomorphhsss that are discretizations of Tdy and FL. Idy so that deg( Fo.v) 1 and deg( Fi,v) = 1. Observing that when the condition (187) applied for y1, 9/2--.................. In E V that form basis of the space V having the dimension dim(V) n,, we see that the condition (187)
Fo = Id: X + X and F1 = IId: X + X. Assume that (X, S(X), (Fe,v)
F: X. Then, by condition (A'), Fo.v
implies the condition (7). Using these observations, Theorem is follows similarly to the proof of Theorem
51

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction accurately summarize and reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [NA]
Justification: Our paper is theoretical. It does not have limitations in the same was more applied paper. All theorems and propositions are proved to be true, and so are not limited in their scope.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
52

---

Answer: [Yes]
The statements of all theoretical statements are given in the main body, and all of their proofs are included in the attached appendix. A reference to the proof of each statement given either immediately before, or immediately after each statement, When appropriate, a proof sketch is given.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [NA]
Justification: Our paper does not have any numerical experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
53

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [NA]
Justification:
Our paper does not include experiments requiring code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: Our paper does not include experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Our paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
54

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: Our paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:://wwurrg..ccoooopccccccaaa
Answer: [Yes]
Justification:
We follow NeurlPS code of ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is fundamental. Our work is not tied particular application. To our knowledge, there are no direct paths to any negative applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
55

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work is theoretical, does not release data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Our work is theoretical and does contain owners of outside assets, like code, data or models.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
56

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
57

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
58


Reviews:
[{"review_id": "P41XehUItE", "reviewer": "Reviewer_mJde", "summary": "The paper addresses the problem of discretizing neural operators, maps between infinite dimensional Hilbert spaces that are trained on finite-dimensional discretizations.  Using tools from category theory, the authors provide a no-go theorem showing that diffeomorfisms between Hilbert spaces may not admit continuous approximations by diffeomorfisms on finite spaces. This highlights the fundamental differences between infinite-dimensional Hilbert spaces and finite-dimensional vector spaces. Despite these challenges, the authors provide positive results, showing that strongly monotone diffeomorphism operators can be approximated in finite dimensions and that bilipschitz neural operators can be decomposed into strongly monotone operators and invertible linear maps. Finally, they observe how such operators can be locally inverted through an iteration scheme.", "questions": "- Neural operators are typically defined between Banach spaces. Why does your theory focus on maps between Hilbert spaces instead?\n- Comment: The work in [1] might have been relevant to cite as well.\n- The main neural operator paper [2] develops theoretical results on the universal approximation theory of neural operators. How do your results relate to the ones in that paper?\n\n[1] F. Bartolucci, E. de Bézenac, B. Raonić, R. Molinaro, S. Mishra, R. Alaifari, Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning, NeurIPS 2023.\n\n[2] Nikola B. Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. \"Neural operator: Learning maps between function spaces with applications to PDEs,\" J. Mach. Learn. Res., 24(89):1–97, 2023.", "limitations": "The paper lacks  examples of applications of the theorems to specific neural operator structures, and some further discussion on the practical impact of the results with examples", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The paper provides  theoretical results addressing the challenging problem of discretizing inherently infinite-dimensional objects (neural operators)", "weaknesses": "- The text and presentation require significant polishing.  It contains numerous typos, poorly formulated sentences, and instances of missing or repeated words\n- While the paper's theoretical focus is valuable, it lacks  examples of specific neural operator structures that meet the theorems or remarks \n- A more detailed  discussion on the  practical impact of this work, accompanied by examples, would be beneficial for the audience\n\nPlease note that my review should be taken with  caution, as I am not  familiar with category theory and did not thoroughly check the mathematical details. My feedback primarily focuses on the presentation and potential impact of the results rather than a rigorous validation of the theoretical content.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable comments and constructive feedback of the reviewer. We are pleased to address all of these below.\n\n1. \"The text and presentation require significant polishing.\"\n\nWe agree and sincerely regret this, and have already made many corrections to the manuscript.\n\n2. \"While the paper's theoretical focus is valuable, it lacks examples of specific neural operator structures that meet the theorems or remarks.\"\n\nIn our approximation results (Theorem 5 and Corollary 1), while we consider a large class of bijective neural operators, approximators can be obtained through  finite rank neural operators (see Def. 10). Finite-rank neural operators are encountered, e.g., as FNOs [37], wavelet neural operators  [Tripura and Chakraborty, Wavelet Neural Operator for solving parametric partial differential equations in computational mechanics problems,  Comp. Meth. Appl. Mech. 2023], and Laplace neural operators [Chen et al, arXiv:2302.08166v2, 2023].\n\nThe neural operators,$$F:u\\to \\sigma\\circ (u+T_2(G(T_1u))),$$studied in our paper include neural operators that are close to those introduced in Kovachki-Lanthaler-Mishra (KLM) [28, 31]. This is discussed in the comments for all reviewers. \n\n3. \"A more detailed discussion on the practical impact of this work, accompanied by examples, would be beneficial for the audience.\" \n\nWe much appreciate this suggestion. In Appendix A.1 we had given an example where we approximate the solution operator $x(t)\\to u(t)$ of the nonlinear elliptic equation $$\\partial_t^2 u(t)-g(u(t))=x(t),\\quad t\\in \\Omega=(0,1),$$with $u=0$ on$\\partial \\Omega,$using a discretization that is based on Finite Element Method.In the case when $g$ is a convex function and the source term $x(t)$ is represented in the form $$x(t)=\\frac {d^2}{dt^2}h(t),$$ the map $F:h\\to u$ is a diffeomorphism in the Sobolev space $H^2$ with the Dirichlet boundary values $u(0)=0$ and $u(1)=0$. The approximation $F\\to F_V$ can be obtained by Galerkin method. We will expand upon this example in the final version of the manuscript. \n\n4. \"Neural operators are typically defined between Banach spaces. Why does your theory focus on maps between Hilbert spaces instead?\"\n\nVia our general framework, we found that strong monotonicity is one of the key ingredients to obtain a \"positive\" result, that is, preserving invariant discretization. Strong monotonicity is defined by using inner products, which is why we have focused on Hilbert spaces.\n\nHowever, the no-go theorem which states that diffeomorphisms of Hilbert spaces cannot be continuously approximated by finite dimensional diffeomorphisms implies directly that the same \"negative\" result holds for general Banach spaces.\n\nThe main challenge for using general Banach spaces for the \"positive\" result is that a map $P_Y : X \\to Y$, that maps a point to the closest points in the subspace $Y \\subset X$, may be set-valued, that is, there may be several nearest points. Nonetheless, several of our results can be generalized to uniformly convex Banach spaces, $X$. For these,\n$$\\|x\\| = \\|y\\| = 1\\hbox{ and }x\\not = y\\quad \\implies  \\|(x+y)/2\\|<1.$$In such a space, for a closed subspace $Y\\subset X$ and $x \\in X$, there is a unique closest point $y\\in Y$ to $x$. (In fact, uniformly convex Banach spaces are strictly convex Banach spaces where the above inequality  is given in a quantitative form). This makes, e.g., the linear discretization $F \\to F_V = P_V F|_V$ well defined. We will include a detailed discussion in the revision on generalizations to  strictly convex spaces.\n\n5. \"Comment: The work in [1] might have been relevant to cite as well.\n\nWe thank the reviewer for bringing this paper to our attention. We will add [1] to our references.\n\n6. \"The main neural operator paper [2] develops theoretical results on the universal approximation theory of neural operators. How do your results relate to the ones in that paper?\"\n\nWe agree with the reviewer that this is an important point. The approximation result in [2] is the universality of neural operators, i.e., to approximate any continuous map by a neural operator. Diffeomorphisms are contained in this result, which holds in the function space setting. However,  even though a general diffeomorphism, $F :\\ X \\to X$, can be approximated by a neural operator, $F^{NO} :\\ X \\to X$, and the neural operator can be approximated by a finite dimensional operator, $F^{NO}_V :\\ V  \\to V$, the proof of the no-go theorem implies that either the approximating infinite dimensional neural operators $F^{NO} : X \\to X$ are not diffeomorphisms or that the approximation of neural operators  by finite dimensional operators, that is, operation $F^{NO}\\to F^{NO}_V$, is not continuous. Note that the present universal approximation results for neural operators have mainly analyzed the approximation of functions  $F : X \\to X$ by neural operators in norms of the spaces $C(K)$, where $K\\subset X$ is compact, but not in the $C^1$-norms.\n\nWe will add this discussion to the Introduction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZIlZ148mpO", "reviewer": "Reviewer_KEed", "summary": "This paper investigates theoretical limitations of discretizing neural operators on infinite-dimensional Hilbert spaces. The authors first prove a \"no-go theorem\" (Theorems 1,2) showing that diffeomorphisms between infinite-dimensional Hilbert spaces cannot generally be continuously approximated by finite-dimensional diffeomorphisms. Then, they provide positive results for certain classes of operators such as strongly monotone (Theorem 3) and bilipschitz neural operators (Theorem 4). They finally provide concrete example of approximation by finite residual ReLU networks (Theorem 5).", "questions": "In Definition 3, why $\\sigma$ is imposed besides $G$?", "limitations": "The authors did not discuss the validity of assumptions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The universality of neural networks has been demonstrated in various settings. However, research on the approximation abilities of operators is relatively scarce. Particularly, the characterization of classes that cannot be approximated is intriguing. This study is important as it succinctly demonstrates the differences between finite-dimensional and infinite-dimensional properties in the manageable setting of Hilbert spaces.\n\n- Moreover, the novel approach of expressing approximation sequences in terms of category theory is noteworthy.", "weaknesses": "- On the other hand, the proofs are based on conventional analytical arguments rather than category-theoretic arguments. Therefore, the \"category theory\" framework might be somewhat exaggerated. It is expected that with refinement of notation and sentence structure, the description could become more perspicuous in the future.\n\n- There is concern that the categorical description may have obscured the contributions typically seen in __traditional approximation theory__ papers. As the authors likely recognize, various topologies are used in function approximation, and this study focuses __only__ on approximation in the norm topology of Hilbert spaces, and does not negate \"all considerable approximation sequences\". So, the impossibility theorem presented here might simply be due to the norm topology being too strong. While the Hilbert structure sounds natural as a generalization of Euclidean structure, in reality, concepts like L2 convergence of Fourier series are quite technical and not necessarily an inevitable notion of convergence. It seems that in pursuit of an elegant categorical description, the diversity of function approximation may have been compromised.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed suggestions, criticisms and endorsement of the reviewer. We address all of these below:\n\n1. \"On the other hand, the proofs are based on conventional analytical arguments rather than category-theoretic arguments.\"\n\nThe proofs are indeed based on analytical arguments. We used category theory as a formalism (similar to the one for object oriented programming) to describe approximation operations in all Hilbert spaces (including non-separable ones). As the collection of all Hilbert spaces cannot be considered as a set (cf. Russel's paradox) but can as a category, we chose to use the language of category theory. In the beginning of the paper, we considered an \"approximation operation\" to avoid difficulties related to formal category theory. \n\n2. \"The impossibility theorem presented here might simply be due to the norm topology being too strong.\"\n\nWe much appreciate the issue raised by the reviewer. We will include the analysis and discussion below in the revised manuscript, in an appendix on generalizations.\n\nWe formulated the approximation functor using norm topology as uniform convergence in compact sets is extensively studied in the theory of neural networks. Norm topology also makes it possible to consider quantitative error estimates. However, we agree with the referee that it is important to understand no-go results in weaker topologies. It turns out that our results can be generalized to a setting where the norm topology is partially replaced by the weak topology. Definition 7 is replaced by the following\n\nDefinition [Weak Approximation Functor]\nWhen $S_0(X)=S(X)$  we define the _weak approximation functor_, that we denote by $\\mathcal A: \\mathcal D\\to\\mathcal B$, as the functor that maps each$(X,F)\\in\\mathcal O_{\\mathcal D}$ to some$(X,S(X),(F_V)_{V \\in S(X)})$ and has the following  the properties\n\n(A')\nFor all $r>0$, all $(X,F)\\in O_{\\mathcal D}$ and all $y\\in X$, it holds that$$\\lim_{V\\to X}\\ \\sup_{x\\in B(0,r)\\cap V} \\ \\langle F_V(x)-F(x),y\\rangle_X=0.$$Moreover, when $F:X\\to X$ is the operator $Id:X\\to X$ or  $-Id:X\\to X$, then $F_V$ is  the operator $Id_V:V\\to V$ or  $-Id V:V\\to V$, respectively.\n\nIn (A') we added conditions on the approximation on the operators $Id$ and $-Id$. Similarly, the continuity of the approximation functors can be generalized in the case where the convergence in the norm topology is replaced by the weak topology. The proof of the no-go theorem generalizes also to this setting; we will add in the Appendix a theorem which states that there are no weak approximation functors that are continuous in the weak topology.\n\n3. In Definition 3, why $\\sigma$ is imposed besides $G$?\n\nThis definition needs to be interpreted with care, which we will clarify in the revised manuscript.  The appearance of the compact operators $T_1$ and $T_2$ makes the discretization of activation function $\\sigma$ and the activation functions inside $G$ in Definition 3 different, and this is one reason why we have introduced both $\\sigma$ and $G$. To consider invertible neural operators, we will below assume that $\\sigma$ is an invertible function, for example, the leaky Relu function. In the  operation $$N:u\\to u +T_2(G(T_1u)),$$ the nonlinear function $G$ is sandwiched between compact operators $T_1$ and $T_2$. The compact operators map weakly converging sequences to norm converging sequences. This is essential in the proofs of the positive results for approximation functors as discussed in the paper. However, we do not have general results on how the operation $$u\\to \\sigma \\circ u$$ can be approximated by finite dimensional operators in the norm topology, but only in the weak topology in the sense of the above \n{Definition} 1 of the Weak Approximation Functor. Nonetheless, one can overcome this difficulty, for example, for using the explicit form of the activation function and choosing different finite dimensional spaces $V_j$ in each layer of the neural operator.\n\nWe address the question whether the activation function $\\sigma$ is relevant in universal approximation results. If the activation function $\\sigma$ is removed, the operator $F$ becomes a sum of a (local) linear operator and a compact (nonlocal) nonlinear integral operator. Moreover, if we compose above operators of the above form, the resulting operator, $H$ say, is also a sum of a (local) linear operator, $W$, and a compact (nonlocal) operator, $K$. The Fr\\'{e}chet derivative of $H$ at $u_0$ is equal to $W$ and a compact linear operator. This means that the Fredholm index of the derivative of $H$ at $u_0$ equal to the index of $W$ is constant, that is, independent of the point $u_0$ where the derivative is computed. In particular, this means that one cannot approximate an arbitrary $C^1$-function $X\\to X$ in compact subsets of $X$ by such neural operators. Indeed, for a general $C^1$-function, the Fredholm index may be a varying function of $u_0$. Thus, $\\sigma$ appears to be relevant for obtaining universal approximation theorems for neural operators. Again, we will add this analysis to the final version of the manuscript.\n\n4. \"The authors did not discuss the validity of assumptions.\"\n\nWe appreciate this criticism and will address it in the final version of the manuscript. The key assumption is that the neural operator is bilipschitz while being of the general form (4). the expressability properties and applicability in designing generative models is discussed in global comments. We also point of that the strong monotonicity used as an assumption in several lemmas and theorems is an intermediate assumption that is absorbed in Theorem 4 where we consider approximation of bi-Lipschitz neural operators. \n\nIn Theorem 5, finite-rank residual neural operators appear as explicit natural approximators of bilipschitz neural operators. Such a perspective has been empirically studied as in [Behrmann, et al   PMLR 2019, pp. 573-582], although in the finite-dimensional case.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LKMYDZkYDw", "reviewer": "Reviewer_K5mB", "summary": "This paper focuses on the continuous discretization in operator learning. This is a very important question since it involves reducing the infinite-dimensional space to a finite-dimensional space in operator learning. The authors present cases where discretization is continuous and cases where it is not. The results are interesting and can be applied to design methods in operator learning.", "questions": "Mentioned in the Weakness.", "limitations": "All right.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proof is solid, and the paper is well-written and organized. I appreciate the results presented in this paper.", "weaknesses": "Since this paper is submitted to NeurIPS and not a mathematical journal, I hope the authors can provide some practical examples, such as solving the Poisson equation \\(\\Delta u = f\\) to learn the operator relationship between \\(f\\) and \\(u\\). By using methods like DeepONet and FNO, it would be beneficial to determine whether the discretization in these methods is continuous or not. I believe this could make the paper more accessible to a broader audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the suggestion to include an example based on the discretization of simple differential equations as it surely helps the readers to quickly understand the essential features of the no-go theorem of the approximation of invertible operators.\n\nOn the positive results, in Appendix A of our paper, we have considered nonlinear discretiation of the operators $u\\to -\\Delta u+G(u)$; see the reply to reviewer mJde under point 3. To exemplify the negative result, we will add in the appendix of the paper the following example on the solution operation of differential equations and the non-existence of approximation by diffeomorphic maps: We consider the elliptic (but not not strongly elliptic) problem (below, called as \"PDE1\")\n$$\nB_su:=-\\frac d{dt}\\bigg((1+t)\\hbox{sign}(t-s)\\frac d{dt} u(t)\\bigg)=f(t),\\quad t\\in [0,1],\n$$\nwith the Dirichlet and Neumann boundary conditions\n$$\nu(0)=0,\\quad \\frac d{dt} u(1)=0.\n$$\nHere, $0\\le s\\le 1$ is a parameter of the coeffient function and $\\hbox{sign}(t-s)=1$ if $t>s$ and $\\hbox{sign}(t-s)<0$ if $t<s$. We consider the weak solutions of PDE1 in the space\n$$\nu\\in H^1_{D,N}(0,1):=\\\\{v\\in H^1(0,1):\\ v(0)=0\\\\}.\n$$\nWe can write\n$$\nB_su=-D_t^{(2)}A_sD_t^{(1)}u,\n$$\nwhere \n$$\nA_sv(t)=(1+t)\\hbox{sign}(t-s)v(t),\n$$ \nparametrized by $0\\le s\\leq 1$, are multiplication operations that are invertible operators, $A_s:L^2(0,1)\\to L^2(0,1)$ (this invertibility makes the equation PDE1 elliptic). Moreover, $D_t^{(1)}$ and $D_t^{(2)}$ are the operators $v\\to \\frac {d}{dt}v$ with the Dirichlet boundary  condition $v(0)=0$ and $v(1)=0$, respectively. We consider the Hilbert space $X=H^1_{D,N}(0,1)$; to generate an invertible operator $G_s:X\\to X$ related to PDE1, we write the source term using an auxiliary function $g$,\n$$\nf(t)=Qg:=-\\frac {d^2}{dt^2}g(t)+g(t).\n$$\nThen the equation,\n$$\nB_su=Qg ,\n$$\ndefines a continuous and invertible operator, \n$$\nG_s:X\\to X,\\quad G_s:g\\to u.\n$$\nIn fact, $G_s=B_s^{-1}\\circ Q$ when the domains of $B_s$ and $Q$ are chosen in a suitable way. The Galerkin method (that is, the standard approximation based on the Finite Element Method) to approximate the equation PDE1  involves introducing a complete basis $\\chi_j(t)$, $j=1,2,\\dots$ of the Hilbert space $X$, the orthogonal projection  \n$$\nP_n:X\\to X_n:= \\hbox{span}\\\\{\\chi_j:\\ j=1,2,\\dots,n\\\\} ,\n$$\nand approximate solutions of PDE1 through solving\n$$\nP_nB_sP_nu_n=P_nQP_ng_n,\\quad u_n\\in X_n, \\ g_n=P_ng.\n$$\nThis means that operator $B_s^{-1}Q:g\\to u$ is approximated by $(P_nB_sP_n)^{-1}P_nQP_n:g_n\\to u_n$, when $P_nB_sP_n:X_n\\to X_n$ is invertible.\n\nThe above corresponds to the Finite Element Method where the matrix defined by the operator $P_nB_sP_n$\nis  $m(s)= [b_{jk}(s)]_{j,k=1}^n\\in \\mathbb R^{n\\times n}$, where\n$$\nm(s)=\\int_0^1 (1+t)\\hbox{sign}(t-s)\\frac d{dt} \\chi_j(t)\\cdot \\frac d{dt} \\chi_k(t)dt,\\quad j,k=1,\\dots,n.\n$$\nSince we used the  mixed Dirichlet and Neumann boundary conditions in the above boundary value problem, we see that for $s=0$ all eigenvalues of the matrix $m(s)$ are strictly positive, and when $s=1$ all eigenvalues  are strictly negative. As the function $s \\to m(s)$ is a continuous matrix-valued function, we see that there exists $s\\in (0,1)$ such that  the matrix $m(s)$ has a zero eigenvalue and is no invertible. Thus, we have a situation where all operators $B_s^{-1}Q:g\\to u$, $s\\in [0,1]$ are invertible (and thus define diffeomorphisms $X\\to X$) but for any basis $\\chi_j(t)$ and any $n$ there exists $s\\in (0,1)$ such that the finite dimensional approximation $m(s):\\mathbb R^n\\to \\mathbb R^n$ is not invertible. This example shows that there is no FEM-based discretization method for which the finite dimensional approximations of all operators $B_s^{-1}Q$,  $s\\in (0,1)$, are invertible. The above example also shows a key difference between finite and infinite dimensional spaces. The operator $A_s:L^2(0,1)\\to L^2(0,1)$ has only continuous spectrum and not eigenvalues nor eigenfunctions whereas the finite dimensional matrices have only point spectrum (that is, eigenvalues). The  continuous spectrum makes it possible to deform the  positive operator $A_s$ with $s=0$  to a negative operator $A_s$ with $s=1$ in such a way that all operators $A_s$, $0\\le s\\le 1$, are invertible but this is not possible to do for finite dimensional matrices. We point out  that the map $s\\to A_s$ is not continuous in the operator norm topology but only in the strong operator topology and the fact that $A_0$ can be deformed to $A_1$ in the norm topology by a path that lies in the set of invertible operators is a deeper result. However, the strong operator topology is enough to make the FEM matrix  $m(s)$ to depend continuously on $s$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JXcAoFhejd", "reviewer": "Reviewer_47x8", "summary": "This paper studies the question of whether a neural operator (or a general diffeomorphism on an infinite dimensional Hilbert space) can be continuously discretized through the lens of category theory. It first proves that there does not exist a continuous approximation scheme for all diffeomorphisms. Then, it shows that neural operators with strongly monotone layers can be continuously discretized, followed by a proof that a biliptschitz neural operator can be approximated by a deep one with strongly monotone layers. Some further consequences are discussed.", "questions": "1. In general, is there any assumption of the Hilbert space studied in this paper? For example, are the Hilbert spaces assumed to be separable?\n2. What is the definition of the convergence of finite-dimensional subspaces used in this paper? Strong convergence of the projection operator? I do not think there is a standard definition for this in elementary functional analysis so it would be helpful to say it explicitly in the paper.\n3. Have you studied the role of the bilipschitz constant in your theorems? For example, how does the number of layers $J$ in Theorem 4 depend on it?", "limitations": "None.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper studies the discretization of a continuous operator, which is a source of error and an important issue in operator learning if one is not careful.\n2. The paper is fairly comprehensive, encompassing both positive and negative results. The study of positive results contains theorems of different flavors.\n3. Although the theory is based on the category theory, the presentation and explanation of the results are relatively clear and accessible for people who are unfamiliar with it.", "weaknesses": "1. Most results presented in the paper are purely theoretical and lack a quantitative or asymptotic estimate. For example,\n   * the notion of continuous approximation functor in Definition 8 does not care about the rate of convergence, and\n   * there is no estimate for the number of layers $J$ in Theorem 4.\n2. No empirical results are there to support the theory. While this is a theoretical paper, some toy experiments that exemplify the theory would be very helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the detailed comments and fair criticisms. We address all of these below:\n\n1. \"Most results presented in the paper are purely theoretical and lack a quantitative or asymptotic estimate.\"\n\nThe proof of Theorem 4 makes it possible to estimate the number $J$ of layers as a function of $\\epsilon$, the Lipschitz constants of the map $F$ and $F^{-1}$, the $C^2$-norm of $F:B(0,2r_1)\\to X$ in  a ball having double the radius $r_1$ where we consider the approximation of $F$. More precisely, $$ J\\leq C\\epsilon^{-2} $$ where $C$ depends on the radius $r_1$ of the ball where $F$ is approximated and the \n$C^2$-norm of the map $F$ in a ball of radius $2r_1$. We will add an explicit formula for $C_0(r_1,\\|F\\|_{B(0,2r_1);X)})$ and its proof in the final version of the paper.\n\nThe main steps of the proof are the following: First, we use spectral theory of the compact operators $T_1$ and $T_2$ to find a finite dimensional subspace $W$ and the projection $P_W$ onto it so that $$\\hat F(x)=(I-P_W)x+P_WF(P_WF) $$ is a diffeomorphism that is close to the operator $F:X\\to X$. Let $f:W\\to W$ be the restricition of $\\hat F$ to $W$. After this we deform  to the invertible linear map $A_1:W\\to W$ is the derivative of $f$at $x=0$, along the path $t\\to f_t,$ $$f_t(x):=\\frac 1t(f(tx)-f(0))+tf(0),\\quad\\hbox{for }t>0,$$ $$ f_t(x):=A_1,\\quad\\hbox{for }t=0.$$ All operators $f_t:W\\to W$ are bi-Lipschitz maps, $f_0(x)=f(x)$ and $f_0(x)=A_1$. We consider the values $t_1=c_1\\epsilon$ and  $t_j=t_1+jh$, for $j=2,\\dots,J_1$, and the operators $$ Id+B_j=f_{t_{j+1}}\\circ f_{t_j}^{-1}\\ : W\\to W,\\quad j=0,1,2,\\dots,J_1.$$ We show that in the ball $B_W(0,2r_1)$ $$\\hbox{Lip}(B_0)=\\hbox{Lip}(f_{t_1}\\circ A_1^{-1}-Id)\\le C_2t_1=C_2c_1\\epsilon.$$ Here, $C_2$ depends on $r_1$ and $C^2$-norm of $F$ as well as on the Lipschitz constants of $F$ and $F^{-1}$ and $c_1$ is chosen to be sufficiently small.\n\nMoreover, we show that for $j\\ge 1$, $$\\hbox{Lip}(B_j)\\leq C\\frac 1{t_j}(t_{j+1}-t_j)=C\\frac 1{t_j}h,$$ where the factor $\\frac 1{t_j}$ appears due to the multiplier $\\frac 1t$ in the definition of $f_t$. To obtain $\\hbox{Lip}(B_j)\\le \\epsilon$, we choose $h\\le c_2\\epsilon^2$. This causes the factor $\\epsilon^{-2}$ in the bound for $J$.\n\nIn addition to this, we consider paths on the Lie group $O(n)$, $n=\\dim(W)$ and show that there are sequence of invertible matrixes $A_j\\in O(n)$, $j=1,2,\\dots,J_2$ such that  $$ Id+B_{J_1+j}=A_{j+1}\\circ A_j^{-1}\\ :W\\to W$$ where $\\hbox{Lip}(B_j)<\\epsilon$ and $A_j$, where $j=J_2$, is either the identity operator $Id:W\\to W$ or a reflection operator $B_e:x\\to x-2\\langle x,e\\rangle e$. Here, we can choose the number of steps to be  $J_2\\leq c_3\\epsilon^{-1}.$ Combining the  operators $Id+B_j$ with $j=1,\\dots, J_1+J_2$, we see that $F$ can be deformed to a linear operator or a reflection operator by combining $J=J_1+J_2$ operators of the form $Id+B_j$. This yields the bound for $J$.\n\n2. \"While this is a theoretical paper, some toy experiments that exemplify the theory would be very helpful.\"\n\nWe much appreciate the reviewer's comment.  In Appendix A.1 we had given an example where we approximate the solution operator $x(t)\\to u(t)$ of the nonlinear elliptic equation $$ \\partial_t^2 u(t)-g(u(t))=x(t),\\quad t\\in (0,1), $$ with $u=0$ on the boundary using a discretization that is based on Finite Element Method. When $g$ is convex and the source term $x(t)$ is represented in the form $$x(t)=\\frac {d^2}{dt^2}h(t),$$ the map $F:h\\to u$ is a diffeomorphism in the Sobolev space $H^2$ with the Dirichlet boundary values. The approximation $F\\to F_V$ can be obtained by Galerkin method. We will expand upon this example in the final version of the manuscript. We will give an example on no-go theorem using the elliptic (but not not strongly elliptic) problem $$B_su:=-\\frac d{dt}\\bigg(\\hbox{sign}(t-s)\\frac d{dt} u(t)\\bigg)=f(t),\\quad t\\in [0,1],$$ $$u(0)=0,\\ \\ \\frac d{dt} u(1)=0.$$ For all $0\\le s\\le 1$ these equations are uniquely solvable, but we show that when we use FEM to approximate those, some of the obtained finite dimensional problems has also a zero eigenvalue and is not solvable.\n\n3. \"Do the Hilbert spaces need to be separable?\"\n\nThe no-go theorem applies also to non-separable Hilbert spaces. Naturally, for such spaces the partially ordered set $S(X)$ of finite dimensional linear subspaces of $X$, that is used as an index set, is huge. In most of our positive results on existence of approximation operations we have assumed that the Hilbert space is separable as we use finite rank neural operators as approximators, and have used this in the orthogonal projectors $P_n$ form $X$ to $\\hbox{span}(\\phi_1,\\dots,\\phi_n)$, where $\\phi_j,$ $j \\in \\mathbb Z_+$, is an enumerable orthonormal basis. However, it seems to us that our results can be generalized to non-separable Hilbert spaces $X$ that have a non-enumerable orthonormal bases. We will check carefully if this generalization is possible.\n\n5. \"What is the definition of the convergence of finite-dimensional subspaces used in this paper?\"\n\nAfter Definition 7, line 223, we defined that the limit $$ \\lim_{V\\to X} y_V=y. $$ This limit can be defined also by endowing the set $S_0(X)\\cup \\{X\\}$ by the topology associated to  the partial ordering of  $S_0(X)\\cup \\{X\\}$, that is, the topology generated by the sets $U_V:=\\{W\\in S_0(X):\\ X\\supset V\\\\}\\cup \\{X\\}$.\n\n6. \"Have you studied the role of the bi-Lipschitz constant in your theorems? For example, how does the number of layers $J$ in Theorem 4 depend on it?\"\n\nThe bi-Lipschitz constraint (and form of neural operator layers) enable us to decompose the map into strongly monotone neural operator layers $G_k=Id+B_k$ (Theorem 4), where $\\mathrm{Lip}(B_k)<\\epsilon$. Here, $\\epsilon \\in (0,1)$ is arbitrary. The bi-Lipschitz constant appears in the proof of the estimate of $J$ (under 1.) We will mention this observation in the final version of the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable comments and detailed questions. We will provide replies to the individual reviewers below, but first would like to make some general statements addressing a few issues raised by all the reviewers.

Common questions: Practical impact/examples of this work?

A practical implication of our result is the description of  bi-Lipschitz neural operators as a composition of discretization invariant layers of invertible finite dimensional neural operators (i.e. neural networks). Such neural operators are useful in generative models where a probability distribution $\mu_0$ supported on a given model manifold $M_0$ is pushed forward by a map $F_\theta$ to a distribution that one would like to be close to an empirical target distribution $\mu_{data}$ supported on some submanifold $M_{data}$ of the Hilbert space $X$. (Here $\mu_{data}$ and $M_{data}$ are unknown and $\theta$ are optimized with samples from $\mu_{data}$). Suppose that we know a priori the topology of the data manifold $M_{data}$ and there is a diffeomorphism $f_0:M_0\to M_{data}.$ As all smooth finite dimensional submanifolds of a Hilbert space are close to some finite dimensional subspace $V$, one can start by assuming that there exists an embedding $f_1:M_0\to P_V(M_{data})$ that is close to $f_0$, where $P_V$ is a finite dimensional orthoprojection onto $V$. By considering the model manifold $M_0$ as a subset of $V$, we can extend the embedding $f_1:M_0\to V$ to a diffeomorphism $F_0:V\to V$. This can be done when the dimension of $V$ is sufficiently large [Puthawala et al., ICML 2022]. 

Furthermore, $F_0$ can be extended to a diffeomorphism $$F_{ext}=F_0\times Id_{V^\perp}:X\to X,$$where $F_{ext}$ maps $x=v+w\in V\oplus V^\perp$ to$$F_{ext}(v+w)=F_0(v)+w.$$The map $F_{ext}$ can be written as$$F_{ext}=Id+P_V\circ G\circ P_V, $$where $P_V$ is a compact linear operator and $G=F_{ext}-Id$. By definition the map $F_{ext}$ is a neural operator diffeomorphism. Thus, diffeomorphic neural operators can be used to obtain generative models. As the finite dimensional subspace $V$ is not a priori known, and its dimension depends on the accuracy required for the generative model, it is natural to consider infinite dimensional neural operators $F:X\to X$ and study their approximation properties.

In our paper we show, in Theorem 3, that strongly monotone neural operators can be approximated continuously by finite dimensional neural operators that are diffeomorphisim (so, invertible). In Theorem 4 we show that any bi-Lipschitz neural operator (not necessarily strongly monotone) can locally be represented as a composition of strongly monotone neural operator layers. This implies that bi-Lipschitz neural operators can be approximated by a composition of invertible, finite dimensional neural networks in a continuous way. This makes invertible neural operators a class that behaves well in finite dimensional approximations. Our results can also be summarized by stating that neural operators conditionally serve as a class of diffeomorphisms of function spaces that are simple enough for well-working approximations but still sufficiently expressive (and may model a rich variety of deformations).

The neural operators$$F:u\to \sigma\circ (u+T_2(G(T_1u)))$$we study include the neural operators that are close to those introduced in Kovachki-Lanthaler-Mishra (KLM) [28, 31].  We have assumed that operators $T_1$ and $T_2$ are compact linear operators; in several cases these can be chosen to be identity embeddings that are maps between different function spaces so that these embeddings are compact. 

Consider a KLM neural operator $F:X\to X$ of the form$$F:u\to \sigma\circ (u+S_2(H(S_1u))),$$where $X=H^m(D)$ and $D\subset \mathbb R^d$ is a bounded set. Moreover, let  $Y = C(\overline D)$ and $Z = C^{m+1}(\overline D)$, where $m>d/2$. Let $h:Y \to X$ be a nonlinear (integral) operator,$$H(u)(x)=\int_D k_\theta(x,y,u(y))u(y)dy,$$where $k_\theta$ is a kernel given by a neural network with sufficiently smooth activation functions $\sigma_j$ of the form$$k_\theta(x,y,t)=\sum_{j=1}^{J} c_{j}(x,y,\theta)\sigma_j(a_{j}(x,y,\theta)t+b_j(x,y,\theta)),$$and $S_1 : X \to Y$ and $S_2  : Z \to X$ the identity embedding operators mapping between function spaces,$$S_j(u)=u.$$Thus,$$F(u)(x)=\sigma(u(x)+\int_{D}k_\theta(x,y,u(y))u(y)dy).$$The Hilbert spaces $X,Y$, and $Z$ are isomorphic and by writing, e.g.,$$S_2\circ H=T_2\circ G,$$where$$T_2=S_2\circ J_Z^{-1},\ G=J_Z\circ h,$$where $J_Z : Z \to X$ is an isomorphism, we can write $F$in the form$$F(u)=\sigma\circ (u+T_2(G(T_1u))),$$in which $T_1:X\to X$ and $T_2:X\to X$ are compact linear operators and $G:X\to X$ a continuous nonlinear operator. In this way, the  KLM-operator $F$ can be written in the form  studied in our paper.

Furthermore, by choosing $k_{\theta}(x,y,u(y)) = k_{\theta}(x - y)$ and $D = \mathbb{T}^d$ as the convolutional kernel and the torus, the map $F$ takes the form of an FNO [37].

Meta-review: The reviewers find this work studying an important problem and encourage the authors to incorporate the suggestions and developments during the review process.
Decision: Accept (poster)