Can neural operators always be continuously discretized?

Authors: takashi  furuya
michael  puthawala
maarten v de hoop
matti  lassas
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 23:05:08

Abstract: In this work we consider the problem of discretization of neural operators in a general setting. Using category theory, we give a no-go theorem that shows that diffeomorphisms between Hilbert spaces may not admit any continuous approximations by diffeomorphisms on finite spaces, even if the discretization is non-linear. This shows how infinite-dimensional Hilbert spaces and finite-dimensional vector spaces fundamentally differ. A key take-away is that to obtain discretization invariance, considerable effort is needed to ensure that finite-dimensional approximations of neural operator converge not only as sequences of functions, but that their representations converge in a suitable sense as well. With this perspective, we give several positive results. We first show that strongly monotone diffeomorphism operators always admit finite-dimensional strongly monotone diffeomorphisms. Next we show that bilipschitz neural operators may always be written via the repeated alternating composition of strongly monotone neural operators and invertible linear maps. We also show that such operators may be inverted locally via iteration provided that such inverse exists. Finally, we conclude by showing how our framework may be used `out of the box' to prove quantitative approximation results for discretization of neural operators.

Main paper: 
 introduction: 
 Neural operators, first introduced in [30] , have become more and more prominent in deep learning on function spaces. As opposed to traditional neural networks that learn maps between finite-dimensional Euclidean spaces, neural operators learn maps between infinite-dimensional function spaces yet may be trained and evaluated on finite-dimensional data through a rigorous notion of discretization. Neural operators are widely used in the field of scientific machine learning [6, 21, 38, 55, 56] , among others, principally because of their discretization invariance. In this work, we consider the fundamental limits of this discretization. Throughout, we emphasize the importance of continuity under discretization.A key ingredient in our analysis is the identification of properties of diffeomorphisms that may be induced by (bijective) neural operators, which are diffeomorphisms themselves. Diffeomorphisms exist in many contexts, for example, in generative models. These involve mapping one probability distribution or measure, µ, over some measurable space to another, X, via a push forward, that is, F # µ(U ) = µ(F -1 (U )) for U ⊂ X. If µ admits a density dµ then, in finite dimensions, we may use the change of variables formula to write dρ(x) = dµ(F -1 (x))|JF (x)| -1 , where JF is the Jacobian of F . Clearly, F must be a bijection with full-rank Jacobian. In other words, F must be a diffeomorphism onto its range. This established diffeomorphisms as natural objects of interest in finite-dimensional machine learning, and helps account for their wide use [20, 25, 31, 44, 54] . In this work, we consider the extension of these efforts from finite to infinite dimensions implemented via neural operators. Although there is no analogue of the change of variables formula in infinite dimensions, we argue that it is, nonetheless, natural to consider the role of diffeomorphisms, and how they may be approximated via diffeomorphisms on finite-dimensional spaces.The question of when operations between Hilbert spaces may be discretized continuously may be understood through an analogy to computer vision. Consider the task of learning a map from one image space to another, for example, a style transfer problem [16] , where the mapping learned does not depend much on the resolution of the images provided. It is natural to think of the map as being defined between (infinite-resolution) continuum images, and then its application to images of a specific resolution. In this analogy, X is a function space (over images m : [0, 1] 2 → R [29] ) that is approximated with a finite-dimensional space R d and the transformation F : X → X is approximated by a map f : R d → R d , where each f acts on images of a particular resolution. An explicit transformation formula can be obtained when f is a diffeomorphism and has a smooth inverse.We introduce a framework based on a generalized notion of neural operator layers including a skip connection and their restrictions to balls rather than compact sets. With bijective neural operators in mind, we give a perspective based on diffeomorphisms in infinite dimensions between Hilbert manifolds. We give a no-go theorem, framed with category theory, that shows that diffeomorphisms between Hilbert spaces may not admit any continuous approximations by diffeomorphisms on finitedimensional spaces, even if the underlying discretization is nonlinear. In this framing the discretization operation is modeled as a functor from the category of Hilbert spaces and C 1 -diffeomorphisms on them to their finite-dimensional approximations. A natural way to mitigate the no-go theorem is described by the introduction of strongly monotone diffeomorphisms and layerwise strongly monotone neural operators. We prove that all strongly monotone neural operator layers admit continuous approximations by strongly monotone diffeomorphsisms on finite-dimensional spaces. We then provide various conditions under which a neural operator layer is strongly monotone. Notably, a bilipschitz (and, hence, bijective) neural operator layer can always be represented by a composition of strongly monotone neural operator layers. Hence, such an operator may be continuously discretized. More constructively, any bilipschitz neural operator layer can be approximated by residual finite-rank neural operators, each of which are strongly monotone, plus a simple isometry. Moreover, these finite-rank residual neural operators are (locally) bijective and invertible, and their inverses are limits of compositions of finite-rank neural operators. Our framework may be used "out of the box" to prove quantitative approximation results for discretization of neural operators.Neural operators were first introduced in [30] . Alternative designs for mappings between function spaces are the DeepONet [34, 40] , and the PCA-Net [7, 12] . In spite of the multitudinous applications of neural operators, the theory of the natural class of injective or bijective neural operators is comparatively underdeveloped; see, for example [2, 14] .Our work is concerned with the of discretization of neural operators through the lens of diffeomorphisms. For recent important work on analyzing the effect of discretization error of Fourier Neural Operators (FNOs) arising from aliasing, see [35] . Our work has connections to infinite-dimensional inference, see e.g. [19] , and approximation theory, see e.g. [13] while bridging the gap with the theory of neural operators.We give a no-go theorem that uses a category theory framing. This contributes to the use of category theory as an emerging tool in the analysis and understanding of neural networks at large. In this sense, we are in league with the recent work generalizing ideas from geometric machine learning using category theory [17] .Discretization obstructions have been encountered in other contexts. Numerical methods that approximate continuous models are known to sometimes fail in surprising ways. A basic example of this is the "locking" phenomenon in the study of the Finite Elements Method (FEM). For example, linear elements used to model bending of a curved surface or beam lock in such a way that the model exhibits a non-physical stiff response to deformations [4] . Understanding this has been instrumental in developing improved numerical methods, such a high order FEM [52] . Furthermore, in discretized statistical inverse problems [27, 36, 51] , the introduction of Besov priors [48, 11] has been found to be essential. Finally, our work extends prior work (not based on deep learning) in discretization of physical or partial differential equations based forward models in inverse problems. The analogous notion of discretization invariance of solution algorithms of inverse problems was studied in [37, 48, 51] and the lack of it (in imaging methods using Bayesian inversion with L 1 priors) in [36, 48] . By considering the neural operator as the physical model, our results state that discretization can be done locally in an appropriate way, together with constructing an inverse.The key results in this paper comprise the following:1. We prove a general no-go theorem showing that, under general circumstances, diffeomorphisms between Hilbert spaces may not admit continuous approximation by finitedimensional diffeomorphisms. In particular, neural operators corresponding to diffeomorphic maps, in general, cannot be approximated by finite-dimensional diffeomorphisms and their associated neural representations.2. We show that strongly monotone neural operator layers admit continuous approximations by strongly monotone diffeomorphsisms on finite-dimensional spaces.3. We show that bilipschitz neural operators can be represented in any bounded set as a composition of strongly monotone, diffeomorphic neural operator layers, plus a simple isometry. These can be approximated by finite-rank diffeomorphic neural operators, where each layer is strongly monotone. For these operators we give a quantitative approximation result. 
definitions and notation: 
 In this section, we give the definitions and notation used throughout the paper. First, we summarize the relevant basic concepts from functional analysis. Then, we introduce generalized neural operators.In this work, all Hilbert spaces, X, are endowed with their norm topology. We denote by B X (r) = B X (0, r) the ball in the space X having the center at zero and radius r > 0. We denote by S(X) the set of all finite-dimensional linear subspaces V ⊂ X. The set S 0 (V ) ⊂ S(X) is a partially ordered lattice. That is, if V 1 , V 2 ∈ S 0 (X) then there is a V 3 ∈ S 0 (X) so that V 1 ⊂ V 3 and V 2 ⊂ V 3 . 1 With Y standing for another Hilbert space, we denote by C n (X; Y ) the set of operators, F : X → Y , having n continuous (Fréchet) derivatives, and C n (X) = C n (X; X).Next, we define what it means that a nonlinear operator or function F : X → X on an infinitedimensional Hilbert space, X, is approximated by operators or functions on finite-dimensional subspaces V ⊂ X. The key is that as V tends to X, the complexity of F V increases and one may hope that the approximation becomes better. We formalize this in the following definition. Definition 1 (ϵ V approximators and weak approximators). (i) Let r > 0, F ⊂ C n (X; X) be a family of functions, and ⃗ ε = (ε V ) V ∈S0(X) be a sequence such that ε V → 0 as V → X. We say that a functionA X : F → × V ∈S0(X) C(B V (0, r); V ), F → (F V ) V ∈S0(X)1 Each element of S0(X) will come to represent a discretization of X. The partially ordered lattice condition will come to represent a notion of common refinement of a discretization. This makes it possible to consider "realistic" choices of discretizations. The condition automatically follows for any discretization scheme that has a notion of "common refinement" of two discretizations. Examples include finite-difference schemes, and the finite elements Galerkin discretization that is based on a triangulation of the domain.is an ⃗ ε-approximation operation for functions F in the ball B X (0, r) taking values in families F V ⊂ C 1 (V ; V ) if A X maps a function F : X → X, where F ∈ F, to a sequence of functions (F V ) V ∈S0(X) , where F V ∈ F V , such that the following is valid: For all F : X → X satisfying ∥F ∥ C n (B X (0,r);X) ≤ M , we haveEQUATIONwhere P V : X → X is the orthogonal projection onto V , that is, Ran(P V ) = V .(ii) We say that A : C n (X; X) → × V ∈S0(X) C(V ; V ), F → (F V ) V ∈S0(X) is a weak approxima- tion operation for the family F ⊂ C n (X; X) if for any F ∈ F and r > 0 it holds that limV →X sup x∈B V (0,r) ∥F V (x) -P V (F (x))∥ X → 0.Note that the condition (i) is stronger than the condition (ii). An example of an approximation operation for the family F = C n (X), that is, an ⃗ ε-approximation operation with all sequencesEQUATIONNonlinear discretization methods that do not rely on P V have been used, for example, in the numerical analysis of nonlinear partial differential equations. Here, X becomes an appropriate Sobolev space, and a Galerkin approximation is implemented through finite-dimensional subspaces, V , spanned by finite element basis functions. We present an example for the nonlinear equation, ∆u(t) -g(u(t)) = ∆x(t) where g is a smooth convex function, whenF : x → u, in Appendix A.1.Below, we will study whether a family, F ⊂ Diff 1 (X), of C 1 diffeomorphisms on X can be approximated by C 1 diffeomorphisms, F V ⊂ Diff 1 (V ), on finite-dimensional subspaces, V . Of course, diffeomorphisms are bijective. Unless stated otherwise, from now on we will omit C 1 and implicitly assume that diffeomorphism are C 1 diffeomorphisms. We introduce two more notions that will play a key role in the further analysis. Definition 2 (Strongly Monotone). We say that a (nonlinear) operator F : X → X on Hilbert space, X, is strongly monotone if there exists a constant α > 0 so thatEQUATIONDefinition 3 (Bilipschitz). We say that F if bilipschitz there exist constants c > 0 and C < ∞ so that for allx 1 , x 2 ∈ X, c ∥x 1 -x 2 ∥ ≤ ∥F (x 1 ) -F (x 2 )∥ ≤ C ∥x 1 -x 2 ∥.In this paper, we are concerned with the modeling of diffeomorphisms between Hilbert spaces by bijective neural operators. Our working definition of neural operator, which generalizes the traditional notion, is given below. We note the presence of a skip connection, which is essential. Definition 4 (Generalized neural operator layer). For Hilbert space X, a layer of a neural operator is a function F : X → X of the formEQUATIONwhere T 1 : X → X and T 2 : X → X are compact linear operators2 and G : X → X is a nonlinear operator in C 1 (X). A generalized neural operator, H : X → X, is given by the compositionEQUATIONwhere each F ℓ , ℓ = 1, . . . , L is of the form (4), the A ℓ : X → X are bounded linear operators and σ : X → X is a continuous operation (for example, a Nemytskii operator defined by a composition with a suitable activation function in function spaces).The generalized neural operators can represent the classical neural operators [30, 33] . For an explicit construction, we refer to Appendix C.1. In the next section, we will study, under what conditions, bounded linear operators A ℓ , Nemytskii operators σ, and neural operator layers F ℓ , for which the generalized neural operator consists, can be continuously discretized.We note that because G ∈ C 1 (X) in Definition 4, it follows that G ∈ L ∞ (X) and Lip X→X (G) < ∞. Given a Hilbert space, X, a layer of a strongly monotone neural operator (respectively, a layer of a bilipschitz neural operator,) is a function F : X → X that is strongly monotone (respectively, bilipschitz). Furthermore, a strongly monotone neural operator (respectively, a bilipschitz neural operator), is a generalized neural operator with strongly monotone (respectively, bilipschitz), layers.3 Category theoretic framework for discretization "Well-behaved" operators between infinite-dimensional Hilbert spaces may have dramatically different behaviors than corresponding "well-behaved" maps between finite-dimensional Euclidean spaces. This observation applies to discretization and neural operators versus neural networks. In this section we explore this. We first present a no-go theorem, that there are no procedures that continuously discretize an isotopy of diffeomorphisms. Next, we introduce strongly monotone neural operator layers, which are strongly monotone diffeomorphisms, and then prove that these allow a continuous approximation functor, that is, continuous approximations by strongly monotone diffeomorphisms on finite dimensional spaces. We finally show that bilipschitz neural operator layers admit a representation via strongly monotone operators and linear maps, allowing for their continuous approximation. 
no-go theorem for discretization of diffeomorphisms on hilbert spaces: 
 In this section, we present our no-go theorem. To formulate the 'impossibility' of something, we must define what is meant by discretization and approximation. Before this, we give an informal statement of the no-go theorem. Theorem 1 (No-go Theorem, Informal). Let A be an approximation scheme that maps diffeomorphisms F on a Hilbert to a sequence of finite-approximations F V that are themselves diffeomorphisms.If F V converges to F as V → X, then A is not continuous, that is, there are maps F (j) that converge to F as j → ∞, but all F (j)V are far from F V .We want to emphasize that most practical numerical algorithms are continuous so that the output depends (in some suitable sense) continuously on the input. This shows that there are no such numerical schemes that approximate infinite-dimensional diffeomorphisms with finite-dimensional ones. In order to prove our no-go theorem in the most general setting, we phrase it in terms of category theory. Namely, we formulate A (which will denote the approximation scheme) as a functor from the category of Hilbert spaces and diffeomorphisms thereon, to their finite-rank approximations. Definition 5 (Category of Hilbert Space Diffeomorphisms). We denote by D the category of Hilbert diffeomorphisms with objects O D that are pairs (X, F ) of a Hilbert space X and a (possibly nonlinear) C 1 -diffeomorphism F : X → X and the set of morphisms (or arrows that 'map' objects to other objects) A that are either 1. (induced isomorphisms) Maps a ϕ that are defined for a linear isomorphism ϕ : X 1 → X 2 of Hilbert spaces X 1 and X 2 that maps the objectsEQUATION. (induced restrictions) Maps a X1,X2 that are defined for a Hilbert space X 1 , its closed subspace X 2 ⊂ X 1 , and an object(X 1 , F 1 ) ∈ O D such that F 1 (X 2 ) = X 2 . Then a X1,X2 maps to the object (X 1 , F 1 ) ∈ O D to the object (X 2 , F 1 | X2 ) ∈ O D .Definition 6 (Category of Approximation Sequences). We denote by B the category of approximation sequences, that has objects O B that are of the form (X, S 0 (X), (F V ) V ∈S0(X) ) where X is a Hilbert space, S 0 (X) ⊂ S(X) = {V | V ⊂ X is a finite dimensional linear subspace}, are partially ordered lattices, V ∈S0(X) V = X, andF V : V → V are C 1 -diffeomorphisms of spaces V ∈ S 0 (X).The set of morphisms A B consists of either 1. Maps A ϕ that are defined for a linear isomorphism ϕ : X 1 → X 2 of Hilbert spaces X 1 and X 2 , and lattices S 0 (X 1 ) and S 0 (X 2 ) = {ϕ(V ) | V ∈ S 0 (X 1 )}, that maps the objects(X 1 , S(X 1 ), (F V ) V ∈S(X1) ) to (X 2 , S(X 2 ), (ϕ • F ϕ -1 (W ) • ϕ -1 ) W ∈S(X2) ), or 2.Maps A X1,X2 that are defined for a Hilbert space X 1 , its closed subspace X 2 ⊂ X 1 , and an object (X 1 , S 0 (X 1 ), (F V ) V ∈S0(X1) ) such that F (X 2 ) = X 2 and S 0 (X 2 ) = {V ∈ S 0 (X 1 ) | V ⊂ X 2 } is a partially ordered lattice. Then A X1,X2 maps the object (X 1 , S 0 (X 1 ), (F V ) V ∈S0(X1) ) to the object (X 2 , S 0 (X 2 ), (F V ) V ∈S0(X2) ).Next, we define the notion of an approximation or discretization functor. In practice, an approximation functor is an operator which maps a function F in an infinite dimensional space X to a function F V that operate in finite dimensional subspaces V of X in such a way that functions F V are close (in a suitable sense) to the function F . Definition 7 (Approximation Functor). We define the approximation functor, denoted by A : D → B, as the functor that maps each (X, F ) ∈ O D to some (X, S 0 (X), (F V ) V ∈S0(X) ) ∈ O B so that the Hilbert space X stays the same. The approximation functor maps all morphisms a ϕ to A ϕ and morphisms a X1,X2 to A X1,X2 , and has the following the properties (A) For all r > 0 and all (X,F ) ∈ O D , lim V →X sup x∈B X (0,r)∩V ∥F V (x) -F (x)∥ X = 0.In separable Hilbert spaces this means that when the finite dimensional subspaces V ⊂ X grow to fill the whole Hilbert space X, then the approximations F V converge uniformly in all bounded subsets to F .We recall the notation lim V →X used above: We consider (S 0 (X), ⊃) as a partially ordered set and say that real numbers y V converge to the limit y as V → X, and denotelim V →X y V = y,if for all ϵ > 0 there is V 0 ∈ S 0 (X) such that for V ∈ S 0 (X) satisfying V ⊃ V 0 it holds that |y V -y| < ϵ.Definition 8. We say that the approximation functor A is continuous if the following holds: Let (X, F ), (X, F (j) ) ∈ O D be such that the Hilbert space X is the same for all these objects and let (X, S 0 (X), (F V ) V ∈S0(X) ) = A(X, F ) be approximating sequences of (X, F ) and (X, S 0 (X), (F j,V ) V ∈S0(X) ) = A(X, F (j) ) be approximating sequences of (X, F (j) ). Moreover, assume that r > 0 and lim j→∞ sup x∈B X (0,r)EQUATIONThen, for all V ∈ S 0 (X) the approximationsEQUATIONThe theorem below states a negative result, namely that there does not exist continuous approximating functors for diffeomorphisms. Theorem 2. (No-go theorem for discretization of general diffeomorphisms) There exists no functor D → B that satisfies the property (A) of an approximation functor and is continuous.The proof is given in Appendix A.4.1, and is quite involved, but we give an overview of some of the steps here. A generalization of Theorem 2, in the case where the norm topology is replaced by the weak topology, is considered in Appendix D.1.A key fact is that for finite dimensional diffeomorphisms the space of smooth embeddings consists of two connected components, one orientation preserving and the other orientation reversing. This is not the case in infinite dimensions, see e.g. [32] and [45] . For an illustration of this, see Figure 1 .The proof proceeds by contradiction. First, we consider the action of the approximation functor as it operates on an isotopy (path of diffeomorphisms) that connects two diffeomorphisms. The first has only orientation-preserving discretizations, and the second only orientation-reversing discretizations.We then show that the image of the path under the approximation functor yields a disconnected path, as the discretization 'jumps' from the orientation preserving component to the orientation reversing component. This violates continuity. To encode the notions of orientation preserving and orientation reversing that allow for a description of nonlinear discretization theory, we use topological degree theory. This generalizes the familiar notion of orientation that uses the sign of the determinant of the Jacobian matrix.In this section and the next two we show that, although Theorem 2 precludes continuous approximation of general diffeomorphisms, stronger constraints on the diffeomorphisms allows one to sidestep the topological obstruction. In this section, in summary, we show that the obstruction to continuous approximation vanishes when the diffeomorphisms in question are assumed to be strongly monotone. Key to our positive result is the following technical result that states that the restriction of the domain and codomain of a strongly monotone diffeomorphism always yields another strongly monotone diffeomorphism. Lemma 1. Let V ⊂ X be a finite-dimensional subspace of X, and let P V : X → X be orthogonal projection onto V . Let F : X → X be a strongly monotone C 1 -diffeomorphism. Then, P V F | V : V → V is strongly monotone, and a C 1 -diffeomorphism.The proof is given in Appendix A.5.1. Lemma 1 implies that the discretization functor A lin defined in (2) is a well-defined functor from strongly monotone C 1 -diffeomorphisms of X to C 1diffeomorphisms of V . Note that the discretization functor A lin on strongly monotone C 1 diffeomorphisms may not be a continuous approximation functor in the strong sense of Definitions 7 and 8, but it is obviously a continuous approximation functor in the weak sense of Definitions 11 and 12. Therefore, we obtain that : Proposition 1. Let A lin be the discretization functor that maps F to P V F | V for each finite subspace V ⊂ X. Let D sm and B sm be categories where F : X → X and F V : V → V are strongly monotone C 1 -diffeomorphisms. Then, the functor A lin : D sm → B sm satisfies assumption (A') of a weak approximation functor in Definition 11, and is continuous in the weak sense of Definition 12.By Proposition 1, strongly monotone maps can be continuously discretized in the weak sense. Thus, we concern under what conditions, the maps, of which generalized neural operator consists, can be strongly monotone. In this subsection, we focus on bounded linear operators and Nemytskii operators (layers of neural operator will be discussed in the next subsection). The following lemma is obviously given by the definition of a strongly monotone map : Lemma 2. Let A : X → X be a linear bounded operator and satisfy ⟨Au, u⟩ ≥ c 0 ∥u∥ 2 X for some c 0 > 0. Then, A : X → X is strongly monotone.Next, assuming that X = L 2 (D; R), we define Nemytskii operator byEQUATIONwhere σ : R → R is continuous. In this case, we can show the following lemma by using [ As seen in Theorem 3 below, strongly monotone layers of neural operators do not suffer from the same topological obstruction to continuous discretization as general diffeomorphisms. We now give sufficient conditions for the layers of a neural operator to be strongly monotone, and show that these conditions imply that those are diffeomorphisms. Lemma 3. All strongly monotone layers of neural operators (F ) defined by (4) are diffeomorphisms.Also, the following theorem is proven in Appendix A.6.3. Theorem 3. Let A lin be the discretization functor that maps F to P V F | V for each finite subspace V ⊂ X. Let D smn and B smn be categories where F : X → X and F V : V → V are strongly monotone C 1 -functions of the form (4). Then, the functor A lin : D smn → B smn satisfies assumption (A), and it is continuous in the sense of Definition 8.The functor defined in Theorem 3 does not suffer from the same topological obstruction as functors for general diffeomorphisms, shown in the no-go Theorem 2.This is because when F V = P V F | V is strongly monotone, its derivative D| x F V : V → V is a strongly monotone matrix at all points x ∈ V . Therefore it is strictly positive definite (see [47, Prop. 12.3] ) and the determinant det(D| x F V ) is strictly positive. Due to this, the orientation of the finite-dimensional approximations never switch signs, and the key technique used in the proof of the no-go Theorem 2 does not apply.A straightforward condition to guarantee strong monotonicity of a neural operator layer is given in Lemma 4. Let F : X → X be a layer of neural operator that is of the form F (u) = u + T 2 G(T 1 u), where T j : X → X, j = 1, 2 are compact operators and G : X → X is a C 1 -smooth map. Assume that Fréchet derivative DG| x of G at x satisfies the following for all x ∈ X,∥DG| x ∥ X→X ≤ 1 2 ∥T 1 ∥ -1 X→X ∥T 2 ∥ -1 X→X . Then, F : X → X is strongly monotone.See Appendixes A.6.1 and A.6.2 for the proofs.Now we show an analogous result to Theorem 3, but applied to bilipschitz neural operators. Moreover, we will show that all neural operator F : X → X that are bilipschitz admit approximations that can be locally inverted using iteration for each point in their range using an iteration. Theorem 4. Let X be a Hilbert space. Then there is e ∈ X, ∥e∥ X = 1 such that the following is true: Let F : X → X be a layer of a bilipschitz neural operator. Then for all r 1 > 0 and ϵ > 0 there are a linear invertible map A 0 : X → X, that is either the identity map or a reflection operator3 x → x -2⟨x, e⟩ X e, and strongly monotone functions H k that are also layers of neural operators such that H k : X → X, H k (x) = x + B k (x), k = 1, 2, . . . , J, where B k : X → X is a compact mapping and satisfies Lip(B k ) < ϵ andEQUATIONMoreover, if F ∈ C 2 (X, X), then J = O(ϵ -2 ).The proof of Theorem 4 is in Appendix A.7.1. Theorem 4 shows that we may always decompose a bilipschitz neural operator into the composition of strongly monotone neural operator layers H j and a reflection operator A 0 . Each H j can be discretized using the continuous functor A lin from Theorem 3.If we consider the discretization (via the construction in Definition 6) using a collection of subsets S 0 (X) ⊂ S(X) such that all V ∈ S 0 (X) satisfy e ∈ V , then the operator A 0 can be discretized byA 0,V = P V • A 0 | V .These mean that if we write a bilipschitz neural operator as a sufficiently deep neural operator where each layer is either of the form Id + B j , where Lip(B j ) < 1, or a reflection operator A 0 . In either case, we may use linear discretization to approximate each layer. So, we may discretize F in a ball B X (0, R) by discretizing each layer H j and A 1 whereF = H J • • • • • H 1 • A 1 .We observe that the number of layers, J, depends on R.We have observed that operators of the form identity plus a compact term are critical for continuous discretization. This insight motivates the introduction of residual networks as approximators within the framework of finite-rank neural operators. In what follows, we assume that X is a separable Hilbert space, with an orthonormal basis φ = {φ n } n∈N . For N ∈ N, we defineE N : X → R N and D N : R N → X by E N u := (⟨u, φ 1 ⟩ X , ..., ⟨u, φ N ⟩ X ) ∈ R N . D N α := n≤N α n φ n . We note that P V N = D N E N , where P V N : X → X is the projection onto V N := span{φ n } n≤N . Using E N ,D N , we define the class of residual networks in the separable Hilbert space, with T, N ∈ N and activation function σ, asR T,N,φ,σ (X) := G : X → X : G = ⃝ T t=1 (Id X + D N • N N t • E N ),N N t : R N → R N are neural networks with activation function σ (t = 1, ..., T ) .The following theorem proves a universality result for each of the layers G, allowing us to obtain a general universality result for the entire network. The statement of the theorem requires the careful construction of a neural operator-representable function Φ. Giving a full description of Φ involves introducing a lot of technical notation, and so the presentation here in the main text leaves the details of the construction of Φ abridged. For the full statement of the theorem and definition of the notation, see Section A.7.2. Intuitively, Φ is the 'wrapping' of a fixed-point process in a neural operator. Theorem 5. Let R > 0, and let F : X → X be a layer of a bilipschitz neural operator, as in Defintion 3. Let σ be the Rectified Cubic Unit (ReCU) function defined by σ(x) := max{0, x} 3 . Then, for any ϵ ∈ (0, 1), there are T, N ∈ N and G ∈ R T,N,φ,σ (X) that has the formG = (Id X + D N • N N T • E N ) • • • • • (Id X + D N • N N 1 • E N ), such that each map (Id X + D N • N N t • E N ) is strongly monotone C 1 -diffeomorphisms on some ball and sup x∈B X (0,R) ∥F (x) -G • A(x)∥ X ≤ ϵ,where A : X → X is a linear invertible map that is either the identity map or a reflection operator x → x -2⟨x, e⟩ X e with some unit vector e ∈ X. Further, G• A : B X (0, R) → G • A(B X (0, R))is invertible, and there is some neural operator Φ : G• A(B X (0, R)) → A(B X (0, R)) so that G • A| B X (0,R) -1 = A -1 • Φ.The proof is given in Section A.7.2. Neural operator Φ becomes a better approximation of the inverse operator when it becomes deeper. Theorem 5 means that neural operators are an operator algebra of nonlinear operators that are closed in composition and when the inverse of a neural operator exists, the inverse operator can be locally approximated by neural operators.In the case when the separable Hilbert space X is the real-valued L 2 -function space L 2 (D; R), residual networks in the separable Hilbert space can be represented as residual neural operators defined by (179). See Lemma 11 for details. Then, we obtain the following Corollary 1. Let D ⊂ R d be a bounded domain, and let φ = {φ n } n∈N be an orthonormal basis in L 2 (D; R). Assume that the orthonormal basis φ include the constant function. Let RN O T,N,φ,σ (L 2 (D; R)) be the class of residual neural operators defined in (179). Then, the statement replacing X with L 2 (D; R) andG ∈ R T,N,φ,ReLU (X) with G ∈ RN O T,N,φ,ReLU (L 2 (D; R)) in Theorem 5 holds.The proof is given by a combination of Theorem 5 and Lemma 11. We note that the assumption that the orthonormal basis φ includes the constant function is satisfied if we choose φ to be a Fourier basis, which yields the Fourier neural operator, see e.g., [39, 38] .Remark 1. In this section, we have shown that residual networks in a separable Hilbert space X, defined in (10) , are universal approximators for layers of bilipschitz neural operators. Additionally, in the specific case where X = L 2 (D; R), residual neural operators, defined in Definition 9, also provide universal approximators for layers of bilipschitz neural operators. We note that the residual network we have discussed is locally invertible but not globally. By introducing invertible residual networks on Hilbert space X, defined in (166), we can similarly prove that these networks by employing sort activation functions (see [3, Section 4] ) are universal approximators for strongly monotone diffeomorphisms with compact support. Specifically, when X = L 2 (D; R), invertible residual neural operators, defined in Definition 9, are also universal approximators for strongly monotone diffeomorphisms with compact support. For further details, we refer to Appendix B. 
quantitative approximation: 
 Quantitative approximation results for neural networks, see e.g. [57] or [23] , can be used to derive quantitative error estimates for discretization operations. Let F :B X (0, r) → X be a non-linear function satisfying F ∈ Lip(B X (0, r); X), in n = 1, or F ∈ C n (B X (0, r); X), if n ≥ 2.Then, F can be discretized using neural networks in the following way: Let ε V > 0 be numbers indexed by the linear subspacesV ⊂ X such that ε V → 0 as V → X. When ⃗ ε = (ε V ) V ∈S(X) , in the sense of Definition 1, an ⃗ ε-approximation operation A N N : F → (F V ) V ∈S(X)in the ball B X (0, r) can be be obtained by defining F V = J -1 V • F V,θ • J V : V → V , where J V : V → R d is an isometric isomorphism, d = dim(V ), F V,θ : R d → R d is 
conclusion: 
 In this work, we have studied the problem of discretizing neural operators between Hilbert spaces. Many physical models concern functions R n → R, for example L 2 (R n ), the computational methods based on approximations in finite dimensional spaces should become better when the dimension of the model grows and tends to infinity. We have focused on diffeomorphisms in infinite dimensions, which are crucial to understand in generative modeling. We have shown that the approximation of diffeomorphisms leads to computational difficulties. We used tools from category theory to produce a no-go theorem showing that general diffeomorphisms between Hilbert spaces may not admit any continuous approximations by diffeomorphisms on finite spaces, even if the approximations are allowed to be nonlinear. We then proceeded to give several positive results, showing that diffeomorphisms between Hilbert spaces may be continuously approximated if they are further assumed to be strongly monotone. Moreover, we showed that the difficulties can be avoided by considering a restricted but still practically rich class of diffeomorphisms. This includes bilipschitz neural operators, which may be represented in any bounded set as a composition of strongly monotone neural operators and strongly monotone diffeomorphisms. We then showed that such operators may be inverted locally via an iteration scheme. Finally we gave a simple example on how quantitative stability questions can be obtained for discretization functors, inviting other researchers to study related questions using more sophisticated methods. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work is theoretical, does not release data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: Our work is theoretical and does not contain owners of outside assets, like code, data or models.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our work does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. 
 
 Appendix: In this section we consider the non-linear discretization theory, see [9] . Let Ω ⊂ R n be a bounded open set with smooth boundary. Let H1 (Ω) = {u ∈ L 2 (Ω) | ∇u ∈ L 2 (Ω)} and H 1 0 (Ω) = {u ∈ H 1 (Ω) | u| ∂Ω =0} be the Sobolev spaces with an integer order of smoothness and H s (Ω), s ∈ R the Sobolev space with a fractional order of smoothness [53] .For simplicity, we assume in the section that n = 1, and that Ω ⊂ R is an interval and we point out that in this case the Sobolev embedding theorem implies H 1 (Ω) ⊂ C(Ω) which significantly simplifies the constructions below.EQUATION) Let F = F (g) : H 1 0 (Ω) → H 1 0 (Ω), F (g) :x → u be the solution operator of the non-linear differential equationEQUATIONEQUATION)where ∆ is the Laplace operator operating in the t variable, where the function x(t) is a physical source term. Because the u that solves ( 12)-( 13) is the minimizer of the strictly convex and lower semi-continuous function H : X → R,EQUATIONthe Weierstrass theorem, see [50] , Theorem II.8.1, implies that ( 12)-( 13) has a unique solution. Moreover, by regularity theory for elliptic equations, see [18] , Theorem 8.13, we have that the solution u is in H 3 (Ω). Moreover, as G is convex and satisfies (11) , the function H in ( 14) is coercive, for any R > 0 there is ρ(R) > 0 such that when ∥x∥ H 1 0 (Ω) ≤ R, then ∥u∥ H 3 (Ω) ≤ ρ(R). The equation ( 12)-( 13) can be written also as an integral equationEQUATIONwhereΨ(t, t ′ ) is the Dirichlet Green's function of the negative Laplacian, that is, -∆Ψ(•, t ′ ) = δ t ′ (•), Ψ(•, t ′ )| ∂Ω = 0, and g * (u) = g(u). We can write (15) asEQUATIONwhere iH 1 0 →H 3/4 : H 1 0 (Ω) → H 3/4 (Ω) and i H 2 →H 1 0 : H 2 (Ω) ∩ H 1 0 (Ω) → H 1 0 (Ω) are compact operators and as H 3/4 (Ω) ⊂ C(Ω), the operators g * : H 3/4 (Ω) → L 2 (Ω) and Q : L 2 (Ω) → H 2 (Ω) ∩ H 1 0 (Ω) are continuous, we see that F = Id + i H 2 →H 1 • (Q • g * ) • i H 1 →H 3/4 : H 1 0 (Ω) → H 1 0 (Ω), is a layer of the neural operator. As G ∈ C m+1 , m ≥ 2, we have that F : H 1 0 (Ω) → H 1 0 (Ω) is C 1 -smooth. Moreover, as G is convex, the operator F : H 1 (Ω) → H 1 (Ω)is strongly monotone and hence by Lemma 3, F has an inverse map F -1 : H 1 (Ω) → H 1 (Ω) that is C 1 -smooth. Thus, the solution u of ( 12)-( 13) can be represented asu = F -1 (Q(x)).Let V ⊂ X = H 1 0 (Ω) be a finite dimensional space. The Galerkin methods to obtain an approximation for the solution of the boundary value problem ( 12)-( 13) using Finite Element Method. To do this, let w ∈ V be the solution of the problemEQUATIONWhen ϕ 1 , . . . , ϕ d ∈ V are a basis of the finite dimensional vector space V , and we write w = d k=1 w k ϕ j , the problem ( 17) is equivalent that (w 1 , . . . , w d ) ∈ R d satisfies the system equations d equationsEQUATIONwhereb jk = Ω ∇ϕ j (t) • ∇ϕ k (t)dt.When x ∈ V , we define the map F (g)V : V → V by setting map F (g) V (x) = w,where w ∈ V is the solution of the problem (17) . WhenF P DE = {F (g) : X → X | g(s) = dG ds (s) , G ∈ C m+2 (R) is convex and satisfies (11)},we define the map A F EM :F (g) → F (g)V for F (g) ∈ F P DE . As the the function g in the equation ( 18) is non-linear, the solution u of the continuum problem ( 15)-( 13) and the solution w of the finite dimensional problem have typically no linear relationship, even if the source x satisfies x ∈ V.EQUATION)As G is convex, the Fréchet derivative of the map F = Id + Q • g * : X → X is a strictly positive linear operator at all points x ∈ X. This, the uniform convergence (19) , and the convergence results for the Galerkin method for semi-linear equations, [49, Theorem 3.2], see also [24] , imply that in the spaceEQUATIONThese imply that A F EM :F (g) → F (g)V is an approximation operation for the function F (g) ∈ F P DE .The map F (g) V : V → V is called the Galerkin approximation of the problem ( 12)- (13) and is an example of the non-linear approximation methods. The properties of the Galerkin approximation is studied in detail in [8] , in particular sections 4.4 and 4.5.On the positive results, in Appendix A.1, we have considered nonlinear discretiation of the operators u → -∆u + g(u). To exemplify the negative result, we consider the following example on the solution operation of differential equations and the non-existence of approximation by diffeomorphic maps: We consider the elliptic (but not not strongly elliptic) problemEQUATIONwith the Dirichlet and Neumann boundary conditionsEQUATIONHere, 0 ≤ s ≤ 1 is a parameter of the coeffient function and sign(t -s) = 1 if t > s and sign(t -s) < 0 if t < s. We consider the weak solutions of ( 21) in the spaceu ∈ H 1 D,N (0, 1) := {v ∈ H 1 (0, 1) : v(0) = 0}.We can writeB s u = -D (2) t A s D (1) t u, where A s v(t) = (1 + t)sign(t -s)v(t),parametrized by 0 ≤ s ≤, are multiplication operations that are invertible operators, A s : L 2 (0, 1) → L 2 (0, 1) (this invertibility makes the equation ( 21) elliptic). Moreover, D(2) t are the operators v → d dt v with the Dirichlet boundary condition v(0) = 0 and v(1) = 0, respectively. We consider the Hilbert space X = H 1 D,N (0, 1); to generate an invertible operator G s : X → X related to (21) , we write the source term using an auxiliary function g,f (t) = Qg := - d 2 dt 2 g(t) + g(t).Then the equation,EQUATIONdefines a continuous and invertible operator,G s : X → X, G s : g → u.In fact, G s = B -1 s • Q when the domains of B s and Q are chosen in a suitable way. The Galerkin method (that is, the standard approximation based on the Finite Element Method) to approximate the equation ( 23) involves introducing a complete basis χ j (t), j = 1, 2, . . . of the Hilbert space X, the orthogonal projectionP n : X → X n := span{χ j : j = 1, 2, . . . , n},and approximate solutions of ( 23) through solvingEQUATION)This means that operator B -1 s Q : g → u is approximated by (P n B s P n ) -1 P n QP n : g n → u n , whenP n B s P n : X n → X n is invertible.The above corresponds to the Finite Element Method where the matrix defined by the operatorP n B s P n is b(s) = [b jk (s)] n j,k=1 ∈ R n×n , where b jk (s) = 1 0 (1 + t)sign(t -s) d dt χ j (t) • d dt χ k (t)dt, j, k = 1, . . . , n.Since we used the mixed Dirichlet and Neumann boundary conditions in the above boundary value problem, we see that for s = 0 all eigenvalues of the matrix b(s) are strictly positive, and when s = 1 all eigenvalues are strictly negative. As the function s → b(s) is a continuous matrix-valued function, we see that there exists s ∈ (0, 1) such that the matrix b(s) has a zero eigenvalue and is no invertible. Thus, we have a situation where all operators B -1 s Q : g → u, s ∈ [0, 1] are invertible (and thus define diffeomorphisms X → X) but for any basis χ j (t) and any n there exists s ∈ (0, 1) such that the finite dimensional approximation b(s) : R n → R n is not invertible. This example shows that there is no FEM-based discretization method for which the finite dimensional approximations of all operators B -1 s Q, s ∈ (0, 1), are invertible. The above example also shows a key difference between finite and infinite dimensional spaces. The operator A s : L 2 (0, 1) → L 2 (0, 1) has only continuous spectrum and not eigenvalues nor eigenfunctions whereas the finite dimensional matrices have only point spectrum (that is, eigenvalues). The continuous spectrum makes it possible to deform the positive operator A s with s = 0 to a negative operator A s with s = 1 in such a way that all operators A s , 0 ≤ s ≤ 1, are invertible but this is not possible to do for finite dimensional matrices. We point out that the map s → A s is not continuous in the operator norm topology but only in the strong operator topology and the fact that A 0 can be deformed to A 1 in the norm topology by a path that lies in the set of invertible operators is a deeper result. However, the strong operator topology is enough to make the FEM matrix b(s) to depend continuously on s.In this section, we give an example of the application of Theorem 2, as applied to the problem of using a neural operator to solve a linear PDE. The guiding idea is to draw inspiration from Figure 1 ; the space of diffeomorphisms on X is disconnected if X is a finite-dimensional space, but connected if it is a Hilbert space.Before constructing the path, we first construct a isotopy of diffeomorphisms in L 2 (Ω) in several stepsEQUATION)Then, define the functionEQUATION) Note that R i (t) ∈ SO(2), for t ≤ i, R i (t) = I, t ≥ i + 1, R i (t) = -I.Next, we consider the 'little ell' space, ℓ 2 := (α 1 , α 2 , . . . , ) :∞ i=1 α 2 i < ∞ .Consider the linear operator R : R → ℓ 2 → ℓ 2 on ℓ 2 given by a pair-wise coordinate representation given by the infinite dimensional 'matrix' with block structureEQUATIONFinally, define R :EQUATIONand define the related operator R :EQUATION)Proposition 3 (R, R are isotopies of smooth diffeomorphisms in ℓ 2 ). The mapsEQUATIONare isotopies of smooth diffeomorphisms in ℓ 2 and agree at t = 1.Proof. We prove that H 1 is an isotopy of smooth diffeomorphisms on ℓ 2 , as the proof of the same for H 2 is very similar.We first show that H 1 (•, t) ∈ diff(ℓ 2 ) for each t.For t < 1, R operates on vectors in ℓ 2 in only a finite number of indices. In those indices it corresponds to a rotation, and hence is a smooth diffeomorphism. When t ≥ 1, it corresponds to scalar multiplication by -1, and so is too a smooth diffeomorphism. Now we show that R(t) is continuous on . The only point where continuity may fail is at t = 1, and there it may only fail in the limit from the left. Given a v ∈ ℓ 2 , we computeEQUATION)Let k * (t) := ⌊ 1 1-t ⌋. Then for any integer i ≤ k, we have thatEQUATIONand likewise when i≥ k * (t) + 1, then R i 1 1-t = I ∈ R 2×2, and so the r.h.s. of Eqn. 31 becomesEQUATION= limEQUATION)As t → 1 -, k * (t) → ∞, and so, by standard estimates, 2∞ i=k * (t) v 2 i → 0. Hence, R(t) is continuous on ℓ 2 at t = 1.Finally, by definition at t = 1, we have thatH 1 (v, 1) = R(1) = -I = R(1) = H 2 (v, 1).Finally, we define H : R ∞ × [0, 1] → R ∞ by gluing the isotopies H 1 and H 2 together byEQUATION)Proposition 4. The function H given by Eqn. 36 is an isotopy of diffeomorphisms in ℓ 2 .Proof. Continuity of H follows from continuity of R, R, and that R(1) = R (1) .For each t ∈ [0, 1], H(•, t) ∈ diff(ℓ 2 ).A.4 Proofs from Sec. 3.1A.4.1 Proof of Theorem 2Proof. Assume that A : D → B is a functor that satisfies the property (A) of an approximation functor and is continuous. Let us consider the case when X = ℓ 2 .Let e ∈ X be a unit vector and B e : X → X be a linear map Bx = x -2⟨x, e⟩ X e. In other words, B is a diagonal matrix diag(-1, 1, . . . ) in some orthogonal basis where e is the first basis vector.By [32, Theorem 2] , see also and [45] , for an infinite dimensional Hilbert space X the space GL(X) of linear invertible maps have only one topological component in the operator norm topology and the set GL(X) is contractible and hence path-connected. This implies that there are linear maps A t : X → X, t ∈ [0, 1] such that A 0 = Id, and A 1 ∈ GL(X) is arbitrary invertible linear map and that t → A t ∈ GL(X) is continuous. Similarly, in an infinite dimensional real Hilbert space X the space O R (X) of the orthogonal operators A : X → X is path-connected in the operator norm topology, see [46, Section 4] . We recall that an orthogonal operator A : X → X is a bounded linear operator satisfying A * A = AA * = I, where A * : X → X is the adjoint (i.e., the transpose) of the operator A : X → X. Observe that in a finite dimensional space R n the set O R (R n ) has two disjoint topological components, those which determinant is 1 and those which determinant is -1, and thus the set O R (R n ) is not connected.A bounded linear operator B : X → X is called a rotation operator of the form B = e S where S : X → X is a linear, bounded, skew-symmetric operator, S * = -S, and eS = I + S + 1 2! S 2 + 1 3! S 3 + . . . . An orthogonal operator A is called a reflection if it is not a rotation.One of the fundamental reasons for the surprising property that the space O R (X) of orthogonal operators in X is path-connected is that in an infinite dimensional real Hilbert space X every orthogonal operator A : X → X can be represented as a product of two rotation operators (that is no valid in the finite dimensional spaces), that is, A = e S1 e S2 where S 1 , S 2 : X → X are skew-symmetric linear operators. Thus any operator A ∈ O R (X) can be connected to the identity operator Id : X → X via a path α : t → e tS1 e tS2 ∈ O R (X), t ∈ [0, 1], so that α(0) = Id and α(1) = A.As a motivating example on the fact that O R (X) is connected in the operator norm topology, is to consider a similar, but easier result in the strong operator topology, that is, topology generated by the evaluation maps A → A(u), u ∈ L 2 (0, 1). Observe that the strong operator topology is weaker than the operator norm topology, but in finite dimensional space those are equivalent.. So, let us next show that the operators Id and -Id are in the same topological component of O R (X) in the strong operator topology. As all infinite dimensional separable Hilbert spaces are isometric to L 2 (0, 1), let us consider the Hilbert space L 2 (R) and the orthogonal linear operators A s : L 2 (0, 1) → L 2 (0, 1), s ∈ [0, 1] that are given byEQUATIONwhere u ∈ L 2 (0, 1) and sign(t) is the sign of the real number t. Then, for every u ∈ L 2 (0, 1) the functionsa u : s → A s (u),are continuous functions a u : [0, 1] → L 2 (0, 1). As A 0 = Id and A 0 = -Id, we see thats → A s ∈ O R (L 2 (0, 1)), s ∈ [0, 1],is a continuous path in the strong operator topology, that connects the operator Id to -Id.We recall that the continuity of the function s → A s in the strong operator topology, means that for all u ∈ L 2 (0, 1) the maps → A s u ∈ L 2 (0, 1), is continuous. As for s > s 0 ∥A s u -A s0 u∥ 2 L 2 (0,1) = s s0 |2u(x)| 2 dx → 0,as s → s 0 , we see that s → A s is continuous in the strong operator topology. All maps A s : L 2 (0, 1) → L 2 (0, 1) are invertible and A 0 = Id, A 1 = -Id. This has implications e.g. for Finite Element method analysis of partial differential equations (See Appendix A.5.2).Note that when X is any infinite dimensional separable Hilbert space with real scalars, there is an isometry J : X → L 2 (0, 1) (e.g., the linear operator mapping an orthogonal basis of X to an orthogonal basis L 2 (0, 1). Then, the maps Ãs : X → X given by Ãs = J -1 • A s • J : X → X, where A s : L 2 (0, 1) → L 2 (0, 1) are given above, are continuous paths in O R (X) from Ã0 = Id : X → X to Ã1 = -Id : X → X. As discussed above, the deep result that Id and -Id can be connected by a continuous path in the operator norm topology of O R (X) is given in [32, Theorem 2] , see also [45] .Let us first warm up by proving the claim in the case when an additional assumption (B) is valid:(B) : When V ⊂ X is finite dimensional discretization maps F V of linear invertible maps F : X → X are linear invertible maps and moreover, the set S 0 (X) = S(X) consists of all linear subspaces of X.Under assumptions (A) and (B), we consider the case when A 1 = -Id and denote by F t = A t : X → X a family of linear maps such that A 0 = Id, and that t→ A t ∈ GL(X) is continuous path of operators connecting Id to A 1 = -Id. Let (X, S(X), (F t,V ) V ∈S(X) ) = A(X, F t ), that is, F t,V : V → V be the linear isomorphism that is the discretizations of the map F t : X → X.As the functor A is continuous, the map t → F t,V is a continuous map from [0, 1] to the space of linear operators endowed with the topology of uniform convergence on compact sets, c.f. limit (7) . As F t,W are linear, this implies that for all t ′ ∈ [0, 1],EQUATIONand hence the map t → F t,W is a continuous map [0, 1] → GL(W ).Let 0 < ϵ 0 < 1/2 and r > 1. Let t 0 = 0 and t 1 = 1. Using Property (A) for operators F tj , j = 0, 1, we see that there are W 0 , W 1 ∈ S 0 (X) such that for all V ∈ S 0 (X) satisfying V ⊃ W j we havesup x∈B X (0,r)∩V ∥F tj ,Wj (x) -F tj (x)∥ X < ϵ 0 , for j ∈ {0, 1}.Let W ∈ S 0 (X) be such a finite dimensional space which dimension is an odd integer and thatEQUATIONClearly, A 0 = Id satisfies A 0 (W ) = W and A 0 : W → W is an invertible linear map. Similarly, the map A 1 = -Id satisfies A 1 (W ) = W and A 1 : W → W is an invertible linear map. These observation and assumptions (A) and (B) imply that F 0,W : W → W and F 1,W : W → W are linear maps and by inequality (39) ,EQUATIONand as the dimension of W is odd, we havedet(A 0 | W ) = det(Id| W ) = 1, det(A 1 | W ) = det(-Id| W ) = -1. Let B 0,s = (1 -s)A 0 | W + sF 0,W : W → W and B 1,s = (1 -s)A 1 | W + sF 0,W : W → W . As ∥(A 0 | W ) -1 ∥ W →W = ∥(A 1 | W ) -1 ∥ W →W = 1, we see using (40) that the maps B -1 0,s : W → W and B -1 1,s : W → W are invertible and that the functions s → B 0,s ∈ GL(W ) and s → B 1,s ∈ GL(W ) are continuous matrix-valued functions which determinants does not vanish andEQUATIONHowever, as t → F t,W is a continuous maps [0, 1] → GL(W ), see (38) , we have thatEQUATIONis a continuous function [0, 1] → R which does not obtain value zero and thus has a constant sign. However, this is in contradiction with formula (41) .Next we return to the main part of the proof where we do not assume that assumption (B) is valid, but we only assume that the assumption (A) is valid. In this case, we use degree theory instead of the determinants.Next, let F t = A t : X → X be a family of linear maps such that A 0 = Id, but that A 1 (x) = B e (x) = x -2e⟨x, e⟩ X is a reflection, where e ∈ X is a unit vector, andt → A t ∈ GL(X) is continuous. Again, let (X, S 0 (X), (F t,V ) V ∈S0(X) ) = A(X, F t ), that is, F t,V : V → V be C 1 - diffeomorphisms that are discretizations of the map F : X → X. As the functor A is continuous, the map t → F t,V | B(0,R) ∈ C(B(0, R)) is a continuous map for all V ∈ S 0 (X) and R > 0. Moreover, we note that by our assumptions on B, the discretized mapF t,V : V → V , is C 1 -diffeomorphism of V .As V is a finite dimensional vector space, we can use finite dimensional degree theory and consider the degree deg(F t,V , V ∩ B X (0, r), p), that is the degree of the map F t,V : V ∩ B X (0, r) → V at the point p. We recall that when Ω ⊂ R d is open and bounded, F : Ω → R d is a C 1 -smooth function and p ∈ R d are such that p ̸ ∈ f (∂Ω) and for all x ∈ f -1 (p) the derivative Df (x) is an invertible matrix, the degree is defined to bedeg(f, Ω, p) = x∈f -1 (p) sgn(det(Df (x))),where sgn(r) is sign of a real number r. Also, the map f → deg(h, Ω, p) is defined for a continuous function h : Ω → R d by approximating h by C 1 -smooth function, see [43] , Definition 1.2.5 on details. Let us denote B W (0, r) = W ∩ B X (0, r).Let r > 0. Recall that by assumption (A), for F 0 = Id : X → X and F 1 = B e : X → X there are finite dimensional spaces V 0 ∈ S 0 (X) and V 1 ∈ S 0 (X) such that e ∈ V 0 and e ∈ V 1 and that whenEQUATIONandEQUATIONMoreover, let W ∈ S 0 (X) be such a finite dimensional linear space thatV 0 ⊂ W and V 1 ⊂ W .Observe that as e ∈ W , we can decompose W asEQUATION)and denote W ′ = {x ∈ W : ⟨x, e⟩ X = 0}. We see thatEQUATIONA 1 = B e : W ′ → W ′ , B e | W ′ = Id.Next we use the facts that F 0 (0) = Id(0) = 0 and F 1 (0) = B e (0) = 0. Let us define the mapsf 0 = F 0 | B W (0,r) , f 1 = F 1 | B W (0,r) , f 0,W = F 0,W | B W (0,r) , f 1,W = F 1,W | B W (0,r) .that are C 1 -smooth maps B W (0, r) → W . Moreover, for j = 0, 1, we have f 0 (∂B W (0, r)) = ∂B W (0, r) and f 1 (∂B W (0, r)) = ∂B W (0, r). Let p j := f j,W (0). Then, by (43) and (44) we haveEQUATIONand dist(f j,W (x), p j ) ≥ 1 2 r, for all x ∈ ∂B W (0, r).This implies thatEQUATIONThen, by [43] , Definition 1.2.5, the formula (48) implies that deg(f j,W , B W (0, r), p j ) = deg(f j , B W (0, r), p j ).Moreover, as (46) , and p j ∈ B W (0, r), we haveF 1 = A 1 = B e satisfiesdeg(f 0 , B W (0, r), p 0 ) = deg(Id, B W (0, r), p 0 ) = sgn(det(Id)) = 1,and deg(f 1 , B W (0, r), p 1 ) = deg(A 1 , B W (0, r), p 1 ) = sgn(det(B e | W )) = -1.Moreover, by our assumptions on B, the discretized maps F j,W : W → W , j = 0, 1 are C 1diffeomorphism. Hence, the inverse image F -1 j,W ({p j }) is a set containing only one point that coincides with f -1 j,W ({p j }) (that in fact is the set {0}). This and the above show that for any R > r, we haveEQUATION= deg(f j , B W (0, r), p j ) = (-1) j .Let us now consider the mapsF t,W : W → W where t ∈ [0, 1], that are C 1 -diffeomorphism f t,W : W → W . As t → F t = A t is a continuous map [0, 1] → GL(X) and A : D → B is a continuous functor, the map t → F t,W is a continuous map [0, 1] → C(B W (0, r); W ).Let us consider t ∈ [0, 1] and denote p t = F t,W (0). As F t,W :W → W is a C 1 -diffeomorphism, the set F t,W (B W (0, r)) is open and hence there is ϵ > 0 such thatEQUATIONThis implies that dist W (p t, F t,W (∂B W (0, r))) > 5ϵ.AsEQUATIONIn particular, this implies thatEQUATIONand thus p t = F t,V (0) and p t = F t,V (0) are in the ball B W (p t, 5ϵ) and hence by (51) these points are in the same topological component of by W \ F t,W (∂B W (0, r)). Hence, it follows from [43] , Theorem 1.2.6 (5) , that for |t -t| < δ, we haveEQUATIONNext, we observe that by (53) , for anyEQUATIONInequalities ( 52), ( 54) and ( 56) imply that for anyEQUATION> inf y∈∂B W (0,r) dist W (p t1 , F t,W (y)) -ϵ > inf y∈∂B W (0,r) dist W (p t, F t,W (y)) -2ϵ -ϵ = dist W (p t, F t,W (∂B W (0, r))) -3ϵ -ϵ > 5ϵ -3ϵ = 2ϵ.As t → F t,W is a continuous map [0, 1] → C(B W (0, r); W ) and the formula ( 57) is valid, it follows from [43] , Theorem 1.2.6 (3) that for t 1 and t 2 satisfying |t 1 -t| < δ and |t 2 -t| < δ, we haveEQUATIONThis and (55) imply thatdeg(F t1,W , B W (0, r), p t2 ) = deg(F t,W , B W (0, r), p t2 ) (59) = deg(F t,W , B W (0, r), p t),In particular, whenEQUATION)As t ∈ [0, 1] is above arbitrary, this implies that the functionEQUATIONis a continuous integer valued function on [0, 1] and thus it is constant on the interval t ∈ [0, 1]. However, by (49) , g(0) = 1 is not equal to g(1) = -1 and hence g(t) can not be constant function on the interval t ∈ [0, 1]. This contradiction shows that the required functor does not exists.A.5 Proofs from Sec. 3.2Proof. Let F : X → X be strongly monotone and C 1 -diffeomorphism. It is obvious that strongly monotonicity implies that strictly monotonicity.We first show that the strongly monotonicity implies that the coercivity. Indeed, by the strongly monotonicity we have⟨F (x) -F (0), x -0⟩ X ≥ α∥x∥ 2 X , which implies that, as ∥x∥ X → ∞, ⟨F (x), x ∥x∥ ⟩ X ≥ ⟨F (0), x ∥x∥ ⟩ X + α∥x∥ X → ∞.Therefore, F : X → X is coercive.Let us consider the operatorF V := P V F | V : V → V. From the definitions, F V : V → V is C 1 andstrongly monotones, and the (Fréchet) derivativeDF V | v at v ∈ V is given by DF V | v = P V (DF | v )| V, which is linear and continuous operator from V to V . Since F V : V → V is C 1 and strongly monotone, it is hemicontinuous, strictly monotones, and coercive. By the Minty-Browder theorem [10, Theorem 9.14-1], F V : V → V is bijective, and then its inverseF -1 V : V → V exists. Next, let v ∈ V . As F V : V → V is strongly monotones, we estimate that for all h ∈ V , ⟨ F V (v + ϵh) -F (x) ϵ , (x + ϵh) -x ϵ ⟩ X = 1 ϵ 2 ⟨F (x + ϵh) -F (x), (x + ϵh) -x⟩ X ≥ α ϵ 2 ∥(x + ϵh) -x∥ 2 X = α∥h∥ 2 X .Taking ϵ → +0, we have that⟨DF V | v (h), h⟩ X ≥ ∥h∥ 2 X . Therefore, DF V | v : V → V is injective for all v ∈ V . Then, it is bijective because V is now finite dimensional. By the inverse function theorem, the inverse F -1 V : V → V is C 1 .Let us consider the elliptic (but not stongly elliptic) problemEQUATIONwith the boundary conditionsEQUATION)The FEM (Finite Element Method) matrix corresponding to this problem is [p jk (s)] n j,k=1 , wherep jk (s) = 1 0 sign(t -s) d dt χ j (t) • d dt χ k (t)dt, j, k = 1, . . . , mand χ j (t), j = 1, 2, . . . are a complete basis of the Hilbert space H 1 D,N (0, 1) := {v ∈ H 1 (0, 1) : v(0) = 0} that correspond to the mixed Dirichlet and Neumann boundary conditions. Note that in the FEM approximation we use only a finite subset of the complete basis of the Hilbert space. Note for u in the canonical domain of the above problem that makes the operator appearing in the above equation selfadjoint (the Friedrichs extension), that is, forEQUATIONEQUATIONt u, where A s is the multiplication with the function sign(t -s), see (37) , and D EQUATIONEQUATIONEQUATIONObserve that Dt and D(2)t have the inverse operatorsEQUATION(2)EQUATIONthat map (DEQUATIONt ). The eigenvalues of the matrix s → [p jk (s)] n j,k=1 change from positive values to negative values when s moves from 0 to 1. Thus, for some value s ∈ (0, 1) the matrix [p jk (s)] n j,k=1 has a zero eigenvalue and is no invertible even though all maps D(1)EQUATIONt , and A s : L 2 (0, 1) → L 2 (0, 1) are invertible. In particular, there are no finite FEM basis in where the Galerkin discretizations of all operators P s , s ∈ [0, 1] are invertible.Let us consider even simpler example: Similarly to the above, if we consider the Galerkin discretizations of the operator A s : L 2 (0, 1) → L 2 (0, 1), we see that the Galerkin matrix corresponding toA s : L 2 (0, 1) → L 2 (0, 1) is [a jk (s)] n j,k=1, wherea jk (s) = 1 0 sign(t -s)ψ j (t) • ψ k (t)dt, j, k = 1, . . . , m,and ψ j (t), j = 1, 2, . . . are a complete basis of the Hilbert space L 2 (0, 1). Again, we see that the eigenvalues of the matrix s → [a jk (s)] n j,k=1 change from positive values to negative values when s moves from 0 to 1. Thus, for some value s ∈ (0, 1) the matrix [a jk (s)] n j,k=1 has a zero eigenvalue and is no invertible even though all maps A s : L 2 (0, 1) → L 2 (0, 1) are invertible. Thus there are no finite basis in where the Galerkin discretizations of all operators A s , s ∈ [0, 1] are invertible. Proof. From the assumption, it holds that∥DF | x -I∥ X→X ≤ ∥T 1 ∥ X→X ∥DG| x ∥ X→X ∥T 2 ∥ X→X ≤ 1 2 .Then, by the mean value theorem, there is 0 < t < 1 such that⟨F (x 1 ) -F (x 2 ), x 1 -x 2 ⟩ X = ∂ ∂t ⟨F (x 1 + t(x 2 -x 1 )) -F (x 2 ), x 1 -x 2 ⟩ X = ∂ ∂t ⟨DF x1+t(x2-x1) (x 2 -x 1 ), x 1 -x 2 ⟩ X ≥ 1 2 ∥x 1 -x 2 ∥ 2 X .These imply that F : X → X is strongly monotone.Proof. Observe that strongly monotone neural operators are coercive, that is,EQUATIONand therefore F : X → X is surjective by Browder-Minty theorem [28, Thm. 2.1], see also [14] considerations for neural operators. Moreover, the derivatives DF | x are linear strongly monotone operators for all x ∈ X, and therefore injective. Observe that the derivative is a linear operator of the form DF | x = I + T 2 • DG| T1x • T 1 : X → X, and as T 1 and T 2 are compact and DG| T x is bounded we see that DF | x is a Fredholm operator of index zero. Observe that⟨DF | x v, v⟩ X = 1 h lim h→0 ⟨ F (x + hv) -F (x) h , (x + hv) -x⟩ X ≥ α∥v∥ 2 X ,where h ∈ R and v ∈ X, and hence DF | x : X → X is an injective linear operator. As DF | x is a Fredholm operator of index zero, this implies that the derivative DF | x : X → X is a bijection. As the Hilbert space X can be identified with it dual space and the operator F : X → X is continuous and hence hemi-continuous, it follows from [28] , Theorem 3.1, that the map F : X → X is a homeomorphism. As it is C 1 -smooth and its derivative is bijective operator DF | x : X → X at all x ∈ X, it follows that F : X → X is a C 1 -smooth diffeomorphism.Proof. To show the well-definedness of the discretization functor A lin , we need to show that, for each strongly monotone C 1 -function F : X → X that is of the form (4),F V = P V F | V : V → V is still strongly monotone C 1 -smooth of the form (4). This is given by Lemma 1. Moreover, such functions are C 1 -smooth diffeomorphisms.To verify assumption (A), let r, ϵ > 0, and let F : X → X be a strongly monotone diffeomorphisms that is of the formF = Id + T 2 • G • T 1 : X → X where T 1 : X → X and T 2 : X → X are compact linear operators, and G : X → X is such that G ∈ C 1 (X). Since T 2 • G • T 1 is a compact mapping, there is a finite-dimensional subspace V ⊂ X, depending on ϵ > 0 such that sup x∈B X (0,r) ∥(Id -P V )T 2 G(T 1 x)∥ X ≤ ϵ, which implies that sup x∈B X (0,r)∩V ∥F V (x) -F (x)∥ X = sup x∈B X (0,r)∩V ∥(Id -P V )T 2 G(T 1 x)∥ X ≤ ϵ.To prove the continuity in the sense of Definition 8 let r > 0 and let V ∈ S 0 (X). Assume thatlim j→∞ sup x∈B X (0,r) ∥F (j) (x) -F (x)∥ X = 0,where F (j) and F are strongly monotone diffeomorphisms F : X → X that are of the form (4). Then, we see that, as j → ∞,sup x∈V ∩B X (0,r) ∥F (j) V (x) -F V (x)∥ X ≤ sup x∈V ∩B X (0,r) ∥P V F (j) (x) -P V F (x)∥ X ≤ sup x∈V ∩B X (0,r) ∥P V ∥ op ∥F (j) (x) -F (x)∥ X ≤ sup x∈B X (0,r) ∥F (j) (x) -F (x)∥ X → 0.A.7 Proofs from Sec. 3.5In the proof of Theorem 4, we prove first few auxiliary lemmas Lemma 5. A layer of a neural operator F : X → X is surjective. In particular, if F : X → X is bilipschitz, then F : X → X is a homeomorphism.Proof. By formula (4) a layer of neural operator F : X → X is of the formF (u) = u + T 2 G(T 1 u)where T 1 : X → X and T 2 : X → X are compact linear operators and G : X → X is a function inC 1 (X). Let c 0 , c 1 > 0 be such that ∥G∥ L ∞ (X) ≤ c 0 and Lip X→X (G) ≤ c 1 .Let p ∈ X and R 0 > ∥T 2 ∥ X→X c 0 + ∥p∥ X . Then K p : X → X, defined by the formula WhenK p (x) = -T 2 • G • T 1 (x) + p, x ∈ X,EQUATIONR 0 ), 0) be the infinite dimensional (Leray-Schauder) degree of the operator Id -K 0 : X → X in the set B(0, R 0 ) with respect to the point p, see [43, Definition 2.2.3]. Let K p;t : X → X be the non-linear compact operators that depend on the parameter t ∈ [0, 1], obtained by multiplying K p (x) by the number t, that is,EQUATIONAs ∥K p;t (x)∥ X ≤ ∥T 2 ∥ X→X c 0 + ∥p∥ X < R 0 for all t ∈ [0, 1] and x ∈ X, we see thatEQUATIONThen by the homotopy invariance of the Leray-Schauder degree, see [43, Theorem 2.2.4(3)], the functionEQUATIONis a constant function. Moreover, by [43, Theorem 2.2.4(1)],deg(Id -K p , B(0, R 0 ), 0) = deg(I -K p;1 , B(0, R 0 ), 0) = d(1) = d(0) = deg(I, B(0, R 0 ), 0) = 1.By [43, Theorem 2.2.4(2)], this implies that the equationx = K p (x), or equivalently, x + T 2 • G • T 1 (x) = p,has a solution x ∈ B X (R 0 ). As p ∈ X was above arbitrary, this implies thatF = Id + T 2 • G • T 1 : X → X is surjection.Finally, if F : X → X is bilipschitz, it is a bijection and its inverse function is Lipschitz function, and thus F : X → X is a homeomorphism.The next lemma shows the existence of a (finite-dimensional) orthogonal subspace so that for each compact operator, projection onto the subspace is a perturbation of the identity under either pre or post composition. Lemma 6. Let T 1 , T 2 : X → X be compact operators and h > 0. There is a finite dimensional space W ⊂ X such that for the orthogonal projector P W : X → X it holds thatEQUATION)Proof. We use the singular value decomposition of compact operators: We can writeT ℓ x = ∞ p=1 ω ℓ,p ⟨x, ψ ℓ,p ⟩ϕ 1,p ,where ψ ℓ,p and ϕ ℓ,p are orthogonal families in X and ω ℓ,p ≥ 0 satisfy ω ℓ,p+1 ≤ ω ℓ,p and ω ℓ,p → 0 as p → ∞. For all h > 0 we can choose P > 0 such that ω ℓ,p < h when p ≥ P . ThenEQUATIONwe see that if W ⊂ X is a linear subspace such that V ⊂ W then ( P p=1 ω ℓ,p ⟨•, ψ ℓ,p ⟩ϕ 1,p ) • P W = P p=1 ω ℓ,p ⟨•, ψ ℓ,∥T ℓ -T ℓ • P W ∥ X→X < h, ∥T ℓ -P W • T ℓ ∥ X→X < h,and moreover,∥T ℓ -P W • T ℓ • P W ∥ X→X < h.LetEQUATIONObserve that for w ∈ W and v ∈ W ⊥ we haveEQUATIONandEQUATIONEQUATION)This means thatF W = Id W ⊥ ⊕ F W | W , where F W | W : W → W isa function which maps the finite dimensional vector space W to itself.The lemma below shows that given an F and ϵ > 0, we may perturb it by a Lipschitz term B so it becomes the operator F W .Lemma 7. For any ϵ > 0, there is a finite dimensional space W ⊂ X such that Lip X→X (F -F W ) < ϵ and for any ball B X (R) ⊂ X, R > 0, the maps F : B X (R) → X andF W : B X (R) → X satisfy ∥F -F W ∥ L ∞ (B X (R)) < 1 2 (1 + R)ϵ.Proof. Let us choose a finite dimensional space W ⊂ X so that Lemma 6 is valid withh = 1 4(1 + ∥G∥ C 1 (X) ))(1 + ∥T 1 ∥ X→X )(1 + ∥T 2 ∥ X→X ) ϵ.The right hand side of ( 72) is chosen so thatLip X→X (F -F W ) = Lip(P W • T 2 • G • T 1 • P W (x) -T 2 • G • T 1 ) ≤ Lip(P W • T 2 • G • T 1 • P W -P W • T 2 • G • T 1 (x)) +Lip(P W • T 2 • G • T 1 (x) -T 2 • G • T 1 ) ≤ ∥T 2 ∥ X→X Lip(G| X→X )∥T 1 (I -P W )∥ X→X + ∥(I -P W )T 2 ∥ X→X Lip(G| X→X )∥T 1 ∥ X→X ≤ 1 2 ε,and∥F -F W ∥ L ∞ (B(0,R)) = sup x∈B(0,R) ∥P W • T 2 • G • T 1 • P W (x) -T 2 • G • T 1 (x)∥ X ≤ sup x∈B(0,R) ∥P W • T 2 • G • T 1 • P W (x) -P W • T 2 • G • T 1 (x)∥ X + sup x∈B(0,R) ∥P W • T 2 • G • T 1 (x) -T 2 • G • T 1 (x)∥ X ≤ ∥T 2 ∥ X→X Lip(G| B(0,∥T1∥ X→X R) )∥T 1 (I -P W )∥ X→X • R +∥(I -P W )T 2 ∥ X→X ∥G∥ L ∞ (B(0,∥T1∥ X→X R)) ≤ 1 4 (R + 1)ϵ.Lemma 8. For any ϵ > 0, there are finite dimensional space W ⊂ X and (possibly non-linear) functions B : X → X and B : X → X such that Lip(B) < ϵ, Lip( B) < ϵ.(81) andEQUATIONF = (Id + B) • F W : X → X.(83) Moreover, B : X → X is a compact non-linear operator of the formEQUATIONwhere F 1 , F 2 , F 3 , F 4 : X → X are functions in C 1 (X) and B : X → X is of the same form. In addition, the operator Id + B : X → X and Id + B : X → X are layers of neural operators.Proof. Below, let C 0 > 0 be such thatEQUATIONand let W be the space in Lemma 6 withh = 1 4(1 + ∥G∥ C 1 (X) ))(1 + ∥T 1 ∥ X→X )(1 + ∥T 2 ∥ X→X ) ϵ. By Lemma 5, F (x) = x + T 2 G(T 1 (x))is by an invertible function F : X → X. Thus we can define a non-linear operatorEQUATION)Then for all x, y ∈ X∥R(x) -R(y)∥ X ≤ ∥(Id -P W )T 2 ∥ X→X ∥G∥ Lip(X,X) ∥T 1 ∥ X→X ∥x -y∥ X +∥T 2 ∥ X→X ∥G∥ Lip(X,X) ∥T 1 (Id -P W )∥ X→X ∥x -y∥ X ≤ 1 2C 0 ϵ∥x -y∥ X .and thus,∥R • (Id + T 2 • G • T 1 ) -1 (x) -R • (Id + T 2 • G • T 1 ) -1 (y)∥ X ≤ 1 2 ϵ∥x -y∥ X .This implies that F W : X → X satisfies for all x, y ∈ X∥(F W • (F ) -1 -Id)(x) -(F W • (F ) -1 -Id)(y)∥ X ≤ 1 2 ϵ∥x -y∥ X ,and thusEQUATIONandEQUATIONMoreover, for all x, y ∈ XEQUATION) Let c 0 > 0 be such that ∥G∥ L ∞ (X) ≤ c 0 . Let R 0 > 0. As ∥T 2 • G∥ L ∞ (X)≤ ∥T 2 ∥c 0 , finite dimensional degree theory, see using [43, Theorem 1.2.6] , as above thatEQUATIONwhenR 1 > R 0 + ∥T 2 ∥c 0 ≥ R 0 + ∥P W • T 2 • G • T 1 • P 2 ∥ L ∞ (X). As R 0 > 0 above is arbitrary, formula (93) implies that F W : X → X is surjective. Thus, we have shown that F W : X → X is a bijective bilipschitz map.Similarly to the above, by replacing (85) by ( 92) and changing the roles of F W and F , we see that for x, y ∈ XEQUATIONEQUATION)The above implies thatEQUATIONEQUATIONwhereEQUATION)Next we use that(Id + H) -1 = (Id + H) -1 • (Id + H -H) = Id -(Id + H) -1 • H, so that (Id + T 2 • G • T 1 ) -1 = Id -(Id + T 2 • G • T 1 ) -1 • (T 2 • G • T 1 ).Observe that by ( 86)B =F W • (F ) -1 -Id (98) =(P W • T 2 • G • T 1 • P W -T 2 • G • T 1 ) • (Id + T 2 • G • T 1 ) -1 =(P W • T 2 • G • T 1 • P W -T 2 • G • T 1 ) -(P W • T 2 • G • T 1 • P W -T 2 • G • T 1 ) • (Id + T 2 • G • T 1 ) -1 • (T 2 • G • T 1 ),and as P W : X → X is a finite rank operator and T 2 : W → W is a compact linear operator, we see that B : X → X is a compact non-linear operator of the formB = P W • F 1 • P W + T 2 • F 2 • P W + P W • F 3 • T 1 + T 2 • F 4 • T 1 ,whereF 1 , F 2 , F 3 , F 4 : X → X are functions in C 1 (X).Moreover, similarly to (86), we see thatF • (F W ) -1 (99) =(Id + T 2 • G • T 1 ) • (Id + P W • T 2 • G • T 1 • P W ) -1 =Id + (T 2 • G • T 1 -P W • T 2 • G • T 1 • P W ) • (I + P W • T 2 • G • T 1 • P W ) -1 ,and henceB =F • (F W ) -1 -Id (100) =(T 2 • G • T 1 -P W • T 2 • G • T 1 • P W ) • (Id + P W • T 2 • G • T 1 • P W ) -1 =(T 2 • G • T 1 -P W • T 2 • G • T 1 • P W ) -(T 2 • G • T 1 -P W • T 2 • G • T 1 • P W ) • (Id + P W • T 2 • G • T 1 • P W ) -1 • (P W • T 2 • G • T 1 • P W ),and again, as P W : X → X is a finite rank operator and T 2 : W → W is a compact linear operator, we see that B : X → X is a compact non-linear operator of the formB = P W • F1 • P W + T 2 • F2 • P W + P W • F3 • T 1 + T 2 • F4 • T 1 ,where F1 , F2 , F3 , F4 : X → X are functions in C 1 (X). For an infinite dimensional Hilbert space X there is a linear isomorphism J : X → X × X, as the cardinality of Hilbert basis of the space X is the same as the cardinality of the Hilbert basis of X × X, see [26, Theorem 3.5] . Then, by writing the isomorphism J as J(x) = (J 1 (x), J 2 (x)) ∈ X × X, we see thatB(x) = (( P W T 2 ) • J) • (J -1 • F1 F3 F2 F4 • J) • (J -1 • P W T 1 )(x),that is a composition of a linear compact operator X → X, a non-linear operator X → X, and a compact operator X → X. Hence we see that Id + B, and similarly Id + B, are layers of neural operators.Now we present our Proof of Theorem 4.Proof. Let F (x) = x + T 2 G(T 1 (x)). Without loss of generality, we may assume that 0 < ϵ < min(∥T 2 ∥ X→X C 1 ∥T 1 ∥ X→X , 1 2 ). By ( 94)EQUATIONandEQUATIONwhich implies thatEQUATIONis a strongly monotone operator. Similarly, we see thatEQUATIONis a strongly monotone operator.Recall that F W maps W ⊥ to itself and it is equal to the identity map in W ⊥ , and moreover F W can be decomposed according to the formula (78). Thus we can writeF W = Id W ⊥ ⊕ (P W • F W • P W ) : W ⊥ ⊕ W → W ⊥ ⊕ W.Below, let us identify W with R n to clarify notations. We denoteEQUATIONBy ( 91) and ( 92) there are c 1 , c 0 > 0 such thatEQUATION)Define 4 for 0 < t ≤ 1EQUATION)For t = 0, we defineEQUATION)where Df | y : R n → R n is the derivative of the map f at the point y, that is considered as a linear map (or a matrix), and we denote its value at vector v by Df | y (v) = Df | y v.Then f 1 (x) = f (x) andEQUATIONand as1 t c 0 |tx -ty| ≤ 1 t |f (tx) -f (ty)| ≤ 1 t c 1 |tx -ty|, we have c 0 |x -y| ≤ |f t (x) -f t (y)| ≤ c 1 |x -y|.Below, letEQUATIONandEQUATIONwhere r 1 appears in the claim of Theorem 4. Observe that then f 0 (B R n (0, r 1 )) ⊂ B R n (0, R 1 ).As f ∈ C 2 (R n , R n ), we have by Taylor's series (with the remainder written in the integral form)EQUATIONThus,EQUATION)Observe that for all x ∈ X,EQUATIONHere, for x, yEQUATIONwhereEQUATION)Let us choose t 1 ∈ (0, 1) such thatEQUATIONEQUATION)Below, we will denoteEQUATION)Next we consider operators f t with t > 0.Let us next consider t 2 , t 3 , . . . , t m+1 ∈ (0, 1] such that t 2 > t 1 , t m+1 = 1, and t k+1 > t k .We have|f t (0)| = t|f (0)| = tR 0 ,and Lip(f t ) ≤ c 1 . Hence,EQUATIONMoreover, for z = f (0) we have f -1 t (z) = 0 and |z| = R 0 , and as the Lip(fEQUATION)Also, as we can writeEQUATIONwe have fort k , t k+1 ∈ (0, 1], k ≥ 1, (f t k+1 -f t k )(x) = 1 t k+1 f (t k+1 x) - 1 t k f (t k x) -( 1 t k+1 - 1 t k )f (0) + (t k+1 -t k )f (0) = ( 1 t k+1 - 1 t k )f (t k+1 x) + 1 t k (f (t k+1 x) -f (t k x)) -( 1 t k+1 - 1 t k )f (0) + (t k+1 -t k )f (0) = ( t k -t k+1 t k ) 1 t k+1 f (t k+1 x) + 1 t k (f (t k+1 x) -f (t k x)) -( 1 t k+1 - 1 t k )f (0) + (t k+1 -t k )f (0) = ( t k -t k+1 t k ) 1 t k+1 f (t k+1 x) + (t k+1 - 1 t k+1 )f (0) + 1 t k (f (t k+1 x) -f (t k x)) -( t k -t k+1 t k )(t k+1 - 1 t k+1 )f (0) -( 1 t k+1 - 1 t k )f (0) + (t k+1 -t k )f (0) = ( t k -t k+1 t k )f t k+1 (x) + 1 t k (f (t k+1 x) -f (t k x)) + C k ,whereEQUATIONEQUATIONTo analyze the above, we denoteEQUATIONMoreover, forEQUATIONand for all x ∈ R n ,EQUATIONMoreover,EQUATIONEQUATIONLetEQUATIONHence by (128), forEQUATIONBy ( 122) and (129), it hold that of all x ∈ R nEQUATIONEQUATIONLet R 2 ≥ R 1 and define for k ≥ 1 a function that is a convex combination of the identity map Id and the mapEQUATIONEQUATIONwhereEQUATIONEQUATION)For k ≥ 1 we have t k ≥ t 1 , and thusEQUATIONThen, when the conditionf t k-1 (x) ∈ B R n (0, R 2 )it holds, we see thatH t k ,t k-1 • f t k-1 (x) = f t k-1 (x) + ϕ(f t k-1 (x))(f t k (f -1 t k-1 (f t k-1 (x))) -f t k-1 (x)) = f t k-1 (x) + 1 • (f t k (f -1 t k-1 (f t k-1 (x))) -f t k-1 (x)) = f t k (f -1 t k-1 (f t k-1 (x))) = f t k (x) ∈ B R n (0, R 2 ). Observe that when x ∈ B R n (0, R 0 ) then f t0 (x) ∈ B R n (0, R 1 ) ⊂ B R n (0, R 2 ). Next, we denote t 0 = 0. Hence, by using induction, we see that for all j = 1, 2, . . . , m + 1 it holds thatH tj ,tj-1 • H tj-1,tj-2 • H t1,t0 • f t0 (x) = f tj (x), and H tj ,tj-1 • H tj-1,tj-2 • H t1,t0 • f t0 (x) = f tj (x) ∈ B R n (0, R 2 ).We recall that above we chose t 1 > 0 such thatEQUATIONWe now choose m so that there aret k , k = 2, 3, . . . , m + 1 satisfying |t k -t k-1 | < 1 m and 1 m < ε t 1 c1 c0 (10 + R 2 ) + 6 R2 ( c1 c0 + 1)R 0 .This means that we can chooseEQUATIONThen,EQUATIONWe define the mapα(t) = f t | K , for 0 < t ≤ 1, Df | 0 , for t = 0, is a continuous map [0, 1] → C 1 (R n ; R n ).The space W is a finite dimensional Euclidean space, and let GL(W ) be the set of linear diffeomorphisms of it, that is, invertible linear operators A : W → W . We consider GL(W ) with the topology given by the operator norm of linear maps. Then GL(W ) is a topological space with two path connected components -those matrices which have a positive determinant and those having a negative determinant.The above implies that Df | 0 can be connected in GL(R n ) either to a matrix B where B is either the identity map, or the diagonal matrix diag(-1, 1, 1, . . . , 1),with a continuous pathβ s ∈ GL(W ), s ∈ [0, 1] with Df | 0 = β 1 and B = β 0 .We need to consider the pathβ s in GL(W ) from β 1 = f 0 = Df | 0 to matrix β 0 = B ∈ O(n).There are some explicit formulas in literature with relatively complicated formulas, see [41] . However, let us next consider estimates using a non-optimal but relatively explicit path. To do this we start from the PU-decomposition of the matrix β 1 = Df | 0 , that we denote β 1 = P U where P = diag (σ 1 , . . . , σ n ) is positive matrix (given in a suitable basis) and U is an orthogonal matrix. Then, we consider the path t → P t U , where t ∈ [0, 1] andP t = P 1-t = exp((1 -t) log(P )).This is a path from the matrix Df | 0 = P U to the matrix U . Observe thatP t2 U = P t1-t2 P t1 U,whereP t1-t2 = diag (σ t1-t2 1 , . . . , σ t1-t2 n ),and when |t 1 -t 2 | ≤ 1, by mean value theorem we have∥P t1-t2 -I∥ ≤ max j=1,...,n (σ t1-t2 j -1) ≤ max j=1,...,n (σ j , σ -1 j ) • ( max j=1,...,n | log σ j |) • |t 1 -t 2 | ≤ (1 + c 1 + c -1 0 ) 2 • |t 1 -t 2 |.That is, using the norm of Df | 0 and the norm of inverse matrix (Df | 0 ) -1 we can bound the length of the path t → P t U , t ∈ [0, 1] (in the operator norm). After this, we need to consider the path fromU to B = Id k × (-Id n-k ) in O(W ).For this, we could use the structure of the Lie group O(n), n = dim(W ). By [15, Thm. 11.3.3] , any matrix U in O(n) can be written as a tensor product of elements in O(2) and possibly an operator ±Id : R → R, that is, in a suitable orthogonal basis a matrix U ∈ O(n) is a block diagonal matrix of 2 × 2 matrixes in O(2) and one or two 1 × 1 matrices ±Id, that is,U =          R 1 . . . R k 0 0 ±1 . . . ±1         , where the matricesR 1 = R 1 (ϑ 1 ), ..., R k = R 1 (ϑ k ) are 2-by-2 rotation matrices in SO(2),Using this, we find a path formU (s), s ∈ [0, 1] either to a matrix Id k × (-Id n-k ), U (s) =          R 1 (sϑ 1 ) . . . R k (sϑ k ) 0 0 ±1 . . . ±1          .Observe that due the block diagonal form of this matrix, the operator norm satisfies∥U (s 2 ) -U (s 1 )∥ R n →R n ≤ max j=1,2...,k ∥R j (s 2 ϑ j ) -R j (s 1 ϑ j )∥ R 2 →R 2 ≤ C * |s 2 -s 1 |,where C * is an absolute constant (that does not depend on the dimension n). We obtain the path s → β s by concatenating the paths from P U to U in GL(n) and from U to B in SO(n). The length of the obtained path β in the operator norm metric can be estimated and it is boundedC(1 + c 1 + c -1 0 ) 2 plus a constant. Moreover, the path s → β s can be decomposed to a product of (C(1 + c 1 + c -1 0 ) 2 + C)ε -1matrices of the form Id + B j where ∥B j ∥ ≤ ε. Summarising the above analysis, we see that J can bounded byEQUATIONWe recall that hereEQUATIONradii R 0 and R 1 are given in formulas (110) and ( 111) and c 0 and c 1 are the bi-Lipschitz constants of f , see (106).Note that c 0 , c 1 , C > 0 are constants independent of ϵ, while R 0 , R 1 , R 2 > 0 depends on |f (0)|. We see that byf = P W F | W . |f (0)| ≤ ∥F (0)∥ X ,thus, the upper bounds of R 0 , R 1 , R 2 are independent of ϵ. By choosing t 1 asEQUATIONHere, if we assume that F ∈ C 2 (X, X), we see that∥f ∥ C 2 (B R n (0,R1);R n ) ≤ ∥F ∥ C 2 (W ;X) ≤ ∥F ∥ C 2 (X;X) < ∞,Thus, we obtainJ = O(ε -2 ).Now we can finish the proof of Theorem 4. Denoteft = f t ⊕ Id W ⊥ , Ht1,t0 = H t1,t0 ⊕ Id W ⊥ βs = β s ⊕ Id W ⊥ and A 0 = B ⊕ Id W ⊥ .Using these notations, we can write F (x) asEQUATIONwhere t j and s j are chosen so that 0 = t 0 < t 1 < . . . t m < t m+1 = 1 and 0 = s 0 < s 1 < . . . s m < s m+1 = 1 and that the Lipschitz constant of the maps Htj+1,tj -Id and βsj • β-1 j-1 -Id are less that ϵ.F W = Id + P W • T 2 • G • T 1 • P W : X → X.Observe that by writing x ∈ X in the form x = x 0 + x 1 , where x 0 = (I -P W )x and x 1 = P W x, we see thatEQUATIONHence, (F W ) -1 : X → X can be written asEQUATIONand thus (F W ) -1 is a layer of a neural operator by definition. Similarly, asf = F W = Id X + P W • (F W -Id W ) • P W : X → X.andEQUATIONfor 0 < t ≤ 1, and we see as above that f t and f -1 t are neural operators. Similarly, we see that β-1 s and β-1 s are neural operators. Hence, all factors in the product (152) are (strictly monotone) neural operators.Theorem 6 (Theorem 5 in the main text). Let X be a separable Hilbert space, and let φ = {φ n } n∈N be an orthonormal basis in X. Let δ ∈ (0, 1), and let R > 0, and let F : X → X be a layer of a bilipschitz neural operator, as in Def. 3. Then, for any ϵ ∈ (0, 1), there are T, N ∈ N and G ∈ R T,N,φ,ReLU (X) that has the formG = (Id X + D N • N N T • E N ) • • • • • (Id X + D N • N N 1 • E N ), such that sup x∈B X (0,R) ∥F (x) -G • A(x)∥ X ≤ ϵ,where A : X → X is a linear invertible map that is either the identity map or a reflection operator x → x -2⟨x, e⟩ X e with some unit vector e ∈ X.Moreover, G • A : B X (0, R) → G • A(B X (0, R))is invertible, and there exists R ′ > 0 such thatA(B X (0, R)) ⊂ B X (0, R ′ ),and denoting byEQUATIONK0 := A(B X (0, R)), K1 := (Id X +D N •N N 1 •E N ) K0 , K2 := (Id X +D N •N N 2 •E N ) K1 , • • • KT := (Id X + D N • N N T • E N ) KT -1 = G • A(B X (0, R)), we have that for each t = 0, 1, ..., T Kt ⊂ Γ t and the mapping G • A| B X (0,R) : B X (0, R) → G • A(B X (0, R)) is bijective and its inverse is given by G • A| B X (0,R) -1 = A -1 • Φ,where Φ : G • A(B X (0, R)) → A(B X (0, R)) is defined byEQUATIONwhereL t : Γ t → Γ t+1 is defined by L t (y) := lim n→∞ π 1 • (⃝ n h=1 ϕ t ) • e 1 (y), where ϕ t : Γ t+1 × Γ t → Γ t+1 × Γ t is defined by ϕ t (x, y) = (y + D N • N N t • E N (x), y),where π 1 (x, y) = x and e 1 (y) = (0, y). Remark 2. Note that, in the proof, we have used the approximation result [22, Theorem 4.3] of Sobolev functions by neural networks with ReLU function x → max{0, x}, which is not differentiable at 0. Alternatively, we can use approximation result [1, Theorem 4.1] by neural networks with ReCU function x → max{0, x} 3 , which are continuously differentiable. Then, each block Id X + D N • N N t • E N in a obtained approximator G is C 1 and strongly monotone on ball Γ t = B(0, R ′ + tδ), that is, it holds that there is an α > 0 such that⟨(Id X + D N • N N t • E N )x, x⟩ X ≥ α∥x -y∥ 2X , x, y ∈ Γ t , which implies that, by the same argument as in Lemma 3,(Id X + D N • N N t • E N ) : Γ t → (Id X + D N • N N t • E N )(Γ t ), is a C 1 -diffeomorphism. Remark 3.From the proof, we can show that, for each t = 1, ..., T L t ( Kt ) ⊂ Kt-1 .Then, the mapping Φ defined in (156) is well-defined.Proof. Let δ, ϵ ∈ (0, 1), and let F : X → X be a layer of a neural operator defined in (4) . Assume that F is bilipschitz, that is, it satisfies (85) and ∥F (x) -F (y)∥ X ≤ C 1 ∥x -y∥ X , for all x, y ∈ X.(157) By Theorem 4, there is T ∈ N such that F can represented in the formF (x) = (Id X -H T ) • (Id X -H T -1 ) • • • • • (Id X -H 1 ) • A(x), for x ∈ B X (0, R),where A : X → X is a linear invertible map that is either the identity map or a reflection operator x → x -2⟨x, e⟩ X e with some unit vector e ∈ X, and H t : X → X are global Lipschitz maps satisfyingEQUATION)for all t. Moreover, the operator H t : X → X is a compact mapping. We choose R ′ > 0 such thatA(B X (0, R)) ⊂ B X (0, R ′ ).We denote byK 0 := A(B X (0, R)), K 1 := (Id X -H 1 )(K 0 ), • • • K T := (Id X -H T )(K T -1 ). Γ 0 := B X (0, R ′ ), Γ 1 := B(0, R ′ + δ), • • • Γ T +1 := B(0, R ′ + (T + 1)δ).We choose R > 0 such that for all t = 0, ..., T + 1E N (K t ), E N (Γ t ) ⊂ B R N (0, R).Since H t : X → X is a compact mapping, for large enough N ∈ N, we have that for all t = 1, ..., T ,EQUATIONNote thatP V N • H t • P V N = D N • HN,t • E N ,where HN,t : R N → R N by HN,t := E N • H t • D N .From (158), we haveHN,t ∈ W 1,∞ (B R N (0, R); R N ).By approximation by ReLU neural networks in Sobolev spaces (see e.g., [22, Theorem 4.3] ) , there is a ReLU neural networkEQUATION)Then, we estimate that by ( 159) and ( 160)EQUATIONAlso, we estimate that by (158) and ( 160)Lip B R N (0, R)→R N (N N t ) ≤ Lip B R N (0, R)→R N ( HN,t ) + Lip B R N (0, R)→R N ( HN,t -N N t ) ≤ Lip X→X (H t ) + ∥ HN,t -N N t ∥ W 1,∞ (B R N (0, R);R N ) ≤ δ. (162)We denote G : X → X byG := (I -D N • N N T • E N ) • (I -D N • N N T -1 • E N ) • • • • • (I -D N • N N 1 • E N ),which belong to R T,N,φ,ReLU (X). Then, by ( 161) and (162), we estimate that for all x ∈ B X (0, R),∥F (x) -G • A(x)∥ X ≤ 1≤t≤T ∥ ⃝ T h=t+1 (I -D N • N N h • E N ) • ⃝ t h=1 (I -H h ) • A(x) -⃝ T h=t (I -D N • N N h • E N ) • ⃝ t-1 h=1 (I -H h ) • A(x)∥ X ≤ 1≤t≤T T h=t+1 1 + Lip B R N (0, R)→R N (N N h ) sup y∈Kt-1 ∥(I -H t )(y) -(I -D N • N N t • E N )(y)∥ X ≤ 1≤t≤T T h=t+1 (1 + δ) sup y∈Kt-1 ∥H t (y) -D N • N N t • E N (y)∥ X ≤ 1≤i≤T (1 + δ) T ϵ T (1 + δ) T ≤ ϵ.Next, as (160), we see that(I -D N • N N 1 • E N )| Γ0 : Γ 0 → Γ 1 , (I -D N • N N 2 • E N )| Γ1 : Γ 1 → Γ 2 , • • • (I -D N • N N T • E N )| Γ T -1 : Γ T -1 → Γ T . and G| Γ0 = (I -D N •N N T •E N )| Γ T -1 •(I -D N •N N T -1 •E N )| Γ T -2 •• • ••(I -D N •N N 1 •E N )| Γ0 ,which means that G| Γ0 maps from Γ 0 to Γ T . For t = 1, ..., T , we define L t : Γ t → Γ t+1 by L t (y) := x * , y ∈ Γ t , where x * ∈ Γ t+1 is a unique fixed point of h t,y : Γ t+1 → Γ t+1 , whereh t,y (x) := y + D N • N N t • E N (x), that is, x * ∈ Γ t+1 is a unique solution of h t,y (x) = x ⇐⇒ y = (I -D N • N N t • E N )(x), x ∈ Γ t+1 .Indeed, there is a unique solution because we have for x 1 , x 2 ∈ Γ t+1 , from (162)∥h t,y (x 1 ) -h t,y (x 2 )∥ X = ∥D N • N N t • E N (x 1 ) -D N • N N t • E N (x 2 )∥ X ≤ δ∥x 1 -x 2 ∥ X ,which implies that h t,y : Γ t+1 → Γ t+1 is a contraction map. Then, we have for x ∈ Γ t-1 , t = 1, ..., T ,L t • (I -D N • N N t • E N )(x) = x.(163) Here, the solution x * ∈ Γ t+1 is given byx * = lim n→∞ x n , where x 0 = 0, x n+1 = y + D N • N N t • E N (x n ). We define φ t : Γ t+1 × Γ t → Γ t+1 × Γ t by φ t (x, y) = (y + D N • N N t • E N (x), y).Let π 1 (x, y) = x and e 1 (y) = (0, y) where π 1 : X × X → X and e 1 : X → X × X are linear maps. Then,L t (y) = x * = lim n→∞ π 1 • (⃝ n h=1 φ t )• e 1 (y). We define Φ P : Γ T → Γ 0 byΦ P := P Γ0 • L 1 • P Γ1 • L 2 • • • • • P Γ T -1 • L T ,where P Γt : X → X is the projection onto the convex set Γ t = B(0, R ′ + tδ). Then, by (163), we have forEQUATIONTherefore, Φ P is the left-inverse of G, that is, G| A(B X (0,R)) : A(B X (0, R)) → X is injective, andG| A(B X (0,R)) : A(B X (0, R)) → G•A(B X (0, R)) is bijective, and the inverse (G| G•A(B X (0,R)) ) -1 : G • A(B X (0, R)) → A(B X (0, R)) is given by (G| G•A(B X (0,R)) ) -1 = Φ,whereΦ : G • A(B X (0, R)) → A(B X (0, R)) is defined by Φ := Φ P | G•A(B X (0,R))and has the formΦ = L 1 • L 2 • • • • • L T | G•A(B X (0,R)) . Therefore, G • A : B X (0, R) → G • A(B X (0, R)) is also bijective, and the inverse is given by(G • A| B X (0,R) ) -1 = A -1 • Φ. Note that G • A(B X (0, R)) ⊂ Γ T .A.8 Production of Quantitative Universal Approximation EstimatesHere, we show how our framework may be used 'out of the box' to produce quantitative approximation results for discretization of neural operators.Let X be a separable Hilbert space. We will consider a Hilbert space X, endowed with its norm topology. Recall that S 0 (X) ⊂ S(X) is a partially ordered lattice of finite dimensional subspaces of X. Next, we consider quantified discretization of a continuous, possibly non-linear function F : X → X, that is, how the discretizations F V : V → V can be chosen (using e.g. neural networks F V having a given architecture for each subspace V ⊂ X) so that the obtained discretization operator A F has the explicitly given error bounds ε V .Using quantitative approximation results for neural networks in R d , see e.g. [57] or [23] , one obtains quantitative results for neural operators. An example of such result is given below.Proposition 5. Let r > 0 and F : B X (0, r) → X be a non-linear function satisfying F ∈ Lip(B X (0, r); X), in n = 1, or F ∈ C n (B X (0, r); X), if n ≥ 2. Let ε V > 0 be numbers indexed by the linear subspaces V ⊂ X such that ε V → 0 as V → X. When d = dim(V ), the space V is identified with R d using an isometric isomorphism J V : V → R d . Then there is a feed forward neural network F V,θ : R d → R d with ReLU-activation functions with at most C(n, d) log 2 ((1 + r) n /ε V ) layers and C(n, d)ε-d/n V log 2 ((1 + r) n /ε V ) non-zero elements in the weight matrices such that A N N : F → (F V ) V ∈S0(X) , where F V = J -1 V • F V,θ • J V : V → V , is an ⃗ ε-approximation operation in the ball B X (0, r). Proof. Let Lip(F : B X (0, r) → X) ≤ M , in n = 1, or ∥F ∥ C n (B X (0,r);X) ≤ M , if n ≥ 2.Let V ⊂ X be a linear subspace of dimension d and ε V > 0. Let us use some orthogonal basis of V to identify V and R d and denote the identifying isomorphism by J : R d → V . The functionF : R d → R d , given by F = J -1 • P V • F • J satisfies Lip( F : B R d (0, r)) → X) ≤ M if n = 1, or ∥ F ∥ C n (B R d (0,r)) ≤ M if n ≥ 2.Then, by applying [57, Theorem 1] or [23] , we see that there exists a feed forward neural network F V,θ : R d → R d with ReLU-activation functions with at mostEQUATIONDue to Def. 1, this yields the claim.In this section, we consider the approximation of diffeomorphisms by globally invertible residual networks on Hilbert spaces. From the viewpoint of no-go theorem, as the class of diffeomorphisms is too large, we focus on the class of strongly monotone C 1 -diffeomorphisms with compact support 5 .In what follow, we employ σ as the GroupSort activation having a group size of 2 (see [3, Section 4] ). As sort activation is 1-Lipschitz, from Lemma 9, R inv L,N,φ,δ,σ (X) is a subset of the class of homeomorphisms. We finally show that, in this case, R inv L,N,φ,δ,σ (X) is an universal approximator for the class of the strongly monotone diffeomorphisms with compact support. Theorem 7. Let R > 0, and let F : X → X be strongly monotone C 1 -diffeomorphism with compact support, and let σ be the GroupSort activation having a group size of 2. Then, for any orthonormal basis {φ n } n∈N ⊂ X, δ ∈ (0, 1), and ϵ ∈ (0, 1), there are T, N ∈ N, and G ∈ R inv T,N,φ,δ,σ (X) such that sup u∈B X (0,R)∥F (u) -G(u)∥ X ≤ ϵ.Moreover, the inverseEQUATIONwhere Lt : X → X is defined byL t (y) := lim n→∞ π 1 • ⃝ n h=1 φt • e 1 (y),where φt :X × X → X × X is defined by φt (x, y) = (y + D N • N N t • E N (x), y),where π 1 (x, y) = x and e 1 (y) = (0, y).Proof. We denote by Diff 1 (X) the class of C 1 -diffeomorphisms between X, and Diff 1 sm (X) the class of strongly monotone C 1 -diffeomorphisms between X. We also denote the support of F ∈ Diff 1 (X) by supp(F ) := {x ∈ X :EQUATIONX). We define F 1 : X → X by F 1 := Id X + P V N (F -Id X )P V N , and we see thatEQUATIONas Id X -F : X → X is a compact mapping. By the same argument in Lemma 1, we can show that F N := E N F D N ∈ Diff 1 sm,c (R N ), and DF N | 0 is a positive definite matrix, which is connected to Id R N . By the similar argument in the proof of Theorem 4, see (105)-(152), we can construct continuous paths f :EQUATIONwhere t j and s j are chosen so that 0 = t 0 < t 1 < . . . t m < t m+1 = 1 and 0 = s 0 < s 1 < . . . s m < s m+1 = 1 and that the Lipschitz constant of the mapsf tj • f -1 tj-1 -Id R N and β sj • β -1 j-1 -Id R N are less that δ.Then, there is T ∈ N such that F N has the formF N = (Id R N -H T ) • (Id R N -H T -1 ) • • • • • (Id R N -H 1 ), where for each t = 1, ..., T , Lip R N →R N (H t ) ≤ δ.Then, remarking that E N D N = Id R N and D N E N = P V N , we see thatF 1 = P V ⊥ N + D N (Id R N -H L ) • • • • • (Id R N -H 1 )E N = (Id X -D N • H T • E N ) • • • • • (Id X -D N • H 1 • E N ).For each t = 1, ..., T , by using [3, Theorem 3 and Observation], δ-Lipschitz function H t : R N → R N can be approximated by a neural network N N t : R N → R N with GroupSort activation σ having a group size of 2 in L ∞ -norm on any compact set, and N N t : R N → R N is δ-Lipschitz continuous.We denoting byG := (Id X -D N • N N T • E N ) • • • • • (Id X -D N • N N 1 • E N ) ∈ R inv T,N,φ,δ,σ (X). Note that Lip X→X (D N • N N ℓ • E N ) ≤ δ.Then, we can show that, by similar way in the first half of proof of Theorem 5, supEQUATIONWith ( 169) and (171), we obtain thatsup u∈B X (0,R) ∥F (u) -G(u)∥ X ≤ sup u∈B X (0,R) ∥F (u) -F 1 (u)∥ X + sup u∈B X (0,R) ∥F 1 (u) -G(u)∥ X ≤ 2ϵ.The representation of inverse G -1 can be given by the same argument in the proof of Theorem 5.Similarly In the main text, we defined generalized neural operators on Hilbert spaces. Here, we give several examples to show that they are extensions of classical neural operators [33, 30] . Example 1. Let X be a Hilbert space. Let G ℓ : X → X be a classical neural operator having the formG ℓ := (W T ℓ + K T ℓ ) • σ(W T ℓ -1 + K T ℓ -1 ) • • • • σ(W 1 + K 1 ),where W ℓ : X → X are linear bounded operators corresponding to the local term and K ℓ : X → X are compact operators corresponding to the non-local term (e.g., integral operators having a smooth kernel or smooth basis). We assume that activation function σ is C 1 . Then, we have G ℓ ∈ C 1 (X; X). Let T 1 = K lif t : X → X and T 2 = K proj : X → X be compact linear operators, which corresponds to lifting and projection, respectively. We denoting byF ℓ := I + T 1 • G ℓ • T 2 ,which is one block of classical neural operators with skip-connection. We also denote by A ℓ = Id and σ = Id. Then, the generalized neural operatorH = F L • • • • • F 1 corresponds to classicalneural operators with skip-connections. In this paper, the skip-connection (ie., the structure of the identity plus some compact mapping) is so important to preserve bijectivity, discussed in Section 3. Example 2. We show that classical neural operators , for exampleF clas : u → (W 2 + K 2 ) • (σ(W 1 u + K 1 (u)),see [30, 33] , can be written in the form of the generalized neural operators that we consider of the formH(x) = A k • σ • F k • A k-1 • σ • F k-1 • • • • • A 1 • σ • F 1 • A 0 ,where A j : X → X are linear operators which may not be bijective andF j = Id + T j,1 • G • T j,2 : X → X.We could start with the observations that for an infinite dimensional Hilbert space X there is a linear isomorphism J m,n : X n → X m , where X m = X × • • • × X. The reason for this is that the cardinality of Hilbert basis of the space X is the same as the cardinality of the Hilbert basis of X n , see [26, Theorem 3.5] .where V N := span{φ n } n≤N , and K(j) N : L 2 (D; R) h → L 2 (D; R) h is a finite rank operator defined by K j (u)(x) := p,q≤N K (j) p,q ⟨u, φ p ⟩ L 2 (D;R) φ q (x), x ∈ D,Then, H : L 2 (D; R) h → L 2 (D; R) h can be written byH = (W (k) + K (k) ) • σ • (W (k-1) + K (k-1) ) • σ • • • • • (W (1) + K (1) ) • σ • (W (0) + K (0) ),which coincides with classical neural operators (assuming that all local weight matrices are invertible) used in e.g., [33] . See Definition 9 as well.Example 4. Let us consider an example of an example of an generalized neural operator which is obtained by composition of non-linear integral operators and smooth activation functions. Let D ⊂ R d be be a bounded domain with a smooth boundary. We consider the case when X = H 1 (D; R h ) = H 1 (D; R) h are Sobolev spaces, and A j = W (j) where W (j) ∈ R h×h are invertible matrices, andF j = Id H 1 + i H 2 →H 1 • K(j) • i H 1 →L 2 W (j)-1 (corresponding to T 1 = i H 1 →L 2 , T 2 = i H 2 →H 1 (W (j) ) -1 , G = K(j) ) where K(j) : L 2 (D; R) h → H 1 (D; R) h is non-linear integral operator K(j) (u)(x) := D k (j) (x, y, u(y))u(y)dy, x ∈ D,where kernel satisfies k (j) ∈ C 3 (D × D × R h ; R h×h ) has uniformly bounded three derivatives. Also, let σ ∈ C 1 (R) be an activation function which derivative is uniformly bounded and we denoteσ * f = σ • f . Then, H : H 1 (D; R) h → H 1 (D; R) h , defined by H = (W (k) + i H 2 →H 1 • K(j) • i H 1 →L 2 ) • σ * • (W (k-1) + i H 2 →H 1 • K(j) • i H 1 →L 2 ) • σ * • • • • • (W (1) + K (1) ) • σ * • (W (0) + i H 2 →H 1 • K(j) • i H 1 →L 2 ),is an generalized neural operator. Here, i H 1 →L 2 and i H 2 →H 1 are compact emending from H 1 (D) to L 2 (D) and H 2 (D) to H 1 (D), respectively. Non-linear integral operator in neural operator has been used in [30] . Example 5. Let us consider an example of a typical neural operator which is a finite composition of layers of neural operators similar to those introduced in [30, 33] , F : X → X of the formEQUATIONwhere X = H m (Ω), Ω ⊂ R d is a bounded set with a smooth boundary and W : X → X is a linear operator. In the above, Y = C(Ω) and Z = C m+1 (Ω), where m > d/2. Moreover, G : Y → Z is a nonlinear integral operatorG(u)(x) = Ω k θ (x, y, u(y))u(y)dy, where k θ , ∂ u k θ ∈ C m+1 (Ω × Ω × R) is a kernel given by a feed-forward neural network with sufficiently smooth activation functions. Here, we assume that σ j ∈ C ℓ (R), ℓ ≥ m + 2 and that the kernel k θ is of the form k θ (x, y, t) = J j=1 c j (x, y, θ)σ j (a j (x, y, θ)t + b j (x, y, θ)).Moreover, T 1 : X → Y and T 2 : Z → X are the identical embedding operatorsT j (u) = u.Due to the choice of function spaces X, Y and Z, the maps T 1 and T 2 are compact operators. Summarizing, F = F σ,W,k , whereEQUATIONFurthermore, by choosing k θ (x, y, u(y)) = k θ (x -y) and Ω = T d as the convolutional kernel and the torus, the map F takes the form of an FNO [39] .Here the activation functions σ and σ j are in different roles as the functions σ j appear inside the integral operator (or between the compact operators T 1 and T 2 ). The function σ is useful in obtaining universal approximation results for neural operators, but as this question is somewhat technical, we postpone this discussion to the end.The appearance of the compact operators T 1 and T 2 makes the discretization of activation function σ and the activation functions inside G in Definition 4 different, and this is the reason why we have introduced both σ and G. To consider invertible neural operators, we will below assume that σ is an invertible function, for example, the leaky Relu function. In the above operator, the nonlinear function G inside compact operators in the operationN : u → u + T 2 (G(T 1 u)),and the compact operators map weakly converging sequences to norm converging sequences. This is essential in the proofs of the positive results for approximation functors as discussed in the paper. However, we do not have general results on how the operationG σ : u → σ • u,can be approximated by finite dimensional operators in the norm topology, but only in the weak topology in the sense of Definition 11 of the Weak Approximation Functor. However, one can overcome this difficulty in two ways. The first way is to use in the discretization a suitable finite dimensional space V that satisfies G σ (V ) ⊂ V . For example, when X = L 2 (Ω) and σ is a leaky relu function, one can choose V to be a space that consists of piecewise constant functions. The second way is choosing a composition of layers of the formEQUATIONG σ : u → σ • u,in different finite dimensional spaces V j . For example, we can consider these operations as mapsEQUATIONEQUATIONThis makes the compositionG σ • N 2 • G σ • N 1 : V 1 → V 3 ,well defined. The maps, G σ , are clearly invertible and one can use Theorems 4 and 5 to analyze when N j : V j → V j are invertible functions.Finally, we return to the question whether the activation function σ is useful in universal approximation results. If the activation function σ is removed (that is, it is the identical map σ id : s → s), the operator F is a sum of a linear operator and a compact nonlinear integral operator,EQUATIONMoreover, if we compose above operators F j of the above form, the obtained operator, G : X → X is also a sum of a linear operator and a compact operator,G(u) = W u + K(u).Moreover, the Frechet derivative of G at u 0 , denoted DG| u0 is the mapDG| u0 : v → W v + D K| u0 v,and, due to the above assumptions on kernel k θ (x, y, u), the derivative is a linear operatorD K| u0 : H m (Ω) → H m+1 (Ω). Proof. Let G ∈ R T,N,φ,σ (L 2 (D; R)) such that G = (Id L 2 (D;R) + D N • N N T • E N ) • • • • • (Id L 2 (D;R) + D N • N N 1 • E N ).Since φ = {φ n } n∈N ⊂ L 2 (D; R) has the constant basis, denoting it by φ 0 :=1 D (x) ∥1 D ∥ L 2 (D;R), where1 D (x) = 1 for x ∈ D, we see that for α = (α 1 , ..., α N ) ∈ R N D N α(x) = n≤N α n φ n (x) = n≤N α n ⟨φ 1 , φ 1 ⟩ L 2 (D;R) φ n (x) = n≤N 1 ∥1 D ∥ L 2 (D;R) ⟨α n • 1 D , φ 1 ⟩ L 2 (D;R) φ n (x).We Then, we have that (182) can be proved by the same argument.EQUATIONWe can generalize the no-go Theorem 2 for the case when approximations and continuity of approximations are considered in the weak topology of the Hilbert space X. This can be done when the approximations of the identity map and the minus one times the identity operator satisfy additional assumptions and the partially ordered subset S 0 (X) is the set of all all linear subspaces S(X) of X.In the case when S 0 (X) = S(X) and Definition 7 the condition (A) can generalized as follows:Definition 11 (Weak Approximation Functor). When S 0 (X) = S(X) we define the weak approximation functor, that we denote by A : D → B, as the functor that maps each (X, F ) ∈ O D to some (X, S(X), (F V ) V ∈S(X) ) ∈ O B so that the Hilbert space X stays the same. The approximation functor maps all morphisms a ϕ to A ϕ and morphisms a X1,X2 to A X1,X2 , and has the following properties (A') For all r > 0, all (X, F ) ∈ O D and all y ∈ X, it holds that lim V →X sup x∈B X (0,r)∩V ⟨F V (x) -F (x), y⟩ X = 0.(185)Moreover, when F : X → X is the operator Id : X → X or -Id : X → X, then F V is the operator Id V : V → V or -Id V : V → V , respectively.Moreover, Definition 8 can generalized as follows so that it uses the weak topology. Definition 12. We say that the approximation functor A is continuous in the weak topology if the following holds: Let (X, F ), (X, F (j) ) ∈ O D be such that the Hilbert space X is the same for all these objects and let (X, S(X), (F V ) V ∈S(X) ) = A(X, F ) be approximating sequences of (X, F ) and (X, S(X), (F j,V ) V ∈S(X) ) = A(X, F (j) ) be approximating sequences of (X, F (j) ). Moreover, assume that for r > 0 and all y ∈ X lim j→∞ sup x∈B X (0,r)EQUATIONThen, for all V ∈ S(X) the approximations F (j)V of F (j) and F V of F satisfy for all y ∈ X lim j→∞ sup x∈V ∩B V (0,r)EQUATIONThe theorem below states a negative result, namely that there does not exist continuous approximating functors for diffeomorphisms. Theorem 8. (No-go theorem for discretization of general diffeomorphisms) There exists no functor D → B that satisfies the property (A') of a weak approximation functor and is continuous in the weak topology.The proof of Theorem 8 is analogous to Theorem 2 by replacing A 1 by the map -Id and considering a linear space V ∈ S(X) having an odd dimension, in which case deg(A 1 : V → V ) = -1. Let F 0 = Id : X → X and F 1 = -Id : X → X. Assume that (X, S(X), (F t,V ) V ∈S(X) ) = A(X, F t ) are approximations of the map F : X → X in the weak topology and that A is continuous in the weak topology. Recall that then F t,V : V → V are C 1 -diffeomorphisms that are discretizations of F : X → X. Then, by condition (A'), F 0,V = Id V and F 1,V = -Id V so that deg(F 0,V ) = 1 and deg(F 1,V ) = -1. Observing that when the condition (187) is applied for y 1 , y 2 , . . . , y n ∈ V that form a basis of the space V having the dimension dim(V ) = n, we see that the condition (187) implies the condition (7) . Using these observations, Theorem 8 is follows similarly to the proof of Theorem 2.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).38th Conference on Neural Information Processing Systems (NeurIPS 2024).By using mapping properties of monotone operators[5], we can replace this definition by using Hilbert spaces Y and Z that are isometric to X, T1 : X → Y and T2 : Z → X as compact linear operators, and G : Y → Z as a C 1 nonlinear operator.Note that we can write the reflection operator across the hyperplane {e} ⊥ , that is, the operator x → x -2⟨x, e⟩X e as a diagonal operator diag(-1, 1, 1, . . . ) in a suitable orthogonal basis of X.The above used homotopy can be replace by a more explicit flow in the Lie group of diffeomorphism, see e.g.[42].We denote the support of F : X → X by supp(F ) := {x ∈ X : F (x) ̸ = x}.We will discuss the function space L 2 (D; R) for easier reading, but all arguments can be replaced with real-valued function space U(D; R) that is a separable Hilbert space. 


Decision: The reviewers find this work studying an important problem and encourage the authors to incorporate the suggestions and developments during the review process.

Reviews: review: paper_summary: paper_summary: The paper addresses the problem of discretizing neural operators, maps between infinite dimensional Hilbert spaces that are trained on finite-dimensional discretizations.  Using tools from category theory, the authors provide a no-go theorem showing that diffeomorfisms between Hilbert spaces may not admit continuous approximations by diffeomorfisms on finite spaces. This highlights the fundamental differences between infinite-dimensional Hilbert spaces and finite-dimensional vector spaces. Despite these challenges, the authors provide positive results, showing that strongly monotone diffeomorphism operators can be approximated in finite dimensions and that bilipschitz neural operators can be decomposed into strongly monotone operators and invertible linear maps. Finally, they observe how such operators can be locally inverted through an iteration scheme.
strength_weakness: strength_weakness: - The text and presentation require significant polishing.  It contains numerous typos, poorly formulated sentences, and instances of missing or repeated words
- While the paper's theoretical focus is valuable, it lacks  examples of specific neural operator structures that meet the theorems or remarks 
- A more detailed  discussion on the  practical impact of this work, accompanied by examples, would be beneficial for the audience

Please note that my review should be taken with  caution, as I am not  familiar with category theory and did not thoroughly check the mathematical details. My feedback primarily focuses on the presentation and potential impact of the results rather than a rigorous validation of the theoretical content.
questions: questions: - Neural operators are typically defined between Banach spaces. Why does your theory focus on maps between Hilbert spaces instead?
- Comment: The work in [1] might have been relevant to cite as well.
- The main neural operator paper [2] develops theoretical results on the universal approximation theory of neural operators. How do your results relate to the ones in that paper?

[1] F. Bartolucci, E. de Bézenac, B. Raonić, R. Molinaro, S. Mishra, R. Alaifari, Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning, NeurIPS 2023.

[2] Nikola B. Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. "Neural operator: Learning maps between function spaces with applications to PDEs," J. Mach. Learn. Res., 24(89):1–97, 2023.
limitations: limitations: The paper lacks  examples of applications of the theorems to specific neural operator structures, and some further discussion on the practical impact of the results with examples
paper_summary: paper_summary: This paper investigates theoretical limitations of discretizing neural operators on infinite-dimensional Hilbert spaces. The authors first prove a "no-go theorem" (Theorems 1,2) showing that diffeomorphisms between infinite-dimensional Hilbert spaces cannot generally be continuously approximated by finite-dimensional diffeomorphisms. Then, they provide positive results for certain classes of operators such as strongly monotone (Theorem 3) and bilipschitz neural operators (Theorem 4). They finally provide concrete example of approximation by finite residual ReLU networks (Theorem 5).
strength_weakness: strength_weakness: - On the other hand, the proofs are based on conventional analytical arguments rather than category-theoretic arguments. Therefore, the "category theory" framework might be somewhat exaggerated. It is expected that with refinement of notation and sentence structure, the description could become more perspicuous in the future.

- There is concern that the categorical description may have obscured the contributions typically seen in __traditional approximation theory__ papers. As the authors likely recognize, various topologies are used in function approximation, and this study focuses __only__ on approximation in the norm topology of Hilbert spaces, and does not negate "all considerable approximation sequences". So, the impossibility theorem presented here might simply be due to the norm topology being too strong. While the Hilbert structure sounds natural as a generalization of Euclidean structure, in reality, concepts like L2 convergence of Fourier series are quite technical and not necessarily an inevitable notion of convergence. It seems that in pursuit of an elegant categorical description, the diversity of function approximation may have been compromised.
questions: questions: In Definition 3, why $\sigma$ is imposed besides $G$?
limitations: limitations: The authors did not discuss the validity of assumptions.
paper_summary: paper_summary: This paper focuses on the continuous discretization in operator learning. This is a very important question since it involves reducing the infinite-dimensional space to a finite-dimensional space in operator learning. The authors present cases where discretization is continuous and cases where it is not. The results are interesting and can be applied to design methods in operator learning.
strength_weakness: strength_weakness: Since this paper is submitted to NeurIPS and not a mathematical journal, I hope the authors can provide some practical examples, such as solving the Poisson equation \(\Delta u = f\) to learn the operator relationship between \(f\) and \(u\). By using methods like DeepONet and FNO, it would be beneficial to determine whether the discretization in these methods is continuous or not. I believe this could make the paper more accessible to a broader audience.
questions: questions: Mentioned in the Weakness.
limitations: limitations: All right.
paper_summary: paper_summary: This paper studies the question of whether a neural operator (or a general diffeomorphism on an infinite dimensional Hilbert space) can be continuously discretized through the lens of category theory. It first proves that there does not exist a continuous approximation scheme for all diffeomorphisms. Then, it shows that neural operators with strongly monotone layers can be continuously discretized, followed by a proof that a biliptschitz neural operator can be approximated by a deep one with strongly monotone layers. Some further consequences are discussed.
strength_weakness: strength_weakness: 1. Most results presented in the paper are purely theoretical and lack a quantitative or asymptotic estimate. For example,
   * the notion of continuous approximation functor in Definition 8 does not care about the rate of convergence, and
   * there is no estimate for the number of layers $J$ in Theorem 4.
2. No empirical results are there to support the theory. While this is a theoretical paper, some toy experiments that exemplify the theory would be very helpful.
questions: questions: 1. In general, is there any assumption of the Hilbert space studied in this paper? For example, are the Hilbert spaces assumed to be separable?
2. What is the definition of the convergence of finite-dimensional subspaces used in this paper? Strong convergence of the projection operator? I do not think there is a standard definition for this in elementary functional analysis so it would be helpful to say it explicitly in the paper.
3. Have you studied the role of the bilipschitz constant in your theorems? For example, how does the number of layers $J$ in Theorem 4 depend on it?
limitations: limitations: None.

Comments: comment: Thank you for your response. We would will address your points in the following way.

- If so, I recommend the authors to reconsider the following phrases in the abstract and conclusion: "Using category theory, we give a no-go theorem." "We used tools from category theory to produce a no-go theorem"

We appreciate the advice, and will follow it. We will replace 

"Using category theory, we give a no-go theorem"

with

"Using analytical arguments, we give a no-go theorem framed with category theory."

and replace

"We used tools from category theory to produce a no-go theorem"

with

"We give a no-go theorem framed with category theory"

in the abstract and conclusion.

- It would be much impactful and significant if the authors could more directly point out any incorrectness of the proof or inappropriateness of the assumption in the previous studies.

There are several papers which use continuous functions (either as elements of infinite dimensional function spaces or metric spaces) to model images or signal and apply statistical methods and invertible neural networks or maps modeling diffeomorphisms. Often in these papers one derives theoretical results in the continuous models and presents numerical results using a finite dimensional approximations. In this process the errors are caused by the discretization and the effect of changing the dimension of the approximate models. We believe that our work meaningfully addresses these questions as applied to injective/bijective neural operators, an important architecture. We hope that our paper inspires further study these points. We can include citations to the following papers, related to these issues.

The below papers which combine neural networks and approximation of diffeomorphisms, as applied to imaging.

- Elena Celledoni · Helge Glöckner · Jørgen N. Riseth, Alexander Schmeding
Deep neural networks on diffeomorphism groups for
optimal shape reparametrization.
BIT Numerical Mathematics (2023) 63:50

- GradICON: Approximate Diffeomorphisms via Gradient Inverse Consistency
Lin Tian · Hastings Greer · François-Xavier Vialard · Roland Kwitt · Raúl San José Estépar · Richard Jarrett Rushmore · Nikolaos Makris · Sylvain Bouix · Marc Niethammer
West Building Exhibit Halls ABC 153

The below papers combine invertible neural networks and statistical models, especially for solving inverse problems (including imaging problems). 

- Alexander Denker , Maximilian Schmidt , Johannes Leuschner and Peter Maass
Conditional Invertible Neural Networks for Medical Imaging. Journal of Imaging 2021, 7(11), 243

- Ardizzone, L.; Kruse, J.; Rother, C.; Köthe, U. Analyzing Inverse Problems with Invertible Neural Networks. In Proceedings of
the 7th International Conference on Learning Representations (ICLR 2019), New Orleans, LA, USA, 6–9 May 2019.

- Anantha Padmanabha, G.; Zabaras, N. Solving inverse problems using conditional invertible neural networks. J. Comput. Phys.
2021, 433, 110194

- Denker, A.; Schmidt, M.; Leuschner, J.; Maass, P.; Behrmann, J. Conditional Normalizing Flows for Low-Dose Computed
Tomography Image Reconstruction. In Proceedings of the ICML Workshop on Invertible Neural Networks, Normalizing Flows,
and Explicit Likelihood Models, Vienna, Austria, 18 July 2020.

- Hagemann, P.; Hertrich, J.; Steidl, G. Stochastic Normalizing Flows for Inverse Problems: A Markov Chains Viewpoint. 
SIAM/ASA Journal on Uncertainty QuantificationVol. 10, Iss. 3 (2022) 10.1137

- Papamakarios, G.; Nalisnick, E.T.; Rezende, D.J.; Mohamed, S.; Lakshminarayanan, B. Normalizing Flows for Probabilistic
Modeling and Inference. Journal of Machine Learning Research 22 (2021) 1-64
Well, I felt sorry for the authors because a reviewer of NeurIPS, once well-known for a top theory-oriented conference of machine leaning, **cannot raise his/her score** simply because he/she is **not familiar with the category theory**. I even feel this is symbolic of the current state of a theory-oriented machine learning conference. It should not be the problem of the individual reviewer him/herself, but the problem of conference's matching systems that mistakenly assign a perfect amateur of the category theory for reviewing a category theoretic research.

This is a suggestion for chairs for future avoidance of mismatches, that the reviewers should be examined if they have fundamental knowledge/background/understandings in the field. I am an expert of expressive power analysis, but not at all of category theory and tropical geometry. Unfortunately, this kind of mismatch happens every year, so I am usually skeptical to any mathematical ''theorems'' published in machine learning conferences.
Thank you for your detailed response. As I mentioned in my initial review, my understanding of category theory is somewhat limited. My feedback has mainly focused on the presentation and potential impact of the results rather than an in-depth validation of the theoretical content.  I am not in a position to increase my score.
Thank you for the detailed response. Since I am not absolutely familiar with category theory and other related work, I am unable to further raise my score, but I acknowledge that I have read through the rebuttal and it appears to be a nice paper overall.
Thank you for detailed clarifications. I would like to keep my score as is.

> The proofs are indeed based on analytical arguments. 

If so, I recommend the authors to reconsider the following phrases in the abstract and conclusion:

> Using category theory, we give a no-go theorem
> We used tools from category theory to produce a no-go theorem

It would be much impactful and significant if the authors could more directly point out any incorrectness of the proof or inappropriateness of the assumption in the previous studies.
Thanks for the reply. I will keep my score.