Title: Optimal deep learning of holomorphic operators between Banach spaces
Venue: NeurIPS 2024
Accepted: yes
Keywords: Deep learning

operator learning

parametric PDEs

deep neural networks

generalization error

optimal algorithms
Abstract: Operator learning problems arise in many key areas of scientific computing where Partial Differential Equations (PDEs) are used to model physical systems. In such scenarios, the operators map between Banach or Hilbert spaces. In this work, we tackle the problem of learning operators between Banach spaces, in contrast to the vast majority of past works considering only Hilbert spaces. We focus on learning holomorphic operators -- an important class of problems with many applications. We combine arbitrary approximate encoders and decoders with standard feedforward Deep Neural Network (DNN) architectures -- specifically, those with constant width exceeding the depth -- under standard $\ell^2$-loss minimization. We first identify a family of  DNNs such that the resulting Deep Learning (DL) procedure achieves optimal generalization bounds for such operators. For standard fully-connected architectures, we then show that there are uncountably many minimizers of the training problem that yield equivalent optimal performance. The DNN architectures we consider are `problem agnostic', with width and depth only depending on the amount of training data $m$ and not on regularity assumptions of the target operator. Next, we show that DL is optimal for this problem: no recovery procedure can surpass these generalization bounds up to log terms. Finally, we present numerical results demonstrating the practical performance on challenging problems including the parametric diffusion, Navier-Stokes-Brinkman and Boussinesq PDEs.

Optimal deep learning of holomorphic operators between Banach spaces

Abstract

Operator learning problems arise in many key areas of scientific computing where Partial Differential Equations (PDEs) are used to model physical systems. In such scenarios, the operators map between Banach or Hilbert spaces. In this work, we tackle the problem of learning operators between Banach spaces, in contrast to the vast majority of past works considering only Hilbert spaces. We focus on learning holomorphic operators - an important class of problems with many applications. We combine arbitrary approximate encoders and decoders with standard feedforward Deep Neural Network (DNN) architectures - specifically, those with constant width exceeding the depth - under standard ℓ 2 -loss minimization. We first identify a family of DNNs such that the resulting Deep Learning (DL) procedure achieves optimal generalization bounds for such operators. For standard fully-connected architectures, we then show that there are uncountably many minimizers of the training problem that yield equivalent optimal performance. The DNN architectures we consider are 'problem agnostic', with width and depth only depending on the amount of training data m and not on regularity assumptions of the target operator. Next, we show that DL is optimal for this problem: no recovery procedure can surpass these generalization bounds up to log terms. Finally, we present numerical results demonstrating the practical performance on challenging problems including the parametric diffusion, Navier-Stokes-Brinkman and Boussinesq PDEs.

1 Introduction

Operator learning is increasingly being investigated for problems arising in computational science and engineering. These problems are often posed in terms of Partial Differential Equations (PDEs), which can be viewed as operators mapping function spaces to function spaces. Depending on the requirements for well-posedness of the PDE, both the input and solution spaces are often Hilbert, or, more generally, Banach spaces. The aim of operator learning is to efficiently capture the dynamic behavior of these operators using surrogate models, typically based on Deep Neural Networks (DNNs). Specifically, we want to learn

where Y is the PDE solution space, X represents the data supplied to the PDE, i.e., possibly multiple functions describing initial and boundary conditions or forcing terms or, equivalently, a vector of parameters defining such functions.

Let µ be a probability measure on X . Given noisy training data

where X 1 , . . . , X m ∼ i . i . d . µ and E i is noise, a typical operator learning methodology consists of three objects: an approximate encoder E X : X → R d X , an approximate decoder D Y : R d Y →Y and a DNN ̂ N : R d X → R d Y . It then approximates F as

The encoder and decoder are either specified by the problem, learned separately from data, or learned concurrently with ̂ N . The goal, as in all supervised learning problems, is to ensure good generalization via the learned operator ̂ F from as little training data m as possible.

1.1 Contributions

As noted in, e.g., [16, 66], the theory of deep operator learning is still in its infancy. We contribute to this growth in the following ways. We consider learning classes of holomorphic operators (Assumption 2.2), with arbitrary approximate encoders E X and decoders D Y . As we explain in §2.3 (see also [52, §5.2], [53, §3.4] and [41]) these operators are relevant in many applications, notably those involving parametric PDEs. The main contributions of this work are as follows.

We consider operators taking values in general Banach spaces. As noted, the vast majority of existing work (with the notable exception of [16]) considers Hilbert spaces.

We consider standard feedforward DNN architectures (constant width, width exceeds depth) and training procedures ( ℓ 2 -loss minimization).

(Theorem 3.1) We construct a family of DNNs such that any approximate minimizer of the corresponding training problem satisfies a generalization bound that is explicit in the various error sources: namely, an approximation error , which decays algebraically in the amount of training data m ; encoding-decoding errors , which depend on the accuracy of the learned encoders and decoders; an optimization error , and; a sampling error , which depends on the noise E i in (1.2).

These DNN architectures are problem agnostic ; they depend on m only. In particular, the architectures are completely independent on the regularity assumptions of target operator.

(Theorem 3.2) We show that training problems based on any family of fully-connected DNNs possess uncountably many minimizers that achieve the same generalization bounds.

(Theorems 3.1-3.2) We provide bounds in both the L 2 µ - and L ∞ µ -norms that hold in high probability, rather than just expectation.

(Theorems 4.1-4.2) We show that the generalization bound is optimal with respect to m : no learning procedure (not necessarily DL-based) can achieve better rates in m up to log terms.

Finally, we present a series of experiments demonstrating the efficacy of DL on challenging problems such as the parametric diffusion, Navier-Stokes-Brinkman and Boussinesq PDEs, the latter two of which involve operators whose codomains are Banach, as opposed to Hilbert, spaces.

1.2 Relation to previous work

Approximating an operator between function spaces with training data obtained through numerical PDEs solves presents a formidable challenge. Nevertheless, in recent years, significant advances have been made through the development of DL techniques, leading to the field of operator learning [15, 40, 51, 53, 56, 58, 60, 62, 69, 83, 93, 103]. These approaches often leverage intricate DNN architectures to approximate the complex mappings inherent in physical modelling scenarios. Many works have also focused on the practical aspects of operator learning in real-world applications [13, 24, 35-37, 42, 45, 48, 49, 59, 61, 64, 65, 70, 73, 77, 80, 81, 84, 96, 98-101, 104, 105].

On the theoretical side, universal approximation theorems for operator learning have been developed in [50, 55, 68, 69] and elsewhere. Such bounds are typically not quantitative in the size of the DNN needed to achieve a certain error. For this, one typically either restricts to specific operators (e.g., certain PDEs) or imposes regularity conditions. One such assumption is Lipschitz regularity - see [10, 16, 55, 66, 87] and references therein. However, learning Lipschitz operators suffers from a curse of parametric complexity [54], meaning that algebraic rates may not be achievable. Another

common assumption is holomorphy. While stronger, it is, as noted, very relevant to operator learning problems involving parametric PDEs. Quantitative approximation results for holomorphic operators have been shown in [26, 31, 41, 55, 71] and elsewhere.

However, these works do not consider the generalization error, i.e., the error incurred when learning the approximation (1.3) from the finite training data (1.2). This is particularly important in applications of operator learning where data is obtained through expensive numerical PDE solves, since such problems are highly data-starved . Several works have tackled this question from the perspective of statistical learning theory and nonparametric estimation [16, 55, 66], but only for Lipschitz operators. As observed in [6, §9.5], this approach generally leads to a best O ( m -1 / 2 ) decay of the L 2 µ -norm error with respect to m . Theorem 4.1 shows that such a rate is strictly suboptimal for learning the classes of holomorphic operators we consider. Our generalization bounds in Theorems 3.1-3.2 do not use such techniques, and yield near-optimal rates in both the L 2 µ -and L ∞ µ -norms. See also [30] for some related work in this direction for reduced-order modelling with convolutional autoencoders.

Our work is inspired by recent research on learning holomorphic, Banach-valued functions [2, 5, 6]. We extend both these works, in particular, [5], to learning holomorphic operators. We also significantly improve the error decay rates in [5] with respect to m and show they can be achieved using substantially smaller DNNs with standard training (i.e., ℓ 2 -loss minimization). See Remarks C.1-C.2. Our theoretical guarantees fall into the category of encoder-decoder-nets [52], which includes the well-known PCA-Net [10] and DeepONet [68] frameworks. As in other recent works [16, 30, 55, 66], in Theorems 3.1-3.2 we assume the encoder-decoder pair ( E X , D Y ) in (1.3) have been learned, and focus on the generalization error when training the DNN ̂ N .

2 Notation, assumptions, setup and examples

2.1 Notation

Let ( X , ∥ · ∥ X ) and ( Y , ∥ · ∥ Y ) be Banach spaces and µ be a probability measure on X . Let ( Y ∗ , ∥ · ∥ Y ∗ ) be the dual of Y and B ( Y ∗ ) be its unit ball. The Bochner and Pettis L p -norms of a (strongly and weakly, respectively) measurable operator F : X → Y are defined as

respectively, for 1 ≤ p < ∞ , and analogously for p = ∞ (see, e.g., [8, 44]). Notice that | | | F | | | L p µ ( X ; Y ) ≤ ∥ F ∥ L p µ ( X ; Y ) for 1 ≤ p < ∞ , while | | | F | | | L ∞ µ ( X ; Y ) = ∥ F ∥ L ∞ µ ( X ; Y ) .

Throughout this work, ℓ p ( N ) , 0 < p ≤ ∞ denotes the standard ℓ p space with (quasi-)norm ∥ · ∥ p . We also define the monotone ℓ p space ℓ p M ( N ) as the space of all sequences z = ( z i ) ∞ i =1 ∈ R N whose minimal monotone majorant ˜ z ∈ ℓ p ( N ) . Here z = (˜ z i ) ∞ i =1 is defined as ˜ z i = sup j ≥ i | z j | .

Given a (componentwise) activation function σ , we consider feedforward DNNs of the form

where A l : R N l → R N l +1 are affine maps, and N 0 = n and N L +2 = k . We define width( N ) = max { N 1 , . . . , N L +1 } and depth( N ) = L . We denote a class of DNNs of the form (2.1) with a fixed architecture (i.e., fixed activation function, depth and widths) as N , and write width( N ) = max { N 1 , . . . , N L +1 } and depth( N ) = L .

2.2 Assumptions and setup

Let F : X → Y be the unknown operator we seek to learn and

be approximate encoders and decoders for X and Y , respectively. As mentioned, we assume that these maps have already been learned, and focus on the training of the DNN ̂ N in (1.3). Our main

results allow for arbitrary encoders and decoders (subject to the assumptions detailed below), and provide generalization bounds that are explicit in these terms: specifically, they depend on how well each encoder-decoder pair approximates the respective identity map on X or Y .

In order to formulate the precise notion holomorphy for the operator F , we require the following. Let D = [ -1 , 1] N and ϱ be the uniform probability measure on D . Given ρ > 1 , we define the Bernstein ellipse E ( ρ ) = { ( x + x -1 ) / 2 : x ∈ C , 1 ≤ | x | ≤ ρ } ⊂ C , and, for convenience, we let E (1) = [ -1 , 1] . Next, for ρ = ( ρ i ) i ∈ N ≥ 1 , we define the Bernstein polyellipse as the product E ( ρ ) = E ( ρ 1 ) ×E ( ρ 2 ) ×··· ⊂ C N .

Definition 2.1 (Holomorphic map) . Let ε > 0 , b ∈ ℓ 1 ( N ) with b ≥ 0 . A Banach-valued function f : D →Y is ( b , ε ) -holomorphic if it is holomorphic in the region

See, e.g., [17, 85]. As noted in [7] we can, by rescaling b , assume that ε = 1 . For convenience, we define the following unit ball, consisting of all such functions of norm at most one over R ( b , 1) :

Assumption 2.2. Let D = [ -1 , 1] N and ϱ be the uniform probability measure on D . (A.I) There is a measurable mapping ι : X → R N such that pushforward ς := ι♯µ is a quasi-uniform measure supported on D and ι | supp( µ ) : X → ℓ ∞ ( N ) is Lipschitz with constant L ι ≥ 0 . (A.II) The operator F has the form F = f ◦ ι , where f ∈ H ( b ) for some b ∈ ℓ p M ( N ) and 0 < p < 1 . (A.III) The map E X := ι d X ◦ ˜ D X ◦ ˜ E X is measurable (here ι d X : X → R d X is the restriction of ι , i.e., ι d X ( X ) = ( ι ( X ) i ) d X i =1 ) and the pushforward ˜ ς := E X ♯µ is absolutely continuous with respect to ϱ . (A.IV) The maps D Y and E Y are linear and bounded.

Now let X 1 , . . . , X m ∼ i . i . d . µ and consider the training data

and E i ∈ Y represents noise. Let N be a class of DNNs N : R d X → R d Y , and define

2.3 Discussion of assumptions

We now discuss (A.I)-(A.IV). In §6 we describe future work on relaxing these assumptions.

(A.I) is a weak assumption. It asserts that there is a Lipschitz map ι under which the pushforward of µ is a quasi-uniform measure supported in D . As we discuss in Example 2.3, this is notably the case when µ is the law of some random field with an affine parametrization involving bounded random variables - a situation that occurs frequently in parametric and stochastic PDE problems. (A.II) describes the specific holomorphy of the operator F - see Remark 2.4 for details. Note that we require b ∈ ℓ p M ( N ) , not just b ∈ ℓ p ( N ) . It is known [7] that one cannot learn holomorphic functions (and hence operators) from finite data if b ∈ ℓ p ( N ) only. (A.III) is a relatively weak assumption. In view of (A.I), we expect it to hold as long as the ˜ D X ◦ ˜ E X ≈ I X sufficiently well. Finally, (A.IV) is a standard assumption, which holds for instance in the case of PCA-Net and DeepONet. The former also enforces the learned encoder ˜ E X to be linear, which is not needed in our setup. Moreover, both approaches usually only deal with the case where both X and Y are Hilbert spaces.

Example 2.3 (Parametric PDEs) A common operator learning problem involves learning the map

and F a specifies a certain PDE depending on a parameter or function a . A standard example is the elliptic diffusion equation over a domain Ω ⊂ R n . Here a = a ( x ) ∈ L ∞ (Ω) =: X is the diffusion coefficient and u = u ( · ; a ) is the solution of the PDE

Problems such as (2.6) are ubiquitous in scientific computing, with many applications in engineering, biology, physics, finance and beyond. In many such applications, it is common to assume that the measure µ on X is the law of a random field

for functions a 0 , ϕ i ∈ X , where the x i are random variables and c i ≥ 0 are scalars that ensure that a ∈ L ∞ (Ω) . Under some mild assumptions, (2.8) is then the Karhunen-Loève (KL) expansion of the measure µ . See, e.g., [91] (see also [55, §3.5.1]). The x i are typically independent. While in some settings, they may have infinite support, it is also common in practice to assume they range between finite maxima and minima. After rescaling, one may therefore assume that x ∈ D = [ -1 , 1] N .

Problems of this type fits into our framework. Suppose that x = ( x 1 , x 2 , . . . ) ∼ ϱ . The measure µ is then given as the pushforward µ = a♯ϱ and f : D → Y is the parametric solution map f : x ∈ D ↦→ u ( · ; a ( x )) ∈ Y . If needed, the map ι can be defined in a number of different ways. Suppose, for instance, that X is a Hilbert space, e.g., X = L 2 (Ω) , and { ϕ i } ∞ i =1 is a Riesz system (this holds, for instance, in the case of a KL expansion, in which case { ϕ i } ∞ i =1 is an orthonormal basis). Then { ϕ i } ∞ i =1 has a unique biorthogonal dual Riesz system { ψ i } ∞ i =1 . We may therefore define ι : a ↦→ ( ⟨ a -a 0 , ψ i ⟩ L 2 (Ω) /c i ) ∞ i =1 . Notice that ι is a bounded linear map and F ( X ) = f ◦ ι ( X ) = f ( x ) for X = a ( x ) ∼ µ . However, evaluating ι is often not required for computations (see §A.1).

This example considers an affine parametrization (2.8) inducing the measure µ . Note that other parametrizations can be considered. Common examples include the quadratic a ( z ; x ) = a 0 ( z ) + ( ∑ ∞ i =1 c i x i ϕ i ( z )) 2 and log-transformed a ( z, x ) = exp ( ∑ ∞ i =1 c i x i ϕ i ( z )) parametrizations [17].

Remark 2.4 (Holomorphy assumption) In the previous example, the operator F stems from the solution map f : D →Y of a parametric PDE. The regularity of solution maps of parametric PDEs has been intensively studied, and it is known that many such maps are ( b , ε ) -holomorphic (hence the resulting operator satisfies (A.II)). Consider, for instance, the affine diffusion problem (2.7)-(2.8). Under a mild uniform ellipticity condition, the solution map of the standard weak form of the PDE f : x ∈ D ↦→ u ( a ( · ; x )) ∈ H 1 0 (Ω) is ( b , ε ) -holomorphic with b = ( b i ) ∞ i =1 and b i = c i ∥ ϕ i ∥ L ∞ (Ω) . See, e.g., [3, Prop. 4.9], as well as §B.3. Similar results are known for other parametric PDEs. This includes parabolic PDEs, various types of nonlinear, elliptic PDEs, PDEs over parametrized domains, parametric hyperbolic problems and parametric control problems. See [19] or [3, Chpt. 4] for reviews.

3 Main results I: upper bounds

We now present our first two main results. In these results, given an optimization problem min t f ( t ) , we say that ˆ t is a τ -approximate minimizer for some τ ≥ 0 if f ( ˆ t ) ≤ min t f ( t ) + τ 2 .

Theorem 3.1 (Existence of good DNN architectures) . Let m ≥ 3 , δ > 0 , 0 < ϵ < 1 and L = L ( m,ϵ ) = log 4 ( m ) + log(1 /ϵ ) . Then there exists a class N of hyperbolic tangent (tanh) DNNs N : R d X → R d Y depending on m and ϵ only with

such that following holds. Suppose that Assumption 2.2 holds and

where I X : X → X is the identity map and c > 0 is a universal constant. Let X 1 , . . . , X m ∼ i . i . d . µ and consider the noisy training data (2.4) with arbitrary noise E i ∈ Y . Then, with probability at least 1 -ϵ , every τ -minimizer ̂ N of (2.5) , where τ ≥ 0 is arbitrary, yields an approximation ̂ F that satisfies

and, if Y is a Hilbert space,

Here, the approximation error terms E app ,q , q = 2 , ∞ , are given by

where a Y = ∥D Y ◦ E Y ∥ Y→Y , C ( b , p, ξ ) > 0 depends on b , p and ξ only and θ = 0 if Y is a Hilbert space (as in (3.5) ) or θ = 1 / 2 otherwise (as in (3.3) -(3.4) ). The other terms are given by

where I Y : Y → Y is the identity map and, if ∥ E ∥ 2 2; Y = ∑ m i =1 ∥ E i ∥ 2 Y ,

(Proofs of this and all other theorems are in §C-G of the supplemental material.) This theorem shows that there is a family of tanh DNNs that yield provable bounds for learning holomorphic operators. The error (3.3)-(3.5) decomposes into an approximation error (3.6), which decays algebraically in the amount of training data m . Later, in Theorems 4.1-4.2, we show that these rates are optimal when Y is a Hilbert space, up to log factors. Next, are the encoding-decoding errors (3.7), which depend on how well the approximate encoder-decoder pairs ( ˜ E X , ˜ D X ) and ( E Y , D Y ) approximate the identity maps on X and Y , respectively. Observe that these terms are increasing in m for fixed encoders and decoders. Therefore, as one expects, the accuracy of the encoder-decoder approximations ˜ D X ◦ ˜ E X ≈ I X and D Y ◦ E Y ≈ I Y should increase with increasing m to ensure decay to zero of the generalization error as m →∞ . The specific terms in (3.7) (for q = 2 ) are quite standard in operator learning. See, e.g., [53, 55]. When the encoders and decoders are computed via PCA, as in PCA-Net, standard bounds can be derived for these terms [53]. For similar analysis in the case of DeepONets, see [55]. Finally, the error (3.3)-(3.5) involves an optimization error E opt , which primarily depends on how accurately the optimization problem (2.5) is solved (i.e., the term τ ), and a sampling error E samp , which depends on the error in the training data (2.4).

Theorem 3.1 allows Y to be a Banach or a Hilbert space. Overall, when Y is only a Banach space, we obtain a weaker L 2 µ -norm bound involving the Pettis norm (3.3) and, moreover, the approximation error E app ,q is worse by a factor of 1 / 2 than when Y is a Hilbert space. (Note that one can establish a bound for the Bochner L 2 µ -norm error when Y is a Banach space via (3.4) and the inequality ∥ · ∥ L 2 µ ( X ; Y ) ≤ ∥ · ∥ L ∞ µ ( X ; Y ) . However, we do not believe the resulting bound is sharp). As we discuss in Remark D.18, the discrepancies between the two cases stem from the lack of an inner product structure and, in particular, the absence of Parseval's identity when Y is a Banach space.

Observe that the DNN architecture in Theorem 3.1 is independent of the smoothness of the operator being learned. We term such an architecture problem agnostic . This theorem considers tanh activations only. However, as we discuss in Remark D.11, other activations can be readily used instead. Other key facets of Theorem 3.1 are the width and depth bounds (3.1). Qualitatively, these agree with empirical practice: namely, better performing DNNs tend to be wider than they are deep, and relatively shallow DNNs perform well in practice (see [24, 25] and references therein). We also see this later in §5.

On the other hand, the family N is not fully connected. As we describe in §C.2.1, while the weights on the final layer can be arbitrary real numbers, the weights and biases in the hidden layers come from a finite (but large) set: they are handcrafted to approximately emulate certain multivariate orthogonal polynomials. Since fully-connected DNNs are typically used in practice, Theorem 3.1 is essentially a theoretical contribution. In our next result, we consider the more practical scenario of fully-connected DNNs.

Theorem 3.2 (Fully-connected DNN architectures are good) . There are universal constants c 1 , c 2 , c 3 , c 4 ≥ 1 such that the following holds. Let m , δ , ϵ and L be as in Theorem 3.1,

where c ( δ ) > 0 depends on δ only, consider any class N of fully-connected DNNs satisfying

Suppose that Assumption 2.2 holds and that the pushforward ς in (A.I) is the tensor-product of a univariate probability distribution with mean zero and variance ω ≳ 1 . Let X 1 , . . . , X m ∼ i . i . d . µ and consider (2.4) with arbitrary E i ∈ Y . Then the following hold with probability at least 1 -ϵ .

(A) Uncountably many 'good' minimizers. The problem (2.5) has uncountably many minimizers that satisfy (3.3) with τ = 0 or (3.5) with τ = 0 if Y is a Hilbert space. They also satisfy (3.4) with τ = 0 and the modified right-hand side √ LE app , ∞ + LE X , ∞ + √ LE Y , ∞ + E opt , ∞ + √ LE samp , ∞ .

(B) Good minimizers are stable. Suppose that E X ∈ L ∞ µ ( X ; R d X ) and let τ o > 0 be arbitrary. Then there is a neighbourhood of DNN parameters around the parameters of each minimizer in (A) for which the approximation corresponding to any parameters in this neigbourhood also satisfies the same bounds as in (A) with τ = τ o .

(C) Good minimizers can be far apart in parameter space. For sufficiently large m , there are at least ( m/ ( c 4 L )) 2 δm minimizers satisfying the bounds in (A) such that, for any two such minimizers, their parameters satisfy ∥ θ ′ ∥ = ∥ θ ∥ and ∥ θ ′ -θ ∥ ≳ 1 .

This theorem states that DL with fully-connected DNN architectures of sufficient width and depth (3.10) can succeed, since there are minimizers that yield the optimal bounds of Theorem 3.1. Such minimizers are uncountably many in number (A), stable to perturbations (B) and many of them (exponentially in m ) have sufficiently distinct and nonvanishing/nonexploding parameters (C). This theorem does not imply that all minimizers are 'good' - an issue we discuss further in §6 - but our numerical results in §5 suggest that (approximate) minimizers obtained through training do, at least for the experiments considered, achieve the rates specified in Theorem 3.1.

4 Main results II: lower bounds

We now show that the various approximation errors are nearly optimal. For this, we ignore the encoding-decoding, optimization and sampling errors and proceed as follows. Let C ( X ; Y ) be the Banach space of continuous operators. We term an (adaptive) sampling map as any map

where X 1 ∈ X , X 2 = X 2 ( F ( X 1 )) ∈ X potentially depends on the previous evaluation F ( X 1 ) , X 3 = X 3 ( F ( X 1 ) , F ( X 2 )) ∈ X , and so forth. Next, we term a reconstruction map as any map R : Y m → L 2 µ ( X ; Y ) . Given this, we let H ( b , ι ) = { F = f ◦ ι : f ∈ H ( b ) } and define

where the infimum is taken over all such L and R . In other words, θ m ( b ) measures how well one can learn holomorphic operators using arbitrary training data and an arbitrary reconstruction procedure.

Theorem 4.1 (Optimal L 2 error rates) . Suppose that (A.I) holds. Then, for any 0 < p < 1 there is a constant c ( p ) > 0 such that the following hold.

(i) For each m ∈ N , there is a b ∈ ℓ p M ( N ) , b ≥ 0 , ∥ b ∥ p, M = 1 such that θ m ( b ) ≥ c ( p ) · m 1 / 2 -1 /p .

This theorem shows that the error E app , 2 in Theorems 3.1-3.2 is optimal, up to log terms, whenever Y is a Hilbert space: there does not exist a reconstruction map surpasses the rate m 1 / 2 -1 /p for learning holomorphic operators. Note that this result applies not only to DL-based procedures, but any procedure that learns such operators from m samples. Another consequence of this result is that adaptive sampling, i.e., active learning , is of no benefit. As shown by Theorems 3.1-3.2, the optimal rate m 1 / 2 -1 /p can, up to log terms, be achieved through inactive learning, i.e., i.i.d. sampling from µ .

Theorem 4.1 considers L 2 -norm. For the L ∞ -norm, we present a somewhat weaker result. Let

where the infimum is taken over all reconstruction maps R : ( X × Y ) m → L ∞ µ ( X ; Y ) only.

Theorem 4.2 (Optimal L ∞ error rates) . Suppose that (A.I) holds and that the pushforward ς is the tensor-product of a univariate probability distribution with mean zero and variance ω ≳ 1 . Then, for any 0 < p < 1 there is a constant c ( p ) > 0 such that the following hold.

(i) For each m ∈ N , there is a b ∈ ℓ p M ( N ) , b ≥ 0 , ∥ b ∥ p, M = 1 such that ˜ θ m ( b ) ≥ c ( p ) · m 1 -1 /p log( m ) .

As with the previous theorem, this result asserts that the rate m 1 -1 /p is optimal in the L ∞ -norm when Y is a Hilbert space. However, it is strictly weaker than Theorem 4.1 as it only considers i.i.d. random sampling from µ , as opposed to arbitrary (adaptive) samples. Note that Theorem 4.1 is an extension of [7, Thm. 4.4]. Theorem 4.2 is new, and is of independent interest since it partially addresses an open problem of [7] about deriving lower bounds in the L ∞ µ -norm, as opposed to just the L 2 µ -norm. See §C.2.3-C.2.4 for more discussion.

5 Numerical experiments

We now present numerical results for DL applied to various different parametric PDE problems, as in Example 2.3. For a full description of our experimental setup, see §A-B.

Since the main objective of this work is to examine the approximation error, we follow a standard setup and fix the encoder and decoders for each experiment, so that E X and D Y in (1.3) do not change for different choices of ̂ N . We also set up our experiments so that encoding-decoding (3.7) and sampling (3.8) errors are zero. We do this in a standard way. To ensure that E X ,q = 0 , we truncate the parametric expansions (2.8) after d terms (henceforth termed the parametric dimension ) and define the encoder E X accordingly. This means we effectively consider a parametric PDE depending on finitely-many parameters. We use Finite Element Methods (FEMs) to both solve the PDE (for generating training and testing data) and define the decoder D Y (see (A.2)). To ensure that E Y ,q = 0 , we compute errors with respect to the Bochner L 2 µ ( X ; ˜ Y ) -norm, where ˜ Y = D Y ( R d Y ) is the FEM discretization of Y . In other words, we use the same FEM code to generate test data and compute the errors as we do to construct the operator approximation ̂ F . See §A.1 for further details.

The DNNs in our experiments are fully-connected and of the form (2.1). We denote by σ L × N DNN a DNN ̂ N with activation function σ , width N and depth L . To solve (2.5) we use Adam [47] with early stopping and an exponentially decaying learning rate. We train our DNN architectures for 60,000 epochs and results are averaged over a number of trials. See §A.2 for further details.

Parametric elliptic diffusion equation. Our first example is the parametric elliptic diffusion equation (2.7). This PDE arises in many scientific computing applications, such as groundwater flow modelling, see, e.g., [95]. We describe the full PDE and its FE discretization in §B.3. In our experiments, we consider both affine (B.1) and log-transformed (B.2) diffusion coefficients. The latter is particularly useful in the groundwater flow problem as the permeability of various layers of sediment can vary on logarithmic scales. Differing from most prior work, we consider a novel mixed variational formulation [32] of (2.7), which has a number of key practical benefits (see §B.3.1). In this case, Y = L 2 (Ω) is a Hilbert space. Fig. 1 compares the error versus the amount of training data m for various DNN architectures for learning the solution map of this PDE in d = 4 and d = 8 parametric dimensions with these two diffusion coefficients. We observe that architectures with the Exponential Linear Unit (ELU) or hyperbolic tangent (tanh) activation generally outperform similar architectures with the Rectified Linear Unit (ReLU) activation (as we discuss in Remark D.11, this difference is in agreement with our theoretical analysis). Overall, the best performing DNNs appear to roughly match the plotted rate m -1 . As we explain further in §B.3.2, this rate is precisely that predicted by our theory. In particular, the parametric solution map (recall Remark 2.4) is ( b , ε ) -holomorphic with b ∈ ℓ p M ( N ) for any p < 2 / 3 , giving an effective convergent rate m 1 / 2 -1 /p that is arbitrarily close to m -1 . Another important fact that we observe is that despite the parametric dimension doubling from 4 to 8, there is little change in the error behaviour.

Parametric Navier-Stokes-Brinkman equations. We next consider the parametric Navier-StokesBrinkman (NSB) equations. See §B.4 and (B.14) for the full definition. Here the solution is a pair

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

( u , p ) , where u is the velocity field and p is the pressure. These equations describe the dynamics of a viscous fluid flowing through porous media with random viscosity. See, e.g., [28, 43, 46, 94]. We use a mixed variational formulation [34] to discretize the PDE. This formulation is more sophisticated that standard variational formulations, but conveys various practical advantages. Unlike the previous example, it leads to Y being either Y = L 4 (Ω) for u or Y = L 2 (Ω) for p . See §B.4.1 for details. Fig. 2 compares a variety of DNN architectures for approximating the velocity field component in d = 4 and d = 8 parametric dimensions. Here again we observe the ELU and tanh DNN architectures outperform similar sized ReLU architectures. We also observe a rate close to m -1 . Note that it is currently unknown whether this or the next example possess the same ( b , ε ) -holomorphy guarantee as that of the previous example. Yet we observe the same rate, and therefore conjecture that such a property does indeed hold in these cases. Similar to the previous example, there is also no deterioration of the rate when moving from d = 4 to d = 8 .

Parametric stationary Boussinesq equation. Our final example is a parametric stationary Boussinesq PDE. See §B.5 and (B.16) for the full definition. Here the solution is a triplet ( u , φ, p ) , where u is the velocity field, φ is the temperature and p is the pressure of the solution. The Boussinesq model arises in a variety of engineering, fluid dynamics and natural convection problems where changes in temperature affect the velocity of a fluid [14, 22, 39]. Similar to the previous example, we consider a fully mixed variational formulation (see §B.5.1), which leads to Y = L 4 (Ω) (for u ), Y = L 4 (Ω) (for φ ) or Y = L 2 0 (Ω) (for p ). Fig. 3 provides numerical results. Our observations are in line with the previous two examples, with the ELU and the smaller tanh networks being most often the best performers in this problem. Once more, the errors roughly correspond to the rate m -1 and there is no deterioration with increasing d .

6 Conclusions and limitations

The purpose of this work was to derive near-optimal generalization bounds for learning certain classes of holomorphic operators that arise frequently in operator learning tasks involving PDEs. Complementing and extending previous works [26, 31, 41, 55, 71] on the approximation of such operators via DNNs, we showing sharp algebraic rates of convergence in m , thus confirming that

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

such operators can be learned efficiently and without the curse of dimensionality . It is notable that the sizes of the various DNNs in Theorems 3.1-3.2 also do not succumb to the so-called curse of parametric complexity [54], since the width and depth bounds are at most algebraic in m .

We end by discussing a number of limitations. First, assumption (A.I) may not hold in some applications. The domain D can easily be replaced by bounded hyperrectangle through rescaling and the condition that ς be quasi-uniform relaxed to quasi-ultraspherical (by considering ultraspherical polynomials). However, it is currently an open problem whether our results can be extended to the case where µ is Gaussian, in which case ς would typically be a tensor-product Gaussian measure on R N and the relevant polynomials would be the Hermite polynomials. Second, the reader may have noticed that the encoder E X defined in (A.III) and used to construct the approximation (2.5) involves the pair ( ˜ E X , ˜ D X ) and the map ι d X . This is a technical requirement - also found in other theoretical works on operator learning - needed to obtain encoding-decoding errors of the form E X ,q , q = 2 , ∞ . It is unknown whether it can be relaxed. It is also unknown whether the assumption on ˜ ς in (A.III) can be relaxed. We believe this can be done, at least if the L 2 µ -norm in (3.2) is replaced by the L ∞ µ -norm. Whether this is possible without modifying (3.2) is currently unknown.

Third, a limitation of Theorem 3.2 is that it only asserts that some minimizers are 'good', not all. Techniques from statistical learning theory can provide stronger bounds that hold for all minimizers. Yet, as noted in §1.2, these tools typically produce slower rates of decay in m . Overcoming this limitation - e.g., by refining these tools for the holomorphic setting or showing that the 'good' minimizers can indeed be obtained via standard training - is a topic of future work.

Finally, as noted, our theorems provided worse generalization bounds when Y is a Banach space than when Y is a Hilbert space. Our numerical results in Figs. 2-3 suggest that this factor is an artefact of the proofs. Whether it can be removed is an interesting open problem.

Acknowledgments and Disclosure of Funding

BA acknowledges the support of the Natural Sciences and Engineering Research Council of Canada of Canada (NSERC) through grant RGPIN-2021-611675. ND acknowledges the support of Florida State University through the CRC 2022-2023 FYAP grant program. The authors would like to thank Gregor Maier for helpful comments and feedback.

References

B. Adcock and N. Dexter. The gap between theory and practice in function approximation with deep neural networks. SIAM J. Math. Data Sci. , 3(2):624-655, 2021.

B. Adcock, S. Brugiapaglia, N. Dexter, and S. Moraga. Deep neural networks are effective at learning high-dimensional Hilbert-valued functions from limited data. In J. Bruna, J. S. Hesthaven, and L. Zdeborová, editors, Proceedings of The Second Annual Conference on Mathematical and Scientific Machine Learning , volume 145 of Proc. Mach. Learn. Res. (PMLR) , pages 1-36. PMLR, 2021.

B. Adcock, S. Brugiapaglia, and C. G. Webster. Sparse Polynomial Approximation of HighDimensional Functions . Comput. Sci. Eng. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2022.

B. Adcock, N. Dexter, and S. Moraga. Optimal approximation of infinite-dimensional holomorphic functions II: recovery from i.i.d. pointwise samples. arXiv:2310.16940 , 2023.

B. Adcock, S. Brugiapaglia, N. Dexter, and S. Moraga. Near-optimal learning of Banachvalued, high-dimensional functions via deep neural networks. Neural Networks (in press) , 2024.

B. Adcock, S. Brugiapaglia, N. Dexter, and S. Moraga. Learning smooth functions in high dimensions: from sparse polynomials to deep neural networks. In S. Mishra and A. Townsend, editors, Numerical Analysis Meets Machine Learning , volume 25 of Handbook of Numerical Analysis , pages 1-52. Elsevier, 2024.

B. Adcock, N. Dexter, and S. Moraga. Optimal approximation of infinite-dimensional holomorphic functions. Calcolo , 61(1):12, 2024.

C. D. Aliprantis and K. C. Border. Infinite Dimensional Analysis: A Hitchhiker's Guide . Springer-Verlag, Berlin, Heidelberg, 3rd edition, 2006.

S. Alnæs, J. Blechta, J. Hake, A. Johansson, B. Kehlet, A. Logg, C. Richardson, J. Ring, M. E. Rognes, and G. N. Wells. The FEniCS Project Version 1.5. Archive of Numerical Software , 3 (100), 2015.

K. Bhattacharya, B. Hosseini, N. B. Kovachki, and A. M. Stuart. Model reduction and neural networks for parametric PDEs. SMAI J. Comput. Math. , 7:121-157, 2021.

D. Boffi, F. Brezzi, and M. Fortin. Mixed Finite Element Methods and Applications . Springer, Berlin, Heidelberg, 1st edition, 2013.

S. Brugiapaglia, S. Dirksen, H. C. Jung, and H. Rauhut. Sparse recovery in bounded Riesz systems with applications to numerical methods for PDEs. Appl. Comput. Harmon. Anal. , 53: 231-269, 2021.

S. Cai, Z. Wang, L. Lu, T. A. Zaki, and G. E. Karniadakis. DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks. J. Comput. Phys. , 436:110296, 2021.

C. Cao and J. Wu. Global regularity for the two-dimensional anisotropic Boussinesq equations with vertical dissipation. Arch. Rational Mech. Anal. , 208:985-1004, 2013.

Q. Cao, S. Goswami, and G. E. Karniadakis. LNO: Laplace neural operator for solving differential equations. arXiv:2303.10528 , 2023.

K. Chen, C. Wang, and H. Yang. Deep operator learning lessens the curse of dimensionality for pdes. arXiv:2301.12227 , 2023.

A. Chkifa, A. Cohen, and C. Schwab. Breaking the curse of dimensionality in sparse polynomial approximation of parametric PDEs. J. Math. Pures Appl. , 103(2):400-428, 2015.

P. G. Ciarlet. The Finite Element Method for Elliptic Problems . Society for Industrial and Applied Mathematics, Philadelphia, PA, 2002.

A. Cohen and R. A. DeVore. Approximation of high-dimensional parametric PDEs. Acta Numer. , 24:1-159, 2015.

E. Colmenares, G. N. Gatica, and S. Moraga. A Banach spaces-based analysis of a new fully-mixed finite element method for the Boussinesq problem. ESAIM Math. Model. Numer. Anal. , 54(5):1525-1568, 2020.

D. D˜ ung, V. K. Nguyen, and D. T. Pham. Deep ReLU neural network approximation in Bochner spaces and applications to parametric PDEs. J. Complexity , 79:101779, 2023.

I. Danaila, R. Moglan, F. Hecht, and S. Le Masson. A newton method with adaptive finite elements for solving phase-change problems with natural convection. J. Comput. Phys. , 274: 826-840, 2014.

J. Daws and C. Webster. Analysis of deep neural networks with quasi-optimal polynomial approximation rates. arXiv:1912.02302 , 2019.

M. De Hoop, D. Z. Huang, E. Qian, and A. Stuart. The cost-accuracy trade-off in operator learning with neural networks. J. Mach. Learn. , 1:299-341, 2022.

T. De Ryck, S. Lanthaler, and S. Mishra. On the approximation of functions by tanh neural networks. Neural Networks , 143:732-750, 2021.

B. Deng, Y. Shin, L. Lu, Z. Zhang, and G. E. Karniadakis. Approximation rates of DeepONets for learning operators arising from advection-diffusion equations. Neural Networks , 153: 411-426, 2022.

N. Dexter, H. Tran, and C. Webster. A mixed ℓ 1 regularization approach for sparse simultaneous approximation of parameterized PDEs. ESAIM Math. Model. Numer. Anal. , 53:2025-2045, 2019.

Y. Dutil, D. R. Rousse, N. B. Salah, S. Lassue, and L. Zalewski. A review on phase-change materials: Mathematical modeling and simulations. Renew. Sustain. Energy Rev. , 15(1): 112-130, 2011.

S. Foucart and H. Rauhut. A Mathematical Introduction to Compressive Sensing . Appl. Numer. Harmon. Anal. Birkhäuser, New York, NY, 2013.

N. R. Franco and S. Brugiapaglia. A practical existence theorem for reduced order models based on convolutional autoencoders. arXiv:2402.00435 , 2024.

N. R. Franco, S. Fresca, A. Manzoni, and P. Zunino. Approximation bounds for convolutional neural networks in operator learning. Neural Networks , 161:129-141, 2023.

G. N. Gatica. A Simple Introduction to the Mixed Finite Element Method . SpringerBriefs Math. Springer, Cham, Switzerland, 2014.

G. N. Gatica, R. Oyarzúa, R. Ruiz-Baier, and Y. D. Sobral. Banach spaces-based analysis of a fully-mixed finite element method for the steady-state model of fluidized beds. Comput. Math. Appl. , 84:244-276, 2021.

G. N. Gatica, N. Nuñez, and R. Ruiz-Baier. New non-augmented mixed finite element methods for the Navier-Stokes-Brinkman equations using Banach spaces. J. Numer. Math. , 31(4): 343-373, 2022.

S. Goswami, K. Kontolati, M. D. Shields, and G. E. Karniadakis. Deep transfer operator learning for partial differential equations under conditional shift. Nat. Mach. Intell. , 4(12): 1155-1164, 2022.

S. Goswami, D. S. Li, B. V. Rego, M. Latorre, J. D. Humphrey, and G. E. Karniadakis. Neural operator learning of heterogeneous mechanobiological insults contributing to aortic aneurysms. J. R. Soc. Interface , 19(193):20220410, 2022.

S. Goswami, A. D. Jagtap, H. Babaee, B. T. Susi, and G. E. Karniadakis. Learning stiff chemical kinetics using extended deep neural operators. arXiv:2302.12645 , 2023.

I. Gühring and M. Raslan. Approximation rates for neural networks with encodable weights in smoothness spaces. Neural Networks , 134:107-130, 2021.

Z. Guo, B. Shi, and C. Zheng. A coupled lattice BGK model for the Boussinesq equations. Int. J. Numer. Meth. Fluids , 39:325-342, 2002.

Z. Hao, Z. Wang, H. Su, C. Ying, Y. Dong, S. Liu, Z. Cheng, J. Song, and J. Zhu. GNOT: A General Neural Operator Transformer for Operator Learning. In International Conference on Machine Learning , pages 12556-12569. PMLR, 2023.

L. Herrmann, C. Schwab, and J. Zech. Neural and spectral operator surrogates: unified construction and expression rate bounds. Adv. Comput. Math. , 50:72, 2024.

A. A. Howard, M. Perego, G. E. Karniadakis, and P. Stinis. Multifidelity deep operator networks for data-driven and physics-informed problems. J. Comput. Phys. , 493:112462, 2024.

W. R. Hwang and S. G. Advani. Numerical simulations of Stokes-Brinkman equations for permeability prediction of dual scale fibrous porous media. Phys. Fluids. , 22:113101, 2010.

T. Hytönen, J. van Neerven, M. Veraar, and L. Weis. Analysis in Banach Spaces, Volume I: Martingales and Littlewood-Paley Theory . Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge. Springer, Cham, 2016.

G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang. Physicsinformed machine learning. Nature Reviews Physics , 3(6):422-440, 2021.

K. Khadra, P. Angot, S. Parneix, and J. P. Caltagirone. Fictitious domain approach for numerical modelling of Navier-Stokes equations. Int. J. Numer. Meth. Fluids , 34:651-684, 2000.

D. P. Kingma and J. Ba. Adam: a method for stochastic optimization. arXiv:1412.6980 , 2017.

K. Kontolati, S. Goswami, M. D. Shields, and G. E. Karniadakis. On the influence of overparameterization in manifold based surrogates and deep neural operators. J. Comput. Phys. , 479:112008, 2023.

S. Koric, A. Viswantah, D. W. Abueidda, N. A. Sobh, and K. Khan. Deep learning operator network for plastic deformation with variable loads and material properties. Eng. Comput. , pages 1-13, 2023.

N. Kovachki, S. Lanthaler, and S. Mishra. On universal approximation and error bounds for Fourier neural operators. J. Mach. Learn. Res. , 22(1):13237-13312, 2021.

N. B. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. M. Stuart, and A. Anandkumar. Neural operator: learning maps between function spaces with applications to PDEs. J. Mach. Learn. Res. , 24(89):1-97, 2023.

N. B. Kovachki, S. Lanthaler, and A. M. Stuart. Operator learning: algorithms and analysis. In S. Mishra and A. Townsend, editors, Numerical Analysis Meets Machine Learning , volume 25 of Handbook of Numerical Analysis , pages 419-467. Elsevier, 2024.

S. Lanthaler. Operator learning with PCA-Net: upper and lower complexity bounds. arXiv:2303.16317 , 2023.

S. Lanthaler and A. M. Stuart. The parametric complexity of operator learning. arXiv:2306.15924 , 2024.

S. Lanthaler, S. Mishra, and G. E. Karniadakis. Error estimates for DeepONets: A deep learning framework in infinite dimensions. Trans. Math. Appl. , 6(1):tnac001, 2022.

S. Lanthaler, Z. Li, and A. M. Stuart. The nonlocal neural operator: universal approximation. arXiv:2304.13221 , 2023.

B. Li, S. Tang, and H. Yu. Better approximations of high dimensional smooth functions by deep neural networks with rectified power units. Commun. Comput. Phys. , 27:379-411, 2020.

Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Neural operator: graph kernel network for partial differential equations. In ICLR , 2020.

Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, A. Stuart, K. Bhattacharya, and A. Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems , 33:6755-6766, 2020.

Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. In ICLR , 2021.

Z. Li, M. Liu-Schiaffini, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Learning chaotic dynamics in dissipative systems. Advances in Neural Information Processing Systems , 35:16768-16781, 2022.

Z. Li, N. B. Kovachki, C. Choy, B. Li, J. Kossaifi, S. P. Otta, M. A. Nabian, M. Stadler, C. Hundt, K. Azizzadenesheli, and A. Anandkumar. Geometry-informed neural operator for large-scale 3D PDEs. arXiv:2309.00583 , 2023.

S. Liang and R. Srikant. Why deep neural networks for function approximation? In ICLR , 2017.

C. Lin, Z. Li, L. Lu, S. Cai, M. Maxey, and G. E. Karniadakis. Operator learning for predicting multiscale bubble growth dynamics. J. Chem. Phys. , 154(10), 2021.

C. Lin, M. Maxey, Z. Li, and G. E. Karniadakis. A seamless multiscale operator neural network for inferring bubble dynamics. J. Fluid Mech. , 929:A18, 2021.

H. Liu, H. Yang, M. Chen, T. Zhao, and W. Liao. Deep nonparametric estimation of operators between infinite dimensional spaces. J. Mach. Learn. Res. , 25:1-67, 2024.

J. Lu, Z. Shen, H. Yang, and S. Zhang. Deep network approximation for smooth functions. SIAM J. Math. Anal. , 53(5):5465-5506, 2021.

L. Lu, P. Jin, and G. E. Karniadakis. DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv:1910.03193 , 2019.

L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nat. Mach. Intell. , 3 (3):218-229, 2021.

L. Lu, X. Meng, S. Cai, Z. Mao, S. Goswami, Z. Zhang, and G. E. Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Comput. Methods Appl. Mech. Engrg. , 393:114778, 2022.

C. Marcati and C. Schwab. Exponential convergence of deep operator networks for elliptic partial differential equations. SIAM J. Numer. Anal. , 61(3):1513-1545, 2023.

H. Mhaskar. Neural networks for optimal approximation of smooth and analytic functions. Neural Comput. , 8(1):164-177, 1996.

K. Michałowska, S. Goswami, G. E. Karniadakis, and S. Riemer-Sørensen. Neural operator learning for long-time integration in dynamical systems with recurrent neural networks. arXiv:2303.02243 , 2023.

S. Moraga. Optimal and efficient algorithms for learning high-dimensional, Banach-valued functions from limited samples . PhD thesis, Simon Fraser University, 2024.

H. H. Nguyen and V. H. Vu. Normal vector of a random hyperplane. Int. Math. Res. Not. IMRN , 2018(6):1754-1778, 2018.

F. Nobile, R. Tempone, and C. G. Webster. A sparse grid stochastic collocation method for partial differential equations with random input data. SIAM J. Numer. Anal. , 46(5):2309-2345, 2008.

V. Oommen, K. Shukla, S. Goswami, R. Dingreville, and G. E. Karniadakis. Learning twophase microstructure evolution using neural operators and autoencoder architectures. npj Comput. Mater. , 8(1):190, 2022.

J. A. A. Opschoor and C. Schwab. Deep ReLU networks and high-order finite element methods II: Chebyshev emulation. arXiv:2310.07261 , 2023.

J. A. A. Opschoor, C. Schwab, and J. Zech. Exponential ReLU DNN expression of holomorphic maps in high dimension. Constr. Approx. , 55(1):537-582, 2022.

J. D. Osorio, Z. Wang, G. Karniadakis, S. Cai, C. Chryssostomidis, M. Panwar, and R. Hovsapian. Forecasting solar-thermal systems performance under transient operation using a data-driven machine learning approach based on the deep operator network architecture. Energy Conversion and Management , 252:115063, 2022.

W. Peng, Z. Yuan, Z. Li, and J. Wang. Linear attention coupled fourier neural operator for simulation of three-dimensional turbulence. Physics of Fluids , 35(1), 2023.

M. Pinkus. N -widths in Approximation Theory . Springer-Verlag, Berlin, 1968.

B. Raoni´ c, R. Molinaro, T. Rohner, S. Mishra, and E. de Bezenac. Convolutional neural operators. In ICLR , 2023.

P. I. Renn, C. Wang, S. Lale, Z. Li, A. Anandkumar, and M. Gharib. Forecasting subcritical cylinder wakes with Fourier neural operators. arXiv:2301.08290 , 2023.

C. Schwab and J. Zech. Deep learning in high dimension: neural network expression rates for generalized polynomial chaos expansions in UQ. Anal. Appl. (Singap.) , 17(1):19-55, 2019.

C. Schwab and J. Zech. Deep learning in high dimension: neural network expression rates for analytic functions in L 2 ( R d , γ d ) . SIAM/ASA J. Uncertain. Quantif. , 11(1):199-234, 2023.

C. Schwab, A. Stein, and J. Zech. Deep operator network approximation rates for Lipschitz operators. arXiv:2307.09835 , 2023.

M. Stoyanov. User manual: Tasmanian sparse grids. Technical Report ORNL/TM-2015/596, Oak Ridge National Laboratory, One Bethel Valley Road, Oak Ridge, TN, 2015.

M. Stoyanov, D. Lebrun-Grandie, J. Burkardt, and D. Munster. Tasmanian, 9 2013. URL https://github.com/ORNL/Tasmanian .

M. K. Stoyanov and C. G. Webster. A dynamically adaptive sparse grid method for quasioptimal interpolation of multidimensional functions. Comput. Math. Appl. , 71(11):2449-2465, 2016.

A. M. Stuart. Inverse problems: a Bayesian perspective. Acta Numer. , 19:451-559, 2010.

S. Tang, B. Li, and Y. Haijun. ChebNet: efficient and stable constructions of deep neural networks with rectified power units via Chebyshev approximation. Commun. Math. Stat. (In press) , 2024.

T. Tripura and S. Chakraborty. Wavelet neural operator for solving parametric partial differential equations in computational mechanics problems. Comput. Methods Appl. Mech. Engrg. , 404:115783, 2023.

S. Wang, A. Faghri, and T. L. Bergman. A comprehensive numerical model for melting with natural convection. Int. J. Heat Mass Transfer. , 53(9-10):1986-200, 2010.

C. L. Winter and D. M. Tartakovsky. Groundwater flow in heterogeneous composite aquifers. Water Resour. Res. , 38(8), 2002.

K. Wu, T. O'Leary-Roseberry, P. Chen, and O. Ghattas. Large-scale Bayesian optimal experimental design with derivative-informed projected neural network. J. Sci. Comput. , 95 (1):30, 2023.

D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks , 94:103-114, 2017.

E. Yeung, S. Kundu, and N. Hodas. Learning deep neural network representations for Koopman operators of nonlinear dynamical systems. In 2019 American Control Conference (ACC) , pages 4832-4839. IEEE, 2019.

M. Yin, E. Ban, B. V. Rego, E. Zhang, C. Cavinato, J. D. Humphrey, and G. Em Karniadakis. Simulating progressive intramural damage leading to aortic dissection using DeepONet: an operator-regression neural network. Journal of the Royal Society Interface , 19(187):20210670, 2022.

M. Yin, E. Zhang, Y. Yu, and G. E. Karniadakis. Interfacing finite elements with deep neural operators for fast multiscale modeling of mechanics problems. Comput. Methods Appl. Mech. Engrg. , 402:115027, 2022.

E. Zappala, A. H. d. O. Fonseca, A. H. Moberly, M. J. Higley, C. Abdallah, J. A. Cardin, and D. van Dijk. Neural integro-differential equations. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 11104-11112, 2023.

J. Zech. Sparse-grid approximation of high-dimensional parametric PDEs . PhD thesis, ETH Zurich, 2018.

Z. Zhang, L. Wing Tat, and H. Schaeffer. Belnet: basis enhanced learning, a mesh-free neural operator. Proc. R. Soc. A , 479(2276):20230043, 2023.

M. Zhu, H. Zhang, A. Jiao, G. E. Karniadakis, and L. Lu. Reliable extrapolation of deep neural operators informed by physics or sparse observations. Comput. Methods Appl. Mech. Engrg. , 412:116064, 2023.

Z. Zou, X. Meng, A. F. Psaros, and G. E. Karniadakis. NeuralUQ: A comprehensive library for uncertainty quantification in neural differential equations and operators. SIAM Rev. , 66(1): 161-190, 2024.

A Experimental setup

In this section, we describe our experimental setup.

A.1 Formulation of the learning problems

We first explain how all our experiments are formulated as operator learning problems. We do this in a standard way, by first truncating the random field which generates the measure µ , and then using an FEM to discretize the output space.

All our examples follow Example (2.3) and involve operators of the form (2.6), where F a represents a different type of PDE in each case (specifically, either an elliptic diffusion, Navier-Stokes-Brinkman or Boussinesq PDE). We consider both affine (2.8) and log-transformed parametrizations of the random field a ( x ) . Thus, in general we write

where g : R → R is a (measurable) map. In the affine case g ( t ) = 1 . In the log-transformed case g ( t ) = exp( t ) .

As discussed, since the main objective of this work is to examine the approximation error, we set up our experiments so that encoding-decoding errors are zero. We do this as follows. First, we fix a parametric dimension d and truncate the expansion in (A.1) after d terms, giving a map a d : [ -1 , 1] d →X and measure µ = a d ♯ϱ d , where ϱ d is the uniform probability measure on [ -1 , 1] d . We then define the operator F as F ( a d ( x )) = f ( x ) = u ( · ; a d ( x )) , where u ( · ; a ) is the solution of the PDE F a u = 0 .

In alignment with our theorems, we focus on the in-distribution performance of the learned approximation ̂ F . This means we define the encoder only on supp( µ ) , as E X ( X ) = x when X = a d ( x ) with x ∈ [ -1 , 1] d . As a result, the encoding-decoding error E X ,q in (3.7) satisfies E X ,q = 0 .

To perform our simulations, we use FEMs to solve the PDE and discretize the output space Y . Let { φ i } K i =1 ⊂ Y be a FEM basis and set d Y = K . Then we define the decoder as

and set ˜ Y = D Y ( R K ) as the discretization of Y .

With this in hand, we now describe the simulation of training data in general terms. First, we draw x 1 , . . . , x m ∼ i . i . d . ϱ d . Then, for each training sample x i , we compute Y i by using the FEM to solve the PDE with parameter X i = a d ( x i ) . Notice that Y i ∈ ˜ Y in this setup.

We consider a DNN architecture with input dimension n 0 = d and output dimension n L +2 = K . After training, we evaluate the learned approximation ̂ F = D Y ◦ ̂ N ◦ E X over supp( µ ) as ̂ F ( X ) = D Y ◦ ̂ N ◦E X ( X ) = D Y ◦ ̂ N ( x ) for X = a d ( x ) with x ∈ [ -1 , 1] d . Finally, as noted, we us the same FEM discretization to generate testing data, which allows us to measure the error with respect to the L 2 µ ( X ; ˜ Y ) -norm. This effectively means that the encoding-decoding error E Y ,q in (3.7) satisfies E Y ,q = 0 as well.

A.2 Computational setup for the numerical experiments

In this work, we investigate the trade-off between the accuracy of the learned operator and the number of samples m used in training. Our methodology is summarized as follows.

(i) Implementation . We use the open-source finite element library FEniCS , specifically version 2019.1.0 [9], and Google's TensorFlow version 2.12.0. More information about TensorFlow can be found at https://www.tensorflow.org/ .

(ii) Hardware . We train the DNN models in single precision on the Digital Research Alliance of Canada's Cedar compute cluster (see https://docs.alliancecan.ca/wiki/Cedar ), using

Intel Xenon Processor E5-2683 v4 CPUs with either 125GB or 250GB per node. The setup for each of our PDEs is as follows. For each experiment we consider training with 14 sets of points of size m ∈ { 10 , 20 , 30 , 40 , 50 , 60 , 70 , 80 , 90 , 100 , 200 , 300 , 400 , 500 } and for 6 different architectures (4 x 40 and 10 x 100 with ReLU, ELU, and tanh activations) over two parametric dimensions ( d = 4 and d = 8 ) and two coefficients ( a 1 ,d from (B.1) and a 2 ,d from (B.2)), giving 336 DNNs to be trained for each trial. For the Poisson PDE and Navier-Stokes-Brinkman PDEs we run 12 trials. For the Boussinesq PDE we run 8 trials due to the larger problem size. For the Poisson PDE, we allocate 336 nodes with 1 × 32 core CPUs running 4 threads (totalling 1344 threads, 6.8 GB RAM per node) each running 3 trials, taking approximately 4 hours and 15 minutes to complete. For the Navier-Stokes-Brinkman PDE, we use the same setup allocating 9.88 GB RAM per node and the runs take approximately 9 hours and 13 minutes for each of the two components of the solution to complete. For the Boussinesq PDE, we allocate 336 nodes with 1 × 32 core CPUs running 4 threads per node (totaling 1344 threads, 10 GB RAM per node) each running 2 trials, taking approximately 12 hours and 32 minutes for each of the 3 components to complete. Given this, the total time required to reproduce the results in parallel with the above setup is approximately 60 hours or 2.5 days. Results were stored locally on the cluster and the estimated total space used to store the data for testing and training and results from computation is approximately 50 GB. Trained models were not retained due to space limitations on the cluster.

(iii) Choice of architectures and initialization . Based on the strategies in [1], we fix the number of nodes per layer N and depth L such that the ratio β := L/N is β = 0 . 5 . In addition, we initialize the weights and biases using the HeUniform initializer from keras setting the seed to the trial number. We consider the Rectified Linear Unit (ReLU)

activation functions in our experiments.

(iv) Optimizers for training and parametrization . To train the DNNs, we use the Adam optimizer [47], incorporating an exponentially-decaying learning rate. We train our models for 60,000 epochs or until converging to a tolerance level of ϵ tol = 5 · 10 -7 in single precision. In light of the nonmonotonic convergence behavior observed during the minimization of the nonconvex loss (see, e.g., [1, 2]), we implement early stopping. More precisely, we save the weights and biases of the partially trained network once the ratio between the current loss and the last checkpoint loss is reduced below 1 / 8 , or if the current weights and biases produce the best loss value observed in training. We then restore these weights after training only if the loss value of the current weights is larger than that of the saved checkpoint.

(v) Training data and design of experiments . First, we define a 'trial' as a complete training run for a DNN approximating a specific function, initialized as mentioned above. Following the setup of §A.1, we run several trials solving the problem:

Given training data { ( X i , Y i ) } m i =1 ⊂ ( X × Y ) m , X i ∼ i . i . d . µ, Y i = F ( X i ) + E i ∈ Y , approximate F ∈ L 2 µ ( X ; Y ) .

We generate the measurements Y i using mixed variational formulations of the parametric elliptic, Navier-Stokes-Brinkman and Boussinesq PDEs discretized using FEniCS with input data X i . The noise E i ∈ Y encompasses the discretization errors from numerical solution. Further details of the discretization can be found in §B. Each of our architectures is trained across a range of datasets with increasing sizes. This involves using a set of training data consisting of values { ( X i , Y i )) } i m =1 , where m denotes the size of the training data and belongs to the set { 10 , 20 , 30 , 40 , 50 , 60 , 70 , 80 , 90 , 100 , 200 , 300 , 400 , 500 } . After training we calculate the testing error for each trial and run statistics across all trials for each dataset.

(vi) Testing data and error metric . The testing data is generated similarly to the training data, obtaining solutions at different points X i ∈ X for i = 1 , . . . , m test . However, the testing data

hyperbolic tangent (tanh)

or Exponential Linear Unit (ELU)

{ ( X i , Y i = F ( X i )+ E i ) } m test i =1 is generated using a deterministic high-order sparse grid collocation method [76]. In particular, we use sparse grid quadrature rules to compute approximations to the Bochner norms

where µ = a d ♯ϱ d is the pushforward measure defined in (A.1) and w i , i = 1 , . . . , m test are the quadrature weights. We use these approximations to compute the relative L 2 µ ( X ; ˜ Y ) error

We use a high order isotropic Clenshaw Curtis sparse grid quadrature rule to evaluate e test F , as described in [2]. This method shows superior convergence over Monte Carlo integration to evaluate the global Bochner error. The sparse grid rule gives m test points at a level ℓ for d dimensions. We rely on the TASMANIAN sparse grid toolkit [88-90] for the generation of the isotropic rule to study the generalization performance of the DNN.

(vii) Visualization . The graphs in Figs. 1-3 show the geometric mean (the main curve) and plus/minus one (geometric) standard deviation (the shaded region). We use the geometric mean because our errors are plotted in logarithmic scale on the y -axis. See [3, Sec. A.1] for further discussion about this choice.

B Description of the parametric PDEs used in the numerical experiments and their discretization

In this section, we provide full details of the parametric PDEs considered in our numerical experiments. We also describe their variational formulations and numerical solution using FEM.

B.1 Parametric coefficients

We consider two parametric coefficients of the form (A.1) in our numerical experiments. Our first coefficient is

where x j ∈ [ -1 , 1] , ∀ j . Our second example involves a log-transformed coefficient, which is a rescaling of an example from [76] of a diffusion coefficient with one-dimensional (layered) spatial dependence given by

where x j ∈ [ -1 , 1] , ∀ j . Here we let β c = 1 / 8 , and β p = max { 1 , 2 β c } , β = β c /β p .

In both cases we consider truncation of the expansion after d terms, giving the map a j,d : [ -1 , 1] d → X , with j = 1 corresponding to (B.1) and j = 2 corresponding to (B.2). Our input samples are then { X i = a j,d ( x i ) } i m =1 ⊂ X with x i ∈ [ -1 , 1] d drawn identically and independently from ϱ d and j ∈ { 1 , 2 } . Note for the Boussinesq problem we also consider an additional parametric dependence in the tensor K describing the thermal conductivity of the fluid. See §B.5 and (B.17).

B.2 Relevant spaces

We require several function space definitions for the development of the mixed variation formulations of the Poisson, Navier-Stokes-Brinkman and Boussinesq PDEs. Let Ω ⊂ R n , n ∈ { 2 , 3 } , be the physical domain of a PDE. We write L p (Ω) , 1 ≤ p ≤ ∞ , for the L p -space of scalar-valued functions with respect to the Lebesgue measure (to avoid confusion with the Bochner space L 2 µ ( X ; Y ) ). We denote the standard Sobolev spaces as W s,p (Ω) for s ∈ R and p > 1 , and write H k (Ω) when p = 2 and s = k . Additionally, we consider the space of traces of functions in H 1 (Ω) , denoted by H 1 / 2 ( ∂ Ω) , and its dual, H -1 / 2 ( ∂ Ω) (see, e.g., [11, Sec. 1.2] for further details). We also define the following closed subspace of H 1 (Ω) :

Here C ∞ 0 (Ω) ∥·∥ 1 , Ω denotes the closure of C ∞ 0 (Ω) (i.e., the space of C ∞ (Ω) functions with compact support) with respect to the norm ∥ · ∥ 1 , Ω , which, for any v ∈ H 1 (Ω) , is given by

For scalar functions u and vector fields v , we use ∇ u and div( v ) to denote their gradient and divergence, respectively. For tensor fields σ and τ , represented by ( σ i,j ) n i,j =1 and ( τ i,j ) n i,j =1 , respectively, we define div ( σ ) as the divergence operator div acting along the rows of σ , and we define the trace and the tensor inner-product as

respectively. Furthermore, we introduce the notation L p (Ω) and L p (Ω) to represent the vectorial and tensorial counterparts of L p (Ω) , respectively, and H 1 (Ω) and H 1 (Ω) for the vectorial and tensorial counterparts of H 1 (Ω) , respectively. Keeping this in mind, we introduce the Banach spaces

The cases of q = 4 / 3 and q = 2 appear in the mixed variational formulations of the considered PDEs, and for the latter we simply write H (div; Ω) .

Often, under certain conditions, such as incompressibility conditions [33, eq.(2.4)], it is convenient to define variants of these spaces. For example, we define

which represents the space of integrable functions with zero trace over Ω . Furthermore, given the decomposition (see, e.g., [32])

with norms we may also consider

as the space of elements in H ( div 4 / 3 ; Ω) with zero mean trace. Finally, we define

and the space of L 2 (Ω) functions with zero integral over Ω as

B.3 The parametric diffusion equation

We now describe our first example, which is the parametric elliptic diffusion equation. Let Ω ⊂ R 2 be a bounded Lipschitz domain, ∂ Ω be the boundary of Ω , f ∈ L 2 (Ω) and g ∈ H 1 / 2 ( ∂ Ω) . Given x ∈ [ -1 , 1] d , we consider the PDE

Here a ( x ) = a ( · , x ) ∈ L ∞ (Ω) =: X is the parametric diffusion coefficient. The terms f and g are not parametric.

B.3.1 Mixed variational formulation

Our first step in precisely defining the problem is to identify sufficient conditions for the solution map x ↦→ u ( x ) to be well-defined. To do this, we diverge from the standard variational formulation involving the space Y = H 1 0 (Ω) (see Remark B.3) and instead consider a mixed variational formulation of (B.9). Using a mixed formulation to study the solution of PDEs offers several benefits over the standard variational formulation. One key advantage is that it allows us to introduce additional variables that can be of physical interest. Additionally, mixed formulations can naturally accommodate different types of boundary conditions and introduce Dirichlet boundary conditions directly into the formulation rather than imposing them on the search space. For further details on mixed formulations, we refer to [32] and references within.

Assume that there exists r, M > 0 such that, for all x ∈ [ -1 , 1] d ,

Then the problem can be recast as a first-order system: given x ∈ [ -1 , 1] d , find ( σ , u )( x ) ∈ H (div; Ω) × L 2 (Ω) such that

Here d and b are the bilinear forms defined by

and the functionals G ∈ ( H (div; Ω)) ′ and J ∈ (L 2 (Ω)) ′ are defined by

For the experiments in this work, we consider Ω = (0 , 1) 2 and f = 10 . For the boundary condition, we consider a constant value u ( z , x ) = 0 . 5 on the bottom of the boundary (0 , 1) ×{ 0 } , and zero boundary conditions on the remainder of the boundary.

B.3.2 Holomorphy assumption

Consider the affine parametrization (B.1). Setting M = 2 . 7 and observing that

for some r < 0 . 00762 , we deduce that (B.10) holds, which makes the mixed variational formulation (B.11) well defined, i.e., for each x ∈ [ -1 , 1] ∞ , there exists a unique solution ( σ , u )( x ) ∈ H (div; Ω) × L 2 (Ω) . Moreover, one can show that the parametric solution map x ↦→ ( σ, u )( x ) is ( b , ε ) -holomorphic for 0 < ε < 0 . 00762 and where b = ( b i ) ∞ i =1 is given by b j = ∥ sin( πj · ) /j 3 / 2 ∥ L ∞ (Ω) = j -3 / 2 . See [74, Prop. A.3.2].

X Axis

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

1

1

0.95+

+0.95

0.9

0.9

0.85

0.85

0.8

0.8

0.75

+0.75

0.7+

0.7

0.65

0.65

0.6

0.6

0.55+

0.55

Y Axis 0.5

0.5YAxis

0.45

+0.45

0.4

0.4

0.35

0.35

0.3

0.3

0.25

0.25

0.2+

0.2

0.15+

+0.15

0.1

0.1

0.05

0.05

Fo

60

0.8

0.7

0.6

0.5

0.4

0.3

0.2

01

0

X Axis

1

0

0.10.20.3

0.4

0.5

0.6

0.70.80.9

0.10.20.30.4

0.5

0.60.70.80.9

60

0.9

60

0.9

0.8

0.8

0.8

0.8

0.7-

0.7

0.7-

0.7

0.6

0.6

0.6

0.6

Y Axis 0.5-

0.5Y AxisY Axis 0.5

0.5 Y Axis

0.4

0.4

0.4

0.4

0.3

0.3

0.3

0.3

0.2-

0.2

0.2-

0.2

0.1-

0.1

0.1-

0.1

o

0

0.10.20.30.40.5.0.60.70.80.9

0

0

0.10.20.30.40.5.0.60.70.80.9

0

un

UDNN

-4.4e-03

0.1 0.15 0.2 0.25 0.3 0.35 0.4

5.0e-01

-5.3e-03

0.1 0.15 0.2 0.25 0.3 0.35 0.4

5.1e-01

In view of this property, this example falls within our theory. Note that b ∈ ℓ p M ( N ) for every p < 2 / 3 . Thus, we expect a theoretical rate of convergence with respect to the amount of training data that is arbitrarily close to m 1 / 2 -3 / 2 = m -1 . This holomorphy result applies to the affine diffusion (B.1), not the log-transformed diffusion (B.2). However, we expect that it is possible to extend [74, Prop. A.3.2] to the latter case.

B.3.3 Finite element discretization

We use so-called conforming Finite Element (FE) discretizations [18, Chp. 3]. Given a number of Degrees of Freedom (DoF) K , this results in finite-dimensional spaces H K ⊆ H (div; Ω) and Q K ⊆ L 2 (Ω) . Specifically, we consider a regular triangulation T K of Ω made up of triangles of minimum diameter h min = 0 . 0844 and maximum diameter h max = 0 . 1146 . This corresponds to a total number of DoF K = 2622 . See Fig. 4 for an illustration of the FE mesh.

Fig. 5 shows a comparison between a reference solution computed by the FEniCS FEM solver and the approximation obtained by an ELU 4 × 40 DNN.

B.4 Parametric Navier-Stokes-Brinkman equations

We next consider a parametric model describing the dynamics of a viscous fluid through porous media. Consider a bounded and Lipschitz physical domain Ω ⊆ R 2 . Given x ∈ [ -1 , 1] d , we consider the incompressible nonlinear stationary Navier-Stokes-Brinkman (NSB) equations: find

where λ = Re -1 and Re is the Reynolds number, a ( x ) = a ( · ; x ) ∈ X := L ∞ (Ω) and a : [ -1 , 1] d × Ω → R + is the random viscosity of the fluid, η ∈ R + is the scaled inverse permeability of the porous media, u is the velocity of the fluid, e ( u ) = 1 2 ( ∇ u +( ∇ u ) t ) is the symmetric part of the gradient, p is the pressure of the fluid and f : Ω → R is an external force independent of the parameters. Here, the fourth condition imposes a zero normal Cauchy stress

for the output boundary ∂ Ω out .In addition, the incompressibility of the fluid imposes the following compatibility condition on u D :

The third condition also imposes a no-slip condition on the walls Ω wall [34, eq.(2.3)].

B.4.1 Mixed variational formulation

The analysis of the detailed mixed formulation used for this problem in the nonparametric case can be found in [34]. Over the last decade, many works have used a mixed formulation employing a Banach space framework, allowing one to solve different PDEs in continuum mechanics in suitable Banach spaces. The advantage of this formulation is that no augmentation is required, the spaces are simpler and closer to the original model, and it allows one to obtain more direct approximations of the variables of physical interest [34, Sec. 1].

Based on the analysis in [34], the mixed variational formulation of the parametric NSB equations in (B.14) becomes: given x ∈ [ -1 , 1] d , find ( u , t , σ , γ )( x ) ∈ L 4 (Ω) × L 2 tr (Ω) × H 0 ( div 4 / 3 ; Ω) × L 2 skew (Ω) such that

for all ( v , s , τ , δ ) ∈ L 4 (Ω) × L 2 tr (Ω) × H 0 ( div 4 / 3 ; Ω) × L 2 skew (Ω) . Numerically, the skew-symmetry of γ is imposed by searching for γ ∈ L 2 (Ω) and setting

Moreover, we impose the Neumann boundary condition via a Nietsche method as in [34, Sec. 5.2]. Specifically, we add

to the second equation where κ ≫ 1 is a large constant (e.g., κ = 10 4 ). As usual in this formulation, the pressure p ∈ L 2 (Ω) can be computed according to the post-processing formula

60

0.9

60

0.9

0.8

0.8

0.8

-0.8

0.7

0.7

0.7

0.7

0.6

0.6

0.6

0.6

Y Axis 0.5

0.5Y Axis V Axis 0.5-

0.5 Y Axis

0.4

10.4

0.4

0.4

0.3

0.3

0.2

0.2-

0.2

0.1

0.1-

0.1

0

o

0

0.10.20.3040.50.60.70.80.9

0

1.3e-040.020.030.040.050.060.070.080.091.0e-01

-3.5e-01-0.2-0.100.10.2 0.30.45.4e-01

1

-1

0.9

0.9

0.9

0.9

0.8

-0.8

0.8

0.8

0.7

0.7

0.7

0.7

0.6

0.6

0.6

0.6

Y Axis 0.5

0.5YAxisY Axis 0.5

0.5 Y Axis

0.4

10.4

0.4-

0.4

0.3

0.3

0.3

0.2

0.2

0.2

P

0.1

0.1

-0.1

o

00.10.20.3040.50.60.70.80.9

0

0.10.20.30:40.50.60.70.80.9

AxisPoNN

1.9e-030.020.030.040.050.060.070.08

1.0e-01

-3.5e-01-0.2-0.100.10.2 0.30.45.4e-01

Note that above we omitted the term x for simplicity.

In our experiments, we consider approximating solutions to the parametric NSB problem with λ = 0 . 1 , a scaled inverse permeability of η = 10 + z 2 1 + z 2 2 , an external force f = (0 , -1) ⊤ and random viscosity a j,d as in (B.1)-(B.2) with j ∈ { 1 , 2 } .

We consider the unit square Ω = (0 , 1) 2 as the domain, an inlet boundary ∂ Ω in = (0 , 1) ×{ 1 } , an outlet boundary ∂ Ω out = { 1 }× (0 , 1) and walls ∂ Ω wall = { 0 }× (0 , 1) ∪ (0 , 1) ×{ 0 } . For simplicity, we use the same mesh as that of the previous example. See Fig. 4. On the Neumann boundary ∂ Ω out we consider a zero normal Cauchy stress, a Dirichlet condition u D = (0 . 0625) -1 (( z 2 -0 . 5)(1 -z 2 ) , 0) on ∂ Ω in and a no-slip velocity on ∂ Ω wall .

Fig. 6 provides a comparison between a reference solution of the vector field u and pressure p computed by the FEniCS FEM solver and the approximation generated by a ELU 4 × 40 DNN.

Remark B.1 (Other auxiliary variables) We report the performance of the DNNs approximating ( u , p )( x ) ∈ L 4 (Ω) × L 2 (Ω) . Note that any solver based on the above formulation outputs several other variables, e.g., ( t , σ , γ )( x ) ∈ L 2 tr (Ω) × H 0 ( div 4 / 3 ; Ω) × L 2 skew (Ω) . One could also approximate these auxiliary variables using DNNs. However, we restrict our experiments to ( u , p ) as these are the primary variables of interest in the problem.

To conclude this discussion, in Fig. 7 we plot the numerical results for the approximation of the pressure p in the above problem. This complements Fig. 2, which showed results for the velocity field u . We once more observe similar results: ELU and tanh DNNs outperform ReLU DNNs, the rate of convergence appears to be close to O ( m -1 ) and there is no degradation with increasing parametric dimension d .

B.5 Parametric stationary Boussinesq equation

To recap, in our first example, we considered a mixed formulation of a parametric diffusion equation that provably satisfies the ( b , ε ) -holomorphy assumption. Using this formulation, we considered

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

problems with nonzero Dirichlet boundary conditions, whereas previous works [2, 19, 27] study the more restrictive case of homogeneous Dirichlet boundary conditions, where u ∈ H 1 0 (Ω) . In our next example, we studied a more complicated parametric PDE, namely, the parametric NSB equations. While this example currently lacks a holomorphy guarantee, we observe a convergence rate that aligns with what we expect. We conjecture that the m -1 rate holds both for this and for even more complicated problems. To illustrate this claim with an example, we now consider a parametric coupled partial differential equation in three dimensions ( Ω ⊂ R 3 ) with two random coefficients affecting different parts of the coupled problem. The nonparametric version of this problem is based on [20].

Specifically, we consider the Boussinesq formulation in [20] that combines a parametric incompressible Navier-Stokes equation with a parametric heat equation. The parametric dependence affects both equations. The Navier-Stokes equation is affected by a parametric variable multiplying the temperature-dependent viscosity, and the equation for heat flow is affected directly by the thermal conductivity of the fluid. To be more precise, given x ∈ [ -1 , 1] d , our goal is to find the velocity u : [ -1 , 1] d × Ω → R 2 , pressure p : [ -1 , 1] d × Ω → R and temperature φ : [ -1 , 1] d × Ω → R of a fluid such that

Here g = (0 , 0 , -1) ⊤ is a gravitational force and K ( x ) = K ( · ; x ) ∈ L ∞ (Ω) , where K : [ -1 , 1] d × Ω → R 3 × 3 is a parametric uniformly-positive tensor describing the thermal conductivity of the fluid. It is given explicitly by

for x ∈ [ -1 , 1] d . The term ϖ : R → R + is a temperature-dependent viscosity given by ϖ ( φ ) = 0 . 1 + exp( -φ ) and the term a ( x ) = a ( · ; x ) ∈ L ∞ (Ω) , where a : [ -1 , 1] d × Ω → R is a parametric variable affecting the viscosity of the fluid. As in the previous example e ( u ) is the symmetric part of ∇ u . Note that in this case, we have ( a ( x ) , K ( x )) ∈ X := L ∞ (Ω) × L ∞ (Ω) .

B.5.1 Fully mixed variational formulation

The complete derivation of a fully-mixed variational formulation for the non-parametric Boussinesq equation in Banach spaces can be found in [20, Sec. 3.1]. To make the presentation simpler we rewrite it for the parametric case. Given x ∈ [ -1 , 1] d , find ( u , t , σ , φ, ˜ t , ˜ σ )( x ) ∈ L 4 (Ω) × L 2 tr (Ω) ×

H 0 ( div 4 / 3 ; Ω) × L 4 (Ω) × L 2 (Ω) × H (div 4 / 3 ; Ω) such that

for all ( v , s , τ , ψ, ˜ σ , ˜ τ ) ∈ L 4 (Ω) × L 2 tr (Ω) × H 0 ( div 4 / 3 ; Ω) × L 4 (Ω) × L 2 (Ω) × H (div 4 / 3 ; Ω) . Here p ∈ L 2 0 (Ω) can be recovered as

As in the previous example, we omitted the term x from this equation for simplicity. For further details on this formulation we refer to [20] and references within.

Given x ∈ [ -1 , 1] d , we approximate the solution ( u , p, φ )( x ) ∈ ( L 4 (Ω) × L 2 0 (Ω) × L 4 (Ω)) of (B.18) by using DNNs and study the approximation capabilities as we increase the number of training samples m . As in the previous example (see Remark B.1), we do not aim to approximate the other variables ( t , σ , ˜ t , ˜ σ )( x ) ∈ L 2 tr (Ω) × H 0 ( div 4 / 3 ; Ω) × L 2 (Ω) × H (div 4 / 3 ; Ω) .

In our experiments, we consider the unit cube Ω = (0 , 1) 3 as the domain in R 3 . Weconsider a nonzero boundary condition u D = (1 , 1 , 0) on the bottom face of the cube ∂ Ω bottom = (0 , 1) × (0 , 1) ×{ 0 } , and zero on the other faces. We set φ D = exp(4( -( z 1 -0 . 5) 2 -( z 2 -0 . 5) 2 )) on ∂ Ω bottom and zero otherwise. For simplicity, we consider the same parametric coefficients a 1 ,d and a 2 ,d given by (B.1) and (B.2), respectively. See Fig. 8 for an example of the solution ( u , p, φ )( x ) for a given x ∈ [ -1 , 1] 4 .

To conclude this section, we provide a comparison of the performance of the DNN architectures in approximating the velocity field u and pressure p for the Boussinesq PDE. This complements Fig. 3, which showed results for the temperature φ . As with φ , the convergence rate for p agrees roughly with the rate m -1 , and does not appear to deteriorate with the parametric dimension d . On the other hand, the convergence rate for the velocity field u is somewhat slower.

C Overview of the proofs

In this section, we first introduce additional notation that is needed for the proofs of the main results. We then give a brief overview of the proofs.

C.1 Additional notation

C.1.1 Lipschitz constants

Let ( X , ∥ · ∥ X ) and ( Y , ∥ · ∥ Y ) be Banach spaces, G : X → Y and B ⊆ X . We define the Lipschitz constant as L = Lip( G ; B , Y ) as the smallest constant L ≥ 0 such that

0

XA

1

Y6.5is

0

0.8

1

1

YAgis

0

-0.8

1

YAxis

0.5

0

0.8

0.9

0.8

0.7

0.6

0.4

Z Axis

0.9

0.8

0.7

0.6

0.4

Z Axis

0.9

0.8

0.7

0.6

0.4

Z Axis

0.6

0.6

0.6

Z Axis 0.5

0.2

Z Axis0.5

0.2

ZAxis0.5

0.2

0.4

0.4

0.4

0.3

0.2

0.3

0.2

目

0.3

0.2

0.1

0.5 Y Axis

0.1

0.5Y Axis

0.1

0.5YAxis

9

01

01

01

1.2e-04

0.40.6

0.8

Uh

1.2

1.6e+00

-2.9e-02

0.2

0.4

Ph

0.6

9.2e-01

-1.5e+02

-50

0-

50

100 1.5e+02

Y.Axis

0

0.8

0.8

0.9

0.8

YAgis

00.8

XAxO.A

0.6

0.2

0

0.9

0.8

Z Axis

0.8

0.6

0.4

40.6

0.4

0.2

日

Z Axis

2 Axis 0.5

0.9

0.8

0.7

1

0.61

0.4

0.3

0.7

0.6

0.5 Z Axs

0.4

2 Axig0.5

0.9

0.8

1

0.7

0.6

0.4

0.3

0.7

0.6

0.5 Z Axis

0.4

0.2

0.5 Y Axis

0.2

0.1

0.5 Y Axis

0.2

0.1

0.5 Y Axis

00.8 x0.4

0

01

0.2

01

0.4

0.2

01

7.6e-04

0.4 0.6 0.8

UDNN

1

1.2

1.6e+00

-2.9e-020.2

0.4

PONN

0.6

9.2e-01

-1.5e+02

-50

Pow

0

50

1.5e+02

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

C.1.2 Sequence spaces

We require some notation for sequences. Let ( Z , ∥ · ∥ Z ) be a Banach space, d = N ∪{∞} and write ν = ( ν k ) d k =1 for an arbitrary multi-index in N d 0 . If Λ ⊆ N d 0 is a finite or countable set of multi-indices and 0 < p ≤ ∞ we define the space ℓ p (Λ; Z ) as the set of all Z -valued sequences c = ( c ν ) ν ∈ Λ for which ∥ c ∥ p ; Z < ∞ , where

When ( Z , ∥ · ∥ Z ) = ( R , | · | ) , we just write ℓ p (Λ) and ∥ · ∥ p . We also define the following:

where z ∗ ( c ) = ( z ∗ ( c ν )) ν ∈ Λ ∈ R | Λ | for c = ( c ν ) ν ∈ Λ . Notice that | | | c | | | p ; Z ≤ ∥ c ∥ p ; Z .

Given a sequence c = ( c ν ) ν ∈ Λ , we write

̸

For d ∈ N ∪ {∞} , we write e j , j = 1 , . . . , d , for the canonical basis vectors in R d . We also write 0 for the multi-index of zeros and 1 for the multi-index of ones. Finally, for multi-indices ν = ( ν k ) d k =1 and µ = ( µ k ) d k =1 , we write ν ≥ µ to mean ν k ≥ µ k , ∀ k and likewise for ν > µ .

C.1.3 Weights and weighted sequence spaces

Let d ∈ N ∪{∞} , Λ ⊆ N d 0 and w = ( w ν ) ν ∈ Λ ≥ 0 be a sequence of nonnegative weights. We define the weighted cardinality of a set S ⊆ Λ as

Given a sequence c = ( c ν ) ν ∈ Λ we write

Next, for a Banach space ( Z , ∥ · ∥ Z ) and 0 < p ≤ 2 , we define the weighted ℓ p w (Λ; Z ) space as the space of Z -valued sequences c = ( c ν ) ν ∈ Λ for which ∥ c ∥ p, w ; Z < ∞ , where

Notice that ∥ · ∥ 2 , u ; Z coincides with the unweighted norm ∥ · ∥ 2; Z .

C.1.4 Legendre polynomials

We write { P n } n ∈ N 0 for the classical Legendre polynomials on [ -1 , 1] with normalization P n (1) = 1 . Since ∫ 1 -1 | P n ( x ) | 2 d x = ( n + 1 / 2) -1 , we define the orthonormal (with respect to the uniform probability measure) Legendre polynomials as

Let D = [ -1 , 1] N as before,

be the set of multi-indices with finitely-many nonzero terms and define the multivariate Legendre polynomials as

Here, the second equality follows from the fact that ψ 0 = 1 . Then it is known that the set

constitutes an orthonormal basis for L 2 ϱ ( D ) [19, §3]. For later use, we also define the sequences

and

Here ξ ≥ 0 will be chosen later in the proof.

C.1.5 Miscellaneous

Given an optimization problem min t f ( t ) , we say that ˆ t is a ( σ, τ ) -approximate minimizer for some σ ≥ 1 and τ ≥ 0 if f ( ˆ t ) ≤ σ 2 min t f ( t ) + τ 2 . (In the main paper, we consider only σ = 1 , but for the proofs it is useful to allow σ > 1 ).

Finally, for convenience, given X 1 , . . . , X m ∈ X , we define the semi-norm

of an operator G : X → Y and

of a function g : R N →Y . Here, as in (A.III), ˜ ς denotes the pushforward measure E X # µ .

C.2 Overview of the proofs

C.2.1 Theorem 3.1

Theorems 3.1-3.2 are based on polynomials, and specifically, procedures for learning efficient Legendre polynomial approximations to holomorphic operators. As observed, these results are based on recent work on learning holomorphic, Banach-valued functions [2, 5, 6]. See also Remark C.1 below.

The proof of Theorem 3.1 involves three mains steps.

(a) Formulation (§D.1) and analysis (§D.2-D.4) of a suitable polynomial learning procedure.

(b) Construction of a family of DNNs that approximately emulates the Legendre polynomials (§D.5).

(c) Analysis of the corresponding training problem (2.5) (§D.6-D.8).

Since our goal in this work is 'agnostic' DNN architectures (i.e., independent of the smoothness of the underlying operator), in step (a) we first define a nonlinear set (D.2) spanned by Legendre polynomials with nonzero indices in certain sets of bounded weighted cardinality. This is effectively a form of sparse polynomial approximation , and the analysis of the resulting learning procedure (D.3) relies heavily on techniques from compressed sensing. In order to bound the encoding-decoding error, we also require several results on Lipschitz continuity (Lemma D.2) and norm equivalences (Lemma D.4) for multivariate polynomials.

Step (b) relies on what have now become fairly standard results in DNN approximation theory: namely, the approximate emulation of orthogonal polynomials via DNNs of given width and depth. We present such a result in Lemma D.9, then use this to define the DNN family N in (D.19).

We then analyze the DNN training problem (2.5) in step (c). Using the emulation result of step (b), we first show that any approximate minimizer ̂ N of (2.5) yields a polynomial that is also an approximate minimizer of the polynomial training problem (D.3) (Lemma D.12). We may then apply the results shown in Step (a) to prove a generalization bound (Theorem D.13). Up to this point, we have not used the holomorphy assumption. We now use this assumption to bound the various best polynomial approximation errors that arise in the previously-derived generalization bound (§D.7). Finally, in §D.8 we put all these estimates together to complete the proof.

Remark C.1 Theorem 3.1 is a generalization and improvement of [5, Thms. 4.1 & 4.2], which deals with the case of learning Banach-valued functions rather than operators. Specifically, the setting of [5] can be considered a special case of this paper where X = ℓ ∞ ( N ) and the encoding error E X ,q = 0 . Moreover, Theorem 3.1 improves the main results of [5] in three key ways. First, the the DNN architectures bounds are much narrower: width( N ) ≲ ( m/L ) 1+ δ versus width( N ) ≲ m 3+log 2 ( m ) in the latter. Second, Theorem 3.1 considers standard training, i.e., ℓ 2 -loss minimization, whereas [5, Thms. 4.1 & 4.2] requires regularization. Third, for Banach spaces, the error decay rate with respect

to m is roughly doubled: E app , 2 = O ( m 1 -1 /p ) in Theorem 3.1 versus O ( m 1 / 2(1 -1 /p ) ) in [5, Thm. 4.1]. Finally, Theorem 3.1 also provides bounds in the L ∞ µ -norm, whereas the results in [5] only consider the L 2 µ -norm.

C.2.2 Theorem 3.2

The proof of Theorem 3.2 relies of three key steps.

(a) Using a minimizer of the polynomial training problem (D.3) to construct a family of minimizers of the DNN training problem (2.5).

(b) Analysis of the corresponding minimizers using the previously-derived bound for polynomial minimizers.

(c) Using the Lipschitz continuity of DNNs to show stability of the DNN minimizer and a permutation argument to show the existence of many parameters that lead to equally 'good' minimizers.

Given any minimizer of the polynomial training problem (D.3), it is straightforward to define a DNN with the desired generalization bound. Unfortunately, this will generally not be a minimizer of (2.5). To achieve the aims of step (a) we proceed as follows. First, we note that a DNN will be a minimizer if the corresponding approximation satisfies ̂ F ( X i ) = ˜ Y i , where ˜ Y i are the closest points to the Y i from ˜ Y = D Y ( R d Y ) . To achieve this, we take the existing DNN then add on a suitable number of additional terms corresponding to the first r > m order-one Legendre polynomials (§E.1). We show that by doing this, we can construct a DNN for which ̂ F ( X i ) = ˜ Y i (Lemma E.1).

In Step (b), we first bound this DNN minimizer in terms of the polynomial minimizer plus the contributions of these additional terms (Lemma E.2). The latter involves the minimal singular value of a certain m × r matrix B , which is the matrix of the linear system that enforces the condition ̂ F ( X i ) = ˜ Y i . We bound this minimal singular value in §E.3.

We then use this to complete the proof of part (A) of Theorem 3.2 in §E.4. In this section, we also complete step (c) to establish parts (B) and (C).

Remark C.2 Like Theorem 3.1, Theorem 3.2 also relies on ideas from [5]. However, [5] does not address fully-connected DNN architectures. To address this challenge, the proof of Theorem 3.2 involves the technical construction described above.

C.2.3 Theorem 4.1

Theorems 4.1 is based on [7, Thm. 4.4]. The basic idea is to consider a family of affine, holomorphic operators (F.4). This allows us to lower bound the quantity θ m ( b ) by the so-called Gelfand width (F.1) of a certain weighted unit ball in a finite-dimensional space. Bounds for such Gelfand widths are known, and this allows us to derive the corresponding result. The main difference between this and [7, Thm. 4.4] is the setup leading to the construction in (F.4).

C.2.4 Theorem 4.2

Theorems 4.2 employs similar ideas, but in a more technical manner. We consider a family of linear, holomorphic operators (G.2), which involves a sum over r groups of m +1 coefficients. We restrict to coefficients that lie in the null space of the corresponding sampling operator. Then, through a series of inequalities, we can lower bound ˜ θ m ( b ) by a sum over r terms, each involving the ℓ 1 -norms of certain vectors in the null space of the matrix of the corresponding sampling operator (G.3). This matrix is a subgaussian random matrix. We now use a technical estimate from [75] for vectors in the null space of subgaussian random matrices, which shows that they cannot be 'spiky'. Applying this and a series of further inequalities yields the result.

D Proof of Theorem 3.1

D.1 Formulation of an approximate polynomial training problem

Both Λ and k will be chosen later in the proof. For any Banach space ( Z , ∥ · ∥ Z ) define the space

Then, given the training data (2.4), consider the problem

where ˜ Y = D Y ( R d Y ) . Here and throughout the proofs, we slightly abuse notation: the polynomial p : R N → R , whereas E X has codomain R d X . However, by construction, p is independent of all but the first d X variables. Hence we may consider p as a function R d X → R . This aside, if ˆ p is an approximate minimizer of (D.3), then we define the approximation to F as

Notice that p ◦ E X = D Y ◦ P ◦ E X , where P : R d X → R d Y is a vector-valued polynomial, since, by (A.IV), the map D Y is linear. The idea exploited in this proof is to construct a class of DNNs N such that (i) all such polynomials P are approximated by members of N and (ii) the polynomial training problem (D.3) is approximated by the DNN training problem (2.5). The first step in this analysis is therefore to analyze the polynomial training problem (D.3).

D.2 Supporting lemmas

We require several lemmas. The first relates the L ∞ ϱ -norm of a polynomial to its Pettis L 2 ϱ -norm.

Lemma D.1 (Nikolskii inequality for polynomials) . Let ( Z , ∥ · ∥ Z ) be any Banach space and p = ∑ ν ∈ S c ν Ψ ν for some finite set S ⊂ F , where c ν ∈ Z . Then

Proof. By definition

Fix z ∗ ∈ B ( Z ∗ ) and write z ∗ ( p ) = ∑ ν ∈ S z ∗ ( c ν )Ψ ν . By the triangle inequality and the definition of the weights (C.3), we have

We now apply the Cauchy-Schwarz inequality, (C.1) and Parseval's identity to get

Since z ∗ was arbitrary, we deduce that

as required.

Next, we require the following bound on the Lipschitz constant of a multivariate polynomial.

Lemma D.2 (Lipschitz continuity for polynomials) . Let ( Z , ∥ · ∥ Z ) be any Banach space and suppose that p = ∑ ν ∈ S c ν Ψ ν for some finite S ⊂ F , where c ν ∈ Z . Then satisfies

where B ∞ ( N ) = { x ∈ R N : ∥ x ∥ ∞ ≤ 1 } is the unit ball of ℓ ∞ ( N ) .

Proof. Fix z ∗ ∈ B ( Z ∗ ) and let ˜ p = z ∗ ( p ) = ∑ ν ∈ S z ∗ ( c ν )Ψ ν =: ∑ ν ∈ S ˜ c ν Ψ ν be the corresponding scalar-valued polynomial. Let x , x ′ ∈ D . Then the mean value theorem gives that

for some 0 ≤ t ≤ 1 . Since B ∞ ( N ) ≡ D , this and the fact that D is convex give that

We now consider the terms in the sum separately. Using the definition of the Legendre polynomials (see §C.1.4), we have

̸

The unnormalized Legendre polynomials satisy 1 = P n (1) = ∥ P n ∥ L ∞ ([ -1 , 1]) and n ( n +1) / 2 = P ′ n (1) = ∥ P ′ n ∥ L ∞ ([ -1 , 1]) . Hence

Now, by definition of the weights u ν and v ν (see (C.3) and (C.4)), we have

and

Hence

Here we also used the fact that u ≥ 1 . We now apply this, the Cauchy-Schwarz inequality and Parseval's identity to get

Substituting this into (D.5) now gives

We now recall that ˜ p = z ∗ ( p ) and z ∗ ∈ B ( Z ∗ ) was arbitrary to get

as required.

We deduce that

We now show that this result can be used to imply a norm equivalence for polynomials. For this we first require the following lemma. Note that in this and subsequent results, we abuse notation and write ˜ ς for both the measure on [ -1 , 1] d X defined in (A.III) and the measure on D = [ -1 , 1] N defined by tensoring this measure (corresponding to the first d X variables x 1 , . . . , x d X ) with the uniform measure on D \ [ -1 , 1] d X (corresponding to the remaining variables x d X +1 , x d X +2 , . . . ).

Lemma D.3 (Closeness of L 2 norms) . Let ( Z , ∥ · ∥ Z ) be any Banach space, ς be the measure defined in (A.I) and ˜ ς be the measure defined in (A.III). Suppose that f ∈ L 2 ς ( D ; Z ) is Lipschitz continuous with constant L = Lip( f ; B ∞ ( N ) , Z ) < ∞ and that f depends only on its first d X variables. Then f ∈ L 2 ˜ ς ( D ; Z ) and

Proof. Fix z ∗ ∈ B ( Z ∗ ) , let g = z ∗ ( f ) ∈ L 2 ς ( D ) and set

where

Notice that

Now, with slight abuse of notation, g ( ι ( X )) = g ( ι d X ( X )) due to the assumption on f . Therefore, using this and the Cauchy-Schwarz inequality,

We deduce that a -b ≤ δ . Since z ∗ was arbitrary, we get

which gives the upper bound. The same argument applied to b 2 -a 2 also gives the lower bound.

Lemma D.4 (Norm equivalences for polynomials) . Let ( Z , ∥ · ∥ Z ) be any Banach space, ς be the measure defined in (A.I), ˜ ς be the measure defined in (A.III) and S ⊂ F with supp( ν ) ∈ { 1 , . . . , d X } , ∀ ν ∈ S . Suppose that

for some sufficiently small universal constant c > 0 . Then the norm equivalence

holds for all p = ∑ ν ∈ S c ν Ψ ν , where c ν ∈ Z .

Proof. Combining Lemmas D.2 and D.3, we see that

where δ = 1 2 | | | p | | | L 2 ϱ ( D ; Z ) √ | S | v ∥ ι d X -ι d X ◦ ˜ D X ◦ ˜ E X ∥ L 2 µ ( X ; ℓ ∞ ( N )) ≤ c | | | p | | | L 2 ϱ ( D ; Z ) / 2 . By (A.I), there are constants c 1 ≥ c 2 > 0 such that

We now take c = c 1 / 2 .

D.3 Analysis of (D.3)

We now analyze (D.3). Our analysis relies on the following result, which shows an error bound subject to a certain discrete metric inequality (D.7).

Lemma D.5 (Discrete metric inequality implies error bounds) . Let Q : Y → ˜ Y = D Y ( R d Y ) be a bounded linear operator and π Q = ∥Q∥ Y→Y . Suppose that

for some α > 0 . Then, for any F ∈ L ∞ µ ( X ; Y ) , p ∈ P S ; ˜ Y and q ∈ P S ; Y , we have

and

Proof. By the triangle inequality and properties of Q , we have

Now, since p, Q ◦ q ∈ P S ; ˜ Y , the second term can be bounded by

This yields the first result.

For the second result, we once more write

For the second term, we use (A.III) to write

Notice that p -Q◦ q is a polynomial supported in a set S with | S | u ≤ 2 k . We now apply Lemma D.1 and (D.7) to obtain

This gives the result.

Theorem D.6 (Error bound for polynomial minimizers) . Let Q : Y → ˜ Y = D Y ( R d Y ) be a bounded linear operator, π Q = ∥Q∥ Y→Y and suppose that (D.7) holds. Let ̂ F be as in (D.4) for some ( σ, τ ) -approximate minimizer ˆ p of (D.3) . Then

and

for all q ∈ P S ; Y , where E = ( E i ) i m =1 ∈ Y m is the (Banach-valued) vector of noise terms.

Proof. We apply the previous lemma with p = ˆ p . This gives

Consider the second term. We have

Consider the second term of this expression. By the triangle inequality and the facts that ˆ p is a ( σ, τ ) -minimizer and Q◦ q is feasible, we obtain

Therefore, we get

We now estimate the first term in this expression as follows:

Therefore, we conclude that

Combining this with (D.10) now gives the result.

D.4 Ensuring (D.7) holds with high probability

For the proof of the next lemma and subsequent steps of the proof, we let Λ = { ν 1 , . . . , ν N } and define the matrix

Lemma D.7. Let 0 < ϵ < 1 , 0 < δ < 1 , k > 0 and suppose that

for some universal constant c 0 > 0 . Let T = { c ∈ R N : ∥ c ∥ 2 = 1 , ∥ c ∥ 0 , v ≤ k } and

Then

with probability at least 1 -ϵ , where c 1 > 0 is a universal constant.

Proof. The proof is based on [12, Thm. 2.13]. Since ∥ c ∥ 1 , v ≤ √ ∥ c ∥ 0 , v ∥ c ∥ 2 ≤ √ k , we see that

Define the random vector X ∈ R N by X = (Ψ ν i ( E X ( X ))) N i =1 for X ∼ µ . Observe that

almost surely, by (A.III) and the definition of the weights. Therefore, by [12, Thm. 2.13], if

it holds that

with probability at least 1 -2 exp( -c 2 δ 2 m/k ) . Here c 0 , c 1 , c 2 > 0 are universal constants. Now observe that

Therefore, we have shown that

with probability at least 1 -2 exp( -c 2 δ 2 m/k ) , provided m satisfies (D.12). To conclude the result, we observe that (D.11) implies (D.12), up to a possible change in the universal constant c 0 . Moreover, it also implies that

Hence we obtain the result.

Lemma D.8. There exist universal constants c 0 , c 1 , c 2 > 0 such that the following holds. Suppose that and

Then (D.7) holds with probability at least 1 -ϵ and constant α ≥ c 2 .

Proof. We shall apply the previous lemma with k replaced by 2 k . First, we estimate θ + and θ -. Let c ∈ T , with T as defined therein with 2 k in place of k . Write p = ∑ ν ∈ Λ c ν Ψ ν for the corresponding (scalar-valued) polynomial. Then

and therefore

Since ∥ c ∥ 0 , v ≤ 2 k , (D.14) implies (D.6). We now apply Lemma D.4 and the fact that ∥ p ∥ L 2 ϱ ( D ) = ∥ c ∥ 2 = 1 to get

for universal constants c 4 ≥ c 3 > 0 . Since c ∈ T was arbitrary to deduce that

We now show that (D.7) holds with the desired probability. Let p, q ∈ P S ; ˜ Y be arbitrary. Then their difference h = p -q can be expressed as

Now, this, (D.14) and Lemma D.4 imply that

Therefore, in order to prove (D.7), it suffices to show that | | | h | | | L 2 ϱ ( D ; Y ) ≲ ∥ h ∥ disc , ˜ ς .

Observe that

where we recall that for a vector z = ( z i ) N i =1 ∈ Y N , we write y ∗ ( z ) = ( y ∗ ( z i )) N i =1 ∈ R N . By linearity, we have y ∗ ( Ad ) = A y ∗ ( d ) . Therefore, by Lemma D.7,

for some universal constant c 5 > 0 , provided m satisfies (D.11). We now set δ = c 3 / (2(1 + c 4 ) c 5 ) to get

provided m satisfies (D.11) with this value of δ . However, this is implied by the condition (D.13). We deduce the result.

D.5 Construction of the DNN family N

Recall that Λ ⊂ F , | Λ | = N is an arbitrary set and S is defined by (D.1). In this and what follows, we slightly abuse notation and consider a DNN N : R d → R as a function R N → R which depends on only the first d variables.

Lemma D.9 (Approximating Legendre polynomials with tanh DNNs) . Let Γ ⊂ Λ with supp( ν ) ⊆ { 1 , . . . , d } , ∀ ν ∈ Γ , and m (Γ) = max ν ∈ Γ ∥ ν ∥ 1 < ∞ . There exists a fully-connected family N o of tanh DNNs N : R d → R with

such that, for any 0 < δ < 1 and ν ∈ Γ , there is a DNN N ν ∈ N o with

Moreover, the zero network 0 : x ↦→ 0 also belongs to N o (trivially, since tanh(0) = 0 ).

where d = ( d ν ) ν ∈ Λ ∈ ˜ Y N satisfies

Proof. We follow the argument of [5, Thm. 7.4], which is based on [79, Prop. 2.6]. Let ν ∈ Γ . Then via the fundamental theorem of algebra we can write Ψ ν ( x ) as a product of ∥ ν ∥ 1 numbers as

Here { r ij } ν i j =1 are the roots of the univariate Legendre polynomial P ν i . We append m (Γ) -∥ ν ∥ 1 ones and write Ψ ν ( x ) as a product of exactly m (Γ) numbers. Now define the affine map

so that A ν ( x ) is the vector consisting of the values d i ( x i -r ij ) for j = 1 , . . . , ν i and i ∈ supp( ν ) and 1 otherwise. To complete the proof, we need to construct a tanh DNN mapping R m (Γ) → R that approximately multiplies these numbers. To do this, we argue as in the proof of [5, Thm. 7.4] to see that there is a tanh DNN N ν with

that satisfies the desired bound (D.17). Since these width and depth bounds are independent of ν , we deduce the result.

Fix δ > 0 , let Γ = ∪ S ∈S S , d = d X and consider the corresponding family N o and DNNs N ν , ν ∈ Γ , asserted by this lemma. For any ν ∈ Γ , we have ν ∈ S for some S ∈ S , and any such S satisfies | S | v ≤ k . Therefore u 2(5+ ξ ) ν = v 2 ν ≤ k for any ν ∈ S and we deduce that

This implies that m (Γ) ≤ k 1 / (5+ ξ ) and therefore

With this in hand, we now define the family N of DNNs N : R d X → R d Y by

Here we also use the fact that | S | ≤ | S | v ≤ k . Notice that this family satisfies

due to the bounds for N o .

Now let N ∈ N and write C = [ c 1 | · · · | c ⌊ k ⌋ ] , where c i ∈ R d Y . Then

Therefore, we can associate N with the space of functions ˜ P S ; ˜ Y , where, for any arbitrary Banach space ( Z , ∥ · ∥ Z ) ,

We now require the following lemma, which relates the distance between a function in ˜ P S ; Z and the corresponding polynomial in P S ; Z .

Lemma D.10 (Discrete norms of polynomials and their approximating DNNs) . Let S ⊂ F , ( Z , ∥ · ∥ Z ) be any Banach space and suppose that p = ∑ ν ∈ S c ν Ψ ν , where c ν ∈ Z . Define

Then

Moreover, if Z = ˜ Y , S ∈ S and (D.7) holds with α satisfying δ √ | S | u /α < 1 then

Proof. By (A.III) and the definition of the N ν we have

We now apply the Cauchy-Schwarz inequality and Parseval's identity to obtain

which gives the first result.

For the second result, we apply (D.7) with q = 0 to get

Using (D.21) and the fact that δ √ | S | /α < 1 now gives the result.

Remark D.11 (Other activation functions) As seen in this section, a key step in our proofs is emulating the polynomials via DNNs of quantifiable width and depth. There is an extensive literature on this topic. See, e.g., [5, 21, 23, 25, 38, 57, 63, 67, 72, 78, 79, 85, 86, 92, 97] and references therein. The proof of Lemma D.9 reduces this to the task of emulating the multiplication operation ( x 1 , . . . , x d ) ∈ R d ↦→ x 1 · · · x d ∈ R via a DNN. As shown in the proof of [5, Thm. 7.4] (which is based on [79, Prop. 2.6]), this can in turn be achieved using a binary tree of ⌈ log 2 ( d ) ⌉ DNNs that approximately compute the multiplication of two numbers ( x, y ) ∈ R 2 ↦→ xy ∈ R . Further, this task can be achieved via the identity xy = (( x + y ) 2 -( x -y ) 2 ) / 4 by using a DNN that approximately computes the squaring function x ∈ R ↦→ x 2 ∈ R . To summarize, provide a DNN of quantifiable width and depth can approximately compute the squaring function, it can also approximately emulate the multivariate Legendre polynomials.

In view of this, we can adapt our main theorems to various other activation functions without change. This includes Rectified Polynomial Units (RePUs), where the emulation is, in fact, exact (see, e.g., [57, Lem. 2.1]). It also includes the Exponential Linear Unit (ELU) used in our numerical experiments and many others. See, for instance, Proposition 4.7 of [38] and the ensuing discussion. Rectified Linear Units (ReLUs) are slightly different, as in this case the depth of the DNN that performs the approximation multiplication depends on the desired accuracy (see, e.g., [79, Prop. 2.6]). One could modify Theorem 3.1 to consider ReLU DNNs, with the result being a worse depth bound than that presented for tanh DNNs.

D.6 Analysis of (2.5)

Lemma D.12 (Approximate minimizers of (2.5) yield approximate minimizers of (D.3)) . Suppose that (D.7) holds, let N be the family of DNNs defined in §D.5 and ̂ N be any ( σ, τ ) -approximate minimizer of (2.5) . Let

where S ∈ S and ˆ c ν ∈ ˜ Y , and define

Then ˆ p is a ( σ ′ , τ ′ ) -approximate minimizer of (D.3) , where

Proof. Let p = ∑ ν ∈ S c ν Ψ ν ∈ P S ; ˜ Y be arbitrary and N ∈ N be the corresponding DNN so that D Y ◦ N = ∑ ν ∈ S c ν N ν . Then by the triangle inequality and the fact that ̂ N is an approximate minimizer, we have

We now apply Lemma D.10 to the last two terms, noting that | S | ≤ | S | v ≤ k since S ∈ S , to obtain

Consider the third term. We first apply (D.7) with q = 0 to get

We then use the triangle inequality to get

Therefore, we obtain

Since p ∈ P S ; ˜ Y was arbitrary we get the result.

Theorem D.13 (Error bound for DNN minimizers) . Let Q : Y → ˜ Y = D Y ( R d Y ) be a bounded linear operator, π Q = ∥Q∥ Y→Y and suppose that (D.7) holds with α ≥ c 0 and α -δ √ k ≥ c 1 for suitable universal constants c 0 , c 1 > 0 . Let N be the family of DNNs defined in §D.5 and ̂ N be any ( σ, τ ) -approximate minimizer of (2.5) . Then the approximation ̂ F = D Y ◦ ̂ N ◦ E X satisfies

and

for all q ∈ P S ; Y .

Proof. Let ˆ p be as in Lemma D.12. Then

For the second term, we have, by (A.III) and Lemma D.10,

We now apply this, Lemma D.12, Theorem D.6 and the facts that α -1 ≲ 1 and σ ′ ≥ 1 to obtain

and

for any q ∈ P S ; Y , where σ ′ and τ ′ are as in (D.22). Due to the various assumptions, these satisfy

We deduce that

and

We now bound the final term. Using Lemma D.10 once more, in combination with the fact that α -δ √ k ≳ 1 , we see that

Here, we also used the fact that ̂ N is an approximate minimizer in the fourth step, as well as the facts that the zero network 0 ∈ N and that D Y is a linear map. Plugging this into the previous expressions now gives the result.

D.7 Bounding the best polynomial approximation error terms

When σ = 1 (as will be the case when we come to prove Theorem 3.1), the error bounds in Theorem D.13 involve best polynomial approximation error terms of the form

for arbitrary q ∈ P

In this section, we construct a suitable polynomial q in the case where (A.II) holds and thereby derive a bound for these term. We first require the following lemma.

Lemma D.14. Let G ∈ L ∞ µ ( X ; Y ) and 0 < ϵ < 1 . Then the following hold.

(a) With probability at least 1 -ϵ on the draw of the X i , we have

(b) Suppose that m ≥ 2 r log(2 /ϵ ) for some r > 0 . Then, with probability at least 1 -ϵ on the draw of the X i , we have

Proof. Observe that the random variable | | | G | | | 2 disc ,µ satisfies

For (a), we use Markov's inequality to get

as required.

The proof of (b) is based on [3, Lem. 7.11]. We repeat it here for convenience. Define the random variable Z i = ∥ G ( X i ) ∥ 2 Y and observe that

Let X i = Z i -E ( Z i ) so that

Now let b = ∥ G ∥ 2 L ∞ µ ( X ; Y ) and observe that

Therefore, Bernstein's inequality for bounded random variables (see, e.g., [29, Cor. 7.31]) implies that

for any t > 0 . We now set t = a + b/r and notice that t 2 m/ 2 ab + bt/ 3 ≥ 3 m 5 r ≥ log(2 /ϵ ) . Therefore,

with probability at least 1 -ϵ . It follows that

with the same probability. Substituting the values for a and b now gives the result.

Lemma D.15. Let F ∈ L ∞ µ ( X ; Y ) , q ∈ P S ; Y be arbitrary and m ≥ 2 r log(6 /ϵ ) for some r > 0 and 0 < ϵ < 1 . Then

and

with probability at least 1 -ϵ .

We also have

Proof. We apply part (b) of Lemma D.14 with ϵ replaced by ϵ/ 3 to the term | | | F -q ◦ ι | | | disc ,µ and parts (a) and (b) of Lemma D.14 with ϵ replaced by ϵ/ 3 to the term | | | q ◦ ι -q ◦ E X | | | disc ,µ . Using the union bound, this gives that

with probability at least 1 -ϵ . This yields

with the same probability. It remains to bound the terms involving q ◦ ι -q ◦ E X . Using (A.III), Lemma D.2 and the fact that q depends on its first d X variables only, we see that

Therefore

Substituting this into the previous bounds and using the definition of E X from (A.III) now gives the result.

We are now ready to choose the polynomial q . First, we require the following technical lemma, which shows the existence of multi-index sets of weighted cardinality k which achieve the desired algebraic rates of convergence.

Lemma D.16. Let f = ∑ ν ∈F c ν Ψ ν satisfy f ∈ H ( b ) for some b ∈ ℓ p ( N ) with b ≥ 0 . Then there are index sets S 1 , S 2 ⊂ F with | S 1 | v , | S 2 | v ≤ k such that

where C ( b , p, ξ ) ≥ 0 depends on b , p and ξ only.

Proof. We first show that c ∈ ℓ p v ( F ; Y ) . By definition of H ( b ) (see (2.3)), f is holomorphic in every Bernstein polyellipse E ( ρ ) for which ρ satisfies

Using [4, Lem. 5.3] (which is based on [102, Cor. B.2.7]) with α = β = 0 , we get that ∥ c 0 ∥ Y ≤ 1 and

for all such ρ , where I ( ν , ρ ) = supp( ν ) ∩ { k : ρ k > 1 } . Define the sequence d 0 = 1 and

Using (C.3)-(C.4), we see that and therefore

for all ρ satisfying (D.25) and some γ = γ ( p, ξ ) ≥ 0 . We now use [4, Lem. 5.4] to deduce that d = ( d ν ) ν ∈F ∈ ℓ p ( N ) with ∥ d ∥ p ≤ C ( b , p, ξ ) for some C ( b , p, ξ ) ≥ 0 depending on b , p and ξ only. Returning to c , this gives

Hence c ∈ ℓ p v ( F ; Y ) with ∥ c ∥ p, v ; Y ≤ C ( b , p, ξ ) , as required.

The second step involves the application of the weighted Stechkin's inequality (see [3, Lem. 3.12]). This gives that

for any q ∈ ( p, 2] and k > 0 . Applying this result with q = 2 implies the existence of the set S 1 (recall that ∥ · ∥ 2 , v ; Y = ∥ · ∥ 2; Y ) and applying it with q = 1 implies the existence of the set S 2 .

We now define the set

̸

Notice that Λ HCI n is isomorphic to an index set in N n 0 by the natural restriction map.

Lemma D.17. Let k > 0 and suppose that Λ ⊇ Λ HCI n for some n ∈ N . Let f = ∑ ν ∈F c ν Ψ ν satisfy f ∈ H ( b ) for some b ∈ ℓ p M ( N ) with b ≥ 0 . Then there exists an index set S ∈ S such that

where f S = ∑ ν ∈ S c ν Ψ ν . Moreover, if Y is a Hilbert space, then we also have that

Here C ( b , p, ξ ) > 0 is a constant depending on b , p and ξ only.

Proof of Lemma D.17. The previous lemma implies that there exist index sets S 1 , S 2 ⊂ F with | S 1 | v , | S 2 | v ≤ k/ 2 such that

Now define S = S 1 ∪ S 2 ∩ Λ and notice that S ⊆ Λ and | S | v ≤ | S 1 | v + | S 2 | v ≤ k . Hence S ∈ S . Since v ν ≥ u ν = ∥ Ψ ν ∥ L ∞ ϱ ( D ) , we have (using [5, Lem. 5.1])

Hence, to complete the proof of the first result, we need only show that

for some constant C ( b , p ) ≥ 0 . First, by construction, Λ ⊇ Λ HCI n contains every anchored set of size at most n . See, e.g.. Therefore ∥ c -c Λ ∥ 1 , u ; Y ≤ ∥ c -c S ∥ 1 , u ; Y for any such set S . The result now follows from [5, Cor. 8.2].

Now suppose that Y is a Hilbert space. Then Parseval's identity gives that

As before, it suffices to show that

This follows from the same approach and [5, Cor. 8.2] once more.

D.8 Final arguments

We are now, finally, ready to prove Theorem 3.1. We first consider the case where Y is a Banach space, then treat the case where Y is a Hilbert space afterwards.

Proof of Theorem 3.1 when Y is a Banach space. We divide the proof into a series of steps.

Step 1: Setup and DNN width/depth bounds. Let m , δ , ϵ and L be as in the theorem statement. We may without loss of generality assume that δ ≤ 1 / 5 . Now let

and where Λ HCI n is as in (D.26). We also set

and where c ≥ 1 is a universal constant that will be chosen in the next step. Finally, let δ = 2 -m /m and N by the tanh DNN family (D.19), where the N ν are as in Lemma D.9 for this Λ and value of δ .

By (D.20) and the definition of ξ and k , we immediately see that

This yields the width and depth bounds (3.1). The rest of the proof is therefore devoted to showing the error bounds (3.3)-(3.4).

Step 2: Ensuring (D.7) holds with probability at least 1 -ϵ/ 2 . A standard bound (see, e.g., the proof of Lemma 6.4 in [5]) gives that N = | Λ | satisfies

Here, in the final step we used the fact that m ≥ 3 and L ( m,ϵ ) ≥ 1 . This and the fact that c ≥ 1 also implies that k ≤ m . Therefore, the right-hand side of (D.13) with ϵ replaced by ϵ/ 2 satisfies

Hence, for sufficiently large c ≥ 1 , we deduce that (D.13) holds with ϵ/ 2 . Using (A.I), we see that

Hence (D.14) is implied by (3.2). We conclude from Lemma D.8 that (D.7) holds with probability at least 1 -ϵ/ 2 and α ≳ 1 .

Step 3: Error analysis. Let f ∈ H ( b ) be the function asserted by (A.II) and define q = f S as the polynomial asserted by Lemma D.17 with n as in (D.29). We also observe that

since f ∈ H ( b ) . We now apply Theorem D.13 with σ = 1 and use (D.31), the definition of δ and (3.8) to see that

with probability at least 1 -ϵ/ 2 , where E 2 ( F, q ) and E ∞ ( F, q ) are as in (D.24), Q : Y → ˜ Y = D Y ( R d Y ) is any bounded linear operator and π Q = ∥Q∥ Y→Y .

Notice that Parseval's identity and (D.35) imply that

Hence, this, the previous bounds, Lemma D.15 with ϵ replaced by ϵ/ 4 and the union bound yield

and

with probability at least 1 -3 ϵ/ 4 , for any r such that m ≥ 2 r log(24 /ϵ ) . In particular, we may choose r = k due the definition of k (D.31). We next bound the discrete error | | | F -Q◦ F | | | disc ,µ . Applying Lemma D.14 with ϵ replaced by ϵ/ 8 and the union bound, we see that

with probability at least 1 -ϵ/ 4 , provided m ≥ 2 r log(16 /ϵ ) . In particular, we may take r = k once more. Substituting this into the previous expressions, setting r = k throughout, using the union bound once more and recalling (3.7), (D.31) and (D.34), we deduce that

and

with probability at least 1 -ϵ . This holds for any bounded linear operator Q : Y → ˜ Y = D Y ( R d Y ) . We now set Q = D Y ◦ E Y , which is linear and bounded by (A.IV) with π Q = ∥D Y ◦ E Y ∥ Y→Y = a Y by definition. Using this, we observe that

Substituting this into the previous expressions and recalling (3.7) now gives

Step 4: Bounding the polynomial error terms. It remains to bound the error terms F -q ◦ ι in (D.37). Using (A.I), (A.II) and Lemma D.17, we now notice that

Recall that n ≥ k . Therefore, using this, (D.31) and (3.6), we see that

and likewise for the L ∞ µ -norm bound. Combining this with (D.37) now completes the proof.

Proof of Theorem 3.1 when Y is a Hilbert space. The two differences in theorem statement when Y is a Hilbert space are: (i) the L 2 -norm error is with respect to the stronger Bochner norm, and (ii) the approximation error terms E app ,q , q = 2 , ∞ , are smaller by a factor of 1 / 2 in the exponent (recall (3.6)). We treat both issues separately.

For the (i), we commence with the supporting results in §D.2. First, we note that Lemmas D.1 and D.2 also hold in the Bochner L 2 -norm, since these results already give upper bounds involving the Pettis L 2 -norm. Next, we observe that the proof of Lemma D.3 is readily adapted to yield an equivalent result in the Bochner L 2 -norm with the same constant δ . The same therefore applies to Lemma D.4.

We next consider the analysis of (D.3) in §D.3. If we replace (D.7) by the condition

then the proof of Lemma D.5 yields the same error bounds, except with the Pettis L 2 -norm replaced by the Bochner L 2 -norm. Theorem D.6 is likewise modified to provide a bound in the Bochner L 2 -norm.

Up to this point, we have not used the fact that Y is a Hilbert space. We now need this property. As in §D.4, the next step is to establish that (D.38) holds with high probability. Lemma D.7 is unchanged, therefore our focus is on Lemma D.8. We now describe the steps needed to modify the proof of this lemma to assert (D.38) subject to the same conditions (D.13)-(D.14). First, let { φ i } d Y i =1 be an orthonormal basis of ˜ Y and write each coefficient c ν i of the function h in (D.16) as c ν i = ∑ d Y j =1 b ij φ j for scalars b ij . Then it is a short exercise to write

where b j = ( b ij ) N i =1 . Since b ij = 0 , ∀ j , whenever c ν i = 0 , this vector also satisfies ∥ b j ∥ 0 , v ≤ 2 k . Hence, by Lemma D.7 and Parseval's identity twice,

We now use the bounds (D.15) (which are unchanged) and set δ = c 3 / (2(1 + c 4 ) c 5 ) once more to get

This gives the desired result.

This completes the changes needed in order to analyze the polynomial training problem (D.3). We next consider the DNN training problem (2.5). §D.5 remains unchanged. After reviewing their proofs,

we see that Lemma D.12 and Theorem D.13 both hold with Pettis norms replaced by Bochner norms whenever (D.38) holds instead of (D.7). Finally, we also observe that Lemma D.15 also holds with Pettis norms replaced by Bochner norms, both in the bound and in the definition (D.24) of E 2 ( F, q ) and E ∞ ( F, q ) .

Having completed the changes needed in all the preparatory results, we now follow the same steps as above in the Banach space case. Steps 1 and 2 are unchanged. For Step 3, we go through and replace Pettis norms by Bochner norms throughout. Using this, we obtain (D.37), except with the Bochner norm on the left-hand side in the first inequality, i.e.,

This concludes the changes needed to address (i). To address (ii), we bound the terms F -q ◦ ι . Using (A.I), (A.II), Lemma D.17, the fact that Y is a Hilbert space and the definitions (D.31) and (D.29) of k and n , we see that

and

Substituting this into (D.39) and recalling (3.6) now completes the proof.

Remark D.18 (Differences between the Banach and Hilbert space case) Having seen the proof, we now summarize these differences as follows. First, the matter of whether the Pettis versus Bochner norm can be used reduces to the choice of such norm in the discrete metric inequality (D.7). When Y is a Banach space, we are able to establish this in terms of the Pettis norm subject to a log-linear scaling between m and k (see Lemma D.8 and (D.13)). However, when Y is also a Hilbert space, we can establish the stronger version (D.38) of this inequality by exploiting the additional structure. This, in short, is what leads to the stronger norm bound in this case.

Second, in the Hilbert space case, we get an improved approximation error. This stems from (D.17) and, specifically, the fact that when Y is a Hilbert space we may use Parseval's identity in the Bochner space L 2 ϱ ( D ; Y ) to bound the L 2 ϱ -norm error term via (D.28). This is not possible when Y is a Banach space, so we settle for bounding this term via (D.27) instead.

E Proof of Theorem 3.2

E.1 Setup

As in §D.1, let Λ ⊂ F with supp( ν ) ⊆ { 1 , . . . , d X } , ∀ ν ∈ Λ , and write N = | Λ | . Let S be as in (D.1), r ∈ N , r > max { m,k } (its precise value will be chosen later in the proof) and define the set

Finally, let 0 < δ < 1 and consider the family N o and the tanh DNNs { N ν } ν ∈ Γ whose existence is implied by Lemma D.9. We will specify Λ , k , r and δ later in the proof.

Next, let

be any minimizer of (D.3), where | S | v ≤ k and define

Let

Now, ˜ Y := D Y ( R d Y ) is a finite-dimensional subspace of the Banach space Y . Hence, for any Y ∈ Y there exists a closest point ˜ Y ∈ ˜ Y , i.e., a point satisfying

Given Y 1 , . . . , Y m , let ˜ Y 1 , . . . , ˜ Y m ∈ ˜ Y be the corresponding closest points. Now define

and where z ∈ N ( B ) \{ 0 } and y ∈ ˜ Y , ∥ y ∥ Y = 1 , are arbitrary. Note that such a z exists, since r > m by assumption. Notice that

for coefficients ¯ c ν ∈ ˜ Y . Therefore, we can write

where and C ∈ R d Y × ( ⌊ k ⌋ + r ) . Finally, we define the approximation

and, for convenience,

E.2 Estimation of the DNN minimizer

Lemma E.1 ( ̂ N is a minimizer) . If B is full rank, then

Therefore, ̂ N is a minimizer of (2.5) .

Proof. Observe that

Here, in the penultimate step we use the facts that B y z = y Bz = 0 since z ∈ N ( B ) and BB † = I since r ≥ m and B is full rank by assumption. This gives the first result.

For the second result, we recall that ˜ Y i is a closest point to Y i from ˜ Y = D Y ( R d Y ) . Therefore, for any DNN N ,

as required.

Lemma E.2 (Bounding ̂ F in terms of F ) . Suppose that (D.7) holds with α ≥ c 0 and α -δ √ k ≥ c 1 , and also that √

̂

where c 0 , c 1 , c 2 > 0 are suitable universal constants. Then the approximation ̂ F satisfies ̂

and

̂

Proof. By the triangle inequality,

̂

̂

Consider the second term. We have ̂

where q = ¯ p -˜ p = ∑ r i =1 ( B † e + y z ) i N e i . Lemma D.10 and (A.III) give that

Now consider the other term in (E.7). Define ˜ q = ∑ r i =1 ( B † e + y z ) i Ψ e i . Then Lemma D.10 and (A.III) once more give that

Here, we also used the fact that | Γ o | = r . We now apply Lemmas D.1 and D.4 and (A.III) once more to get

Wenext analyze the term | | | ˜ q | | | L 2 ϱ ( D ; Y ) . Let y ∗ ∈ Y ∗ . Since y ∗ (( B † e + y z ) i ) = ( B † y ∗ ( e )+ y ∗ ( y ) z ) i , we have

Hence, by Parseval's identity,

We now use the definition of e and the inequality | | | e | | | 2; Y ≤ ∥ e ∥ 2; Y to obtain

We next apply the triangle inequality and Lemma D.10 once more to get

Combining this with (E.7) and (E.9), we deduce that

and

It remains to bound the term | | | ˆ p | | | L 2 ϱ ( D ; Y ) . Using (D.7) and the fact that ˆ p is a minimizer of (D.3) and that the zero polynomial is feasible for (D.3), we get

Combining this with the previous bound and (E.6) now completes the proof.

E.3 Estimation of σ min ( B )

Recall that a matrix A ∈ R m × n is a subgaussian random matrix if its entries are i.i.d. subgaussian random variables with mean zero and variance one (see, e.g., [29, Def. 9.1]). The following result can be found in, e.g., [29, Ex. 9.3].

Lemma E.3 (Smallest singular value of a subgaussian random matrix) . Let A ∈ R m × n be a subgaussian random matrix and σ min be the smallest singular value of 1 √ m A . Then, for all 0 < t < 1 ,

where c 1 , c 2 > 0 are universal constants.

Lemma E.4 (Bounding σ min ( B ) ) . Suppose that √ mδ ≤ √ ω/ 8 ,

where ω is the variance of the univariate probability measure as specified in Theorem 3.2 and

for some universal constant c ≥ 1 . Then, with probability at least 1 -ϵ , the matrix B is full rank and

Proof. Define the matrices

Then, since r ≥ m ,

Now, for any c ∈ C r ,

We deduce that ∥ B -B ′ ∥ 2 ≤ √ mδ ≤ √ ω/ 8 . Hence

Now let c ∈ C r and p = ∑ r i =1 c i Ψ e i be the corresponding polynomial. Then, by (A.I), (A.III) and Lemma D.2,

Here, in the third step we used the fact that | Γ o | v = 3 5+ ξ r , since u e i = √ 3 . We deduce that ∥ B ′ -B ′′ ∥ 2 ≤ √ ω/ 8 and therefore

It remains to show that σ min ( B ′′ ) ≥ √ ω/ 2 with high probability. By construction, Ψ e j ( x ) = √ 3 x j . Now recall that the pushforward ς is a tensor-product of a univariate probability measure supported in [ -1 , 1] with mean zero and variance ω > 0 . Therefore

where A = ( ι ( X j ) i / √ ω ) r,m i,j =1 ∈ R r × m . By construction, the entries of A are i.i.d. subgaussian random variables with mean zero and variance one. Hence A is a subgaussian random matrix. We now apply Lemma E.3. Let t = 1 / 4 and observe that

by assumption. Therefore,

This gives the result.

E.4 Final arguments

We are now ready to complete the proof of Theorem 3.2.

Proof of Theorem 3.2, Statement (A). We divide the proof into a series of steps.

Step 1: Setup. Let m , δ , ϵ and L be as in the theorem statement. We once more assume without loss of generality that δ ≤ 1 / 5 . Let n be as in (D.29) and Λ = Λ HCI n , let ξ be as in (D.30) and

where c 1 ≥ 1 is a constant that will be chosen in the next step. Let

where ω is the variance of the univariate probability measure and

where c 2 ≥ 2 will also be chosen in the next step. Note that d X ≥ r > m by assumption. Finally, let S be as in (D.1), Γ be as in (E.1) and { N ν } ν ∈ Γ be the corresponding family of tanh DNNs ensured by Lemma D.9. Finally, let ̂ F be given by (E.5), with DNN ̂ N as in (E.4).

Step 2: ̂ N is a minimizer. By construction,

If ν = e i , then ∥ ν ∥ 1 = 1 . Hence m (Γ) ≤ 1 + max S ∈S m ( S ) ≤ 1 + k 1 / (5+ ξ ) = 1 + k δ (see §D.5). Using the values of k and r , we see that

Therefore, ̂ N ∈ N is feasible for (2.5). Lemma E.1 now implies that ̂ N is a minimizer, provided B is full rank. We will show that this holds in the next step.

Step 3: Ensuring that σ min ( B ) ≥ √ ω/ 4 with probability at least 1 -ϵ/ 4 . We seek to use Lemma E.4. By definition of δ , we have that √ mδ ≤ √ ω/ 8 . Now (D.34), (E.13) and (3.9) imply that (E.10)-(E.11) hold, the latter with ϵ replaced by ϵ/ 4 . Hence Lemma E.4 implies the result.

Step 4: Ensuring (D.7) holds with probability at least 1 -ϵ/ 4 . This step is very similar to Step 2 of the proof of Theorem 3.1. The only difference comes in the estimation of N in (D.32), since now N = | Γ | ≤ | Λ | + r . Since m ≥ 3 , we have

where in the second step we use the fact that log( t ) ≤ t for t > 0 . Conversely, if m ≥ log(1 /ϵ ) , then

Therefore, (D.33) with ϵ/ 4 reads

Hence, due to the definition of k , we get that (D.13) holds once more with probability at least 1 -ϵ/ 4 . The remainder of this step is identical to Step 2 of the proof of Theorem 3.1.

Step 5: Error analysis. We now apply Lemma E.2 to the approximation ̂ F . Since σ min ( B ) ≳ √ ω ≳ 1 , δ ≤ δ √ r ≤ 1 and r ≥ m , we deduce that ̂

and

̂

with probability at least 1 -ϵ/ 2 , where F = ˆ p ◦ E X and ˆ p is the corresponding minimizer of (D.3). We now appeal to Theorem D.6 with σ = 1 and τ = 0 , recalling that α ≳ 1 and δ √ k ≲ 1 , to get that

and, since δ √ r ≲ 1 ,

for any q ∈ P S ; Y and linear operator Q : Y → ˜ Y = D Y ( R d Y ) with π Q = ∥Q∥ Y→Y . Consider the final term. Since ˜ Y i ∈ ˜ Y is the closest point to Y i = F ( X i ) + E i we have

We now use this, the fact that ˆ p is a minimizer and Q◦ q is feasible in combination with triangle inequality to get

Now let f ∈ H ( b ) be the function asserted by (A.II) and q = f S be the polynomial asserted by Lemma D.17. Substituting this into the previous expression, recalling (D.35) and using the definition of δ , we deduce that

with probability at least 1 -ϵ/ 2 . Here E 2 ( F, q ) is as in (D.24), ˜ E ∞ ( F, q ) is as in (D.24) with k replaced by m and E ′ samp , ∞ = ∥ E ∥ 2; Y = √ LE samp , ∞ .

Now observe that the first bound is identical to the corresponding bound in (D.36), except with E opt , 2 replaced by ∥ z ∥ 2 +2 -m . Following the same arguments as in Step 3 of the proof of Theorem 3.1, this gives the corresponding bound in (D.37), which is

The second bound above is identical to the corresponding bound in (D.36), except with E opt , ∞ replaced by √ r ∥ z ∥ 2 +2 -m , E samp , ∞ and E ∞ ( F, q ) replaced by E ′ samp , ∞ and ˜ E ∞ ( F, q ) , respectively, and with √ k replaced by √ m . We once more follow the same arguments as in Step 3, with these changes. This yields the corresponding version of (D.37), which is

Here E ′ X , ∞ = LE X , ∞ and E ′ Y , ∞ = √ LE Y , ∞ .

Having done this, we then use the bounds from Step 4 of the proof of Theorem 3.1, to get

where E ′ app , ∞ = √ LE app , ∞ .

Step 6: Existence of uncountably many minimizers. Let z = ( z i ) r i =1 ∈ N ( B ) \{ 0 } be any vector and consider z 1 = θ 1 z and z 2 = θ 2 z for θ 1 , θ 2 ∈ [ -1 , 1] with θ 1 = θ 2 . Then these vectors define functions ¯ p 1 and ¯ p 2 as in (E.2) and DNNs ̂ N 1 and ̂ N 2 as in (E.3). Suppose that ̂ N 1 = ̂ N 2 . Then, since D Y is linear, we have that ¯ p 1 = ¯ p 2 . But then, by definition and the fact that y ∈ ˜ Y\{ 0 } , we must have

̸

Suppose that θ 1 > θ 2 without loss of generality and let x ∈ D be the vector (sign( z i )) ∞ i =1 . Then, Ψ e i ( x ) = √ 3sign( z i ) and therefore

since ∥ N e i -Ψ e i ∥ L ∞ ϱ ( D ) ≤ δ and δ ≤ 2 -m /r 2 by definition. We now choose z with ∥ z ∥ 2 = 2 -m /r ≤ 2 -m . Note that this choice of z does not change the error bound, except for a constant. It also yields √

̸

which is a contradiction. Hence ̂ N 1 = ̂ N 2 . Thus, we have shown that any θ ∈ [ -1 , 1] leads to a distinct DNN minimizer that satisfies the desired bounds. We get the result.

Step 7: Modifications when Y is a Hilbert space. The modifications required when Y is a Hilbert space are identical to those needed in the proof of Theorem 3.1 (see §D.8). We omit the details.

Proof of Theorem 3.2, Statement (B). Let N = N θ : R d X → R d Y be a tanh DNN, where θ ∈ R D are the network parameters (weights and biases). Let ∥ · ∥ ( d X ) , ∥ · ∥ ( d Y ) and ∥ · ∥ ( D ) be arbitrary norms on R d X , R d Y and R D , respectively. Then, since the activation function is a Lipschitz function, we have

where c θ > 0 is a constant depending on θ .

Now let ̂ N = ̂ N ˆ θ and ̂ F = D Y ◦ ̂ N ◦ E X and consider F = D Y ◦ N ◦ E X for N = N θ . Since D Y : R d Y →Y is linear and therefore bounded, we have

We deduce that

Therefore, there exists a neighbourhood around ˆ θ for which

for all parameters θ in the neighbourhood. The result now follows.

Proof of Theorem 3.2, Statement (C). By construction, the DNN ̂ N defined in (E.4) contains subnetworks that compute the DNNs N e i , i = 1 , . . . , r , which themselves are approximations to the Legendre polynomials Ψ e i . The construction of these subnetworks was described in the proof of

Lemma D.9 as the composition of an affine map defined by the fundamental theorem of algebra and a tanh DNN that approximately multiplies m (Γ) numbers. In this specific case, we have

Therefore, the corresponding affine map (D.18) A e i : R d X → R m (Γ) has a bias vector that is all ones, except for a single entry that has value zero. We deduce that the bias vector b of the full DNN (E.4) contains a subblock of size rm (Γ) that is all ones, except for r zeroes. There are ( rm (Γ) r ) rearrangements of this subblock, each of which leading to a bias vector b ′ with ∥ b -b ′ ∥ ≳ 1 . Moreover, b ′ leads to the same DNN, after permuting the various weight matrices in the corresponding way. Indeed, if P is a permutation matrix, then σ ( Wx + b ) = P -1 σ ( PWx + Pb ) , since σ acts componentwise.

It remains to bound ( rm (Γ) r ) from below. We have

By (E.13), we have that r ≥ 2 m . Now, by the definition (E.1) of Γ ,

where S is as in (D.1) for k as in (E.12) and Λ = Λ HCI n as in (D.26) with n as in (D.29). Now consider the set S = { l e 1 } for some l ∈ N . Then

Therefore, S ∈ S provided (2 l +1) 1 /δ ≤ k and l ≤ n . Set l = ⌊ ( k δ -1) / 2 ⌋ and observe that l ≤ n since k ≤ n and δ ≤ 1 / 5 by assumption. Therefore S ∈ S . Using the definition of k , we get that

for all sufficiently large m . The result now follows.

F Proof of Theorem 4.1

The proof of Theorem 4.1 will follow as a consequence of the following result. For this, we require the following notation. Given 0 < p ≤ ∞ , s ∈ N and a sequence c = ( c i ) ∞ i =1 , we let

̸

where supp( z ) = { i ∈ N : z i = 0 } for z = ( z i ) i ∈ N ∈ R N .

Theorem F.1. For any 0 < p < 1 then term θ m ( b ) defined in (4.2) satisfies

As noted, the proof of this theorem is based on [7, Thm. 4.4]. We recap the details as they will also be needed in the proof of the next result. First, we recall some basic definitions. See [82] or [29, Ch. 10] for more details. Let K be a subset of a normed space ( X , ∥ · ∥ X ) . Then its Gelfand m -width is

An equivalent representation is

The Gelfand width is related to the following quantity:

where ∆ is an arbitrary (potentially nonlinear) reconstruction map and Γ is an adaptive sampling map. By this, we mean that

where Γ 1 : X → R is linear and, for i = 2 , . . . , m , Γ i : X × R i -1 → R is linear in its first component.

Proof of Theorem F.1. We proceed in a series of steps.

Step 1: Setup. Define the functions

Notice that these functions form an orthonormal system in L 2 ϱ ( D ) . (A.I) implies that these functions form a Riesz system in L 2 ς ( D ) , with Riesz constants that are ≍ 1 . Hence they have a (unique) biorthogonal Riesz system { ψ i } ∞ i =1 ⊂ L 2 ς ( D ) . Now define Φ i = ϕ i ◦ ι and Ψ i = ψ i ◦ ι . Let G ∈ L 2 µ ( X ; R ) be arbitrary. Then

Consider the denominator. Using (A.II) and fact that the ψ i form a Riesz system, we see that

We deduce that

Now let b ≥ 0 with b ∈ ℓ 1 ( N ) and I ⊂ N with | I | = N . Using [7, Lem. 5.2] we see that the function

for any y ∈ Y , ∥ y ∥ Y = 1 and c = ( c i ) i ∈ N ⊂ R N with | c | ≤ b (i.e., | c i | ≤ b i , ∀ i ), where c > 0 is a universal constant.

Step 2: Reduction to a discrete problem. Let L and R be arbitrary sampling and reconstruction maps as in (4.2). Following [7, Lem. 5.3], let F = f ◦ ι and observe that

due to (4.1). Notice that Γ is an adaptive sampling map of the form defined above. Now let y ∗ ∈ B ( Y ∗ ) be such that | y ∗ ( y ) | = ∥ y ∥ Y . Then, by (F.3),

and therefore where Γ : R | I | → R m is given by

By biorthogonality, ⟨ y ∗ ( F ) , Ψ i ⟩ L 2 µ ( X ; R ) = c ∥ y ∥ Y c i = c · c i , which implies that

where

Therefore,

where the infimum is taken over all adaptive sampling maps Γ and reconstruction maps ∆ . Since L and R were arbitrary, we get

Step 3: Derivation of the lower bounds. The next step is identical to the proof of Theorem 4.4 in [7]. This gives (F.1).

Proof of Theorem 4.1. We use Theorem F.1. For (i), we let b = ( b i ) ∞ i =1 by defined by

This sequence b ∈ ℓ p M ( N ) with ∥ b ∥ p, M = ∥ b ∥ p = 1 . Moreover, we have

For (ii) we let b = ( b i ) ∞ i =1 be defined by b i = c p ( i log 2 ( i )) -1 /p , where c p = (∑ ∞ i =1 1 / ( i log 2 ( i )) ) -1 /p . This sequence b ∈ ℓ p M ( N ) with ∥ b ∥ p, M = ∥ b ∥ p = 1 . Moreover,

as required.

G Proof of Theorem 4.2

Much as in the previous section, the proof of Theorem 4.2 is a consequence of the following result. Theorem G.1. Suppose that the pushforward ς in (A.I) is a tensor-product of a univariate probability measure. Then the term ˜ θ m ( b ) defined in (4.3) satisfies

Proof of Theorem G.1. We proceed in a similar series of steps to those of the last proof.

Step 1: Setup. Let π : N → N be a bijection that gives a nonincreasing rearrangement of b , i.e., b π (1) ≥ b π (2) ≥ · · · . Now, let r ∈ N be arbitrary and consider the index set

Notice that | I | = r ( m +1) . Define the matrix

and notice that we may write

Let σ be the one-dimensional probability measure associated with ς . Then notice that each A l is a random matrix whose entries are drawn i.i.d. from σ . Since σ is supported in [ -1 , 1] , we deduce that

the A l are independent subgaussian random matrices with the same distribution. Write γ for this distribution. Let t 1 , . . . , t r > 0 and E l,t l be the event

We will make a suitable choice of t 1 , . . . , t r later.

Step 2: Reduction to a discrete problem. Let

Notice that any c ∈ C also satisfies c ∈ N ( A ) . Now let f = f c be as in (F.4) (we make the dependence on c explicit now for convenience). Let x ∈ D . Then

We deduce that where F c = f c ◦ ι . This implies that

where, for convenience, we let L : F ↦→{ X i , F ( X i ) } i m =1 . Therefore,

Now observe that F 0 = 0 and 0 ∈ C . Hence

For a > 0 , the function x ↦→ max { x, a -x } is minimized at x = a/ 2 and takes value a/ 2 there. We deduce that

Now, by (A.I),

With this in hand, we conclude that

We now use the definition of C to write

Step 3: Bounding the expected error. Fix l = 1 , . . . , r and suppose the event E l,t l defined (G.1) occurs. Let u l be the corresponding vector and define

By construction, we have that c l ∈ N ( A l ) and | ( c l ) i | ≤ b i , ∀ i ∈ I l . We deduce that

Now observe that

Therefore, we get that

whenever the event E l,t l occurs. Using the law of total expectation, we deduce that

for any fixed t l > 0 . We now appeal to [75, Thm. 1.4]. This shows that

where c 1 , c 2 > 0 are universal constants. We may without loss of generality assume that c 2 ≥ c 1 ≥ 1 . Now set t l = c 2 log(2 c 2 m 2 ) ≥ c 1 log( m +1) . Hence

We deduce that P ( E t,t l ) > 1 / 2 and therefore

Now observe that

Substituting this into (G.3), we deduce that

Since r was arbitrary, we may take the limit r →∞ . We now use the fact that

to obtain the result.

Proof of Theorem 4.2. Using Theorem G.1, statements (i) and (ii) are derived in exactly the same way as in the proof of Theorem 4.1

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We thoroughly discuss the main claims made in the abstract and introduction and the necessary assumptions to show them. Our main theoretical contributions directly address these claims. We also have several further remarks after these results to provide further context for our work. We also provide detailed numerical experiments showing that DNN architectures compatible with the setup for the theoretical results actually achieve the presented rates of approximation for challenging operator learning problems posed in Banach spaces in terms of the number of samples needed to achieve a given tolerance.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss our main assumptions and their relative strengths in detail in a separate section, §2.3. We also end the paper with a section discussing limitations. See §6.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have a detailed discussion of the assumptions needed to show our theoretical results in §2.2-2.3. We provide further discussion of the results themselves in §3-4 to place the theoretical advancements in this work in the broader context of the operator learning literature. We provide full proofs of our results in the supplemental material.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all of the necessary source code in the supplemental material. Furthermore, we provide a detailed discussion of the setup for the numerical experiments in §A-B of the supplemental material. Given the code and description of the experiments, reproducing the experiments is straightforward.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide all the necessary software to reproduce our experiments along with instructions for running the code to generate the results.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all of the necessary hyperparameters to obtain our experimental results as well as the optimizers used for training and the software used to generate our training and testing data. All of these are open source, no proprietary data was used in this work.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We plot not just the average error over all of the trials but also the (corrected) sample standard deviation of the transformed sample with shaded plots to provide an estimate of the variability in the runs.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The experimental details are provided in the Appendix §A.2. All computational resources are reported, including the type of workers, memory requirements, storage requirements, and time of execution.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have complied with the NeurIPS Code of Ethics in the preparation of this manuscript. No human subject data was used to generate the results for our numerical experiments and data-related concerns are not relevant to this work.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is primarily foundational, and the examples considered do not directly impact society.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work does not release any data or models that have a high risk for misuse. All code is open source and no proprietary data was used in this work.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The code submitted in the supplemental material to both generate the data for training and testing our models and generate the experimental results was written by the authors. No other code or datasets were used in the production of this work.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: New code and data is included as a zip file as supplemental material.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not use crowdsourcing for our experiments and no research was conducted with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not use crowdsourcing for our experiments and no research was conducted with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work examines learning holomorphic operators between Banach spaces. It shows that these learning tasks can be tackled using encoder-decoder approaches with deep neural networks (DNNs), achieving optimal approximation rates up to log factors. The architectures used are problem-agnostic and depend only on the amount of available training data. The study also delves into theoretical aspects of the associated training problem for standard feed-forward networks and presents numerical findings to support its claims.\n\nSTRENGTHS:\n* The paper is well-written and includes a thorough review of current research on learning holomorphic operators.\n* The results are new and interesting, expanding on previous work by addressing holomorphic mappings between arbitrary Banach spaces and allowing standard training methods for the DNNs utilized.\n* Although some assumptions and results are technical, I appreciate how the authors provide a clear overview of their meaning and interpretation. The proof techniques are explained well and summarized in the appendix, making them accessible without diving into deep details.\n* The theoretical findings are also backed up by practical experiments.\n\nWEAKNESSES:\nOverall I do not have any serious concerns about this work and consider it to be a valuable contribution. Some questions are listed below.\n\nQUESTIONS:\n* Perhaps I missed it, but I could not locate a precise reference discussing the holomorphicity of the Navier-Stokes-Brinkman and Boussinesq equations, similar to Section B.3.2. for the Diffusion problem. It would be helpful if such a discussion could be included in the manuscript.\n\n* Could you briefly summarize to what extent Theorems 4.1 and 4.2 require different techniques compared to those used in [7]? How does the use of a general Banach space $\\mathcal{X}$ complicate the analysis?\n\n* Regarding the experimental results, do you have any insights into why different activation functions perform differently across individual tasks? Can this behavior be explained using the results from Theorems 3.1 and 3.2?\n\n* Do the authors think the results from Theorem 3.2 could be extended to include information on optimization procedures used for solving (2.5) (e.g., stochastic gradient descent and variants thereof)? Such an extension would enhance the practical relevance of the result. I am only seeking an educated guess or a high-level opinion, without expecting detailed analysis.\n\n* I would appreciate it if the authors could briefly summarize the differences between the Banach space and Hilbert space cases for $\\mathcal{Y}$. Specifically, could you indicate precisely where the inner product structure is utilized in the proofs to achieve better rates?  Also please double check formulas (3.3) and (3.5) in this context, currently it seems that they are the same.\n\nLIMITATIONS:\nLimitations and social impact have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors provide theoretical results on sample efficiency for learning holomorphic neural operators $\\mathcal{X}\\rightarrow\\mathcal{Y}$ (results are available for both Hilbert and Banach spaces) with a neural network of a particular structure. The structure of the network is encoder $\\mathcal{X}\\rightarrow\\mathbb{R}^{d_1}$, decoder $\\mathcal{Y}\\rightarrow\\mathbb{R}^{d_2}$ and DNN $\\mathbb{R}^{d_1}\\rightarrow \\mathbb{R}^{d_2}$, with both encoder and decoder considered to be given and DNN trained on $l_2$ loss in a usual supervised manner.\n\nBesides the upper bound on approximation error, authors also provide lower bound and conclude that active learning (deliberate collection of data used to train neural operator) is not justified, since non-adaptive training is already optimal up to log terms.\n\nAuthors illustrate their theoretical results (relative test error vs number of samples for selected DNN configurations) with several numerical experiments, including diffusion equation in $D=2$, Navier-Stokes-Brinkman equation in $D=2$ and Boussinesq equation in $D=3$.\n\nSTRENGTHS:\nOperator learning is a relatively recent DL setup, so novel theoretical results are of considerable interest. Authors managed to significantly improve previously known bounds and show that DNN can obtain near-optimal performance (in terms of the number of samples needed) in non-adaptive iid settings.\n\nThe contribution is primarily theoretical, but formal statements are well explained, supplemented with discussions and examples which significantly simplifies understanding of the results.\n\nWEAKNESSES:\nOverall, in my opinion, the research is of excellent quality, so I can not point to any major weaknesses.\n\nThe two main issues that can be relevant are: (i) the requirements for the encoder and decoder are not sufficiently discussed, and I find restricting the assumption that they are already available, and (ii) the significance of the numerical confirmation is not apparent.\n\nQUESTIONS:\n1. After line 102 there is a definition of Bochner and Pettis $L^p$ norms. Presumably, the later expression is a definition of $|||F|||$ which is not apparent.\n2. I want to clarify several statements on the encoder and decoder. \n   1. In (1.3) authors introduce encoder $\\mathcal{E}$ and decoder $\\mathcal{D}$, and later approximate encoder and decoder $\\widetilde{\\mathcal{E}}$, $\\widetilde{\\mathcal{D}}$ after line 114. What is the difference between those?\n   2. In particular, in the discussion (lines 145, 146) of assumption A.IV (line 131) that requires both $\\mathcal{E}$ and $\\mathcal{D}$ to be linear and bounded, authors claim that $\\widetilde{\\mathcal{E}}$ is not required to be linear. It is not clear to me how this agrees with the fact that $\\mathcal{E}_{\\mathcal{X}}$ is defined in A.III as a composition of approximate encoder and decoder. Can the authors please clarify that?\n   3. According to (1.3) encoder and decoder are defined for the approximate operator $\\hat{F}$. This means, as I understand, they are not rigidly fixed by $F$ since presumably $F$ can be approximated with many different encoders and decoders. Approximate encoders and decoders are not formally defined, so it is not clear what they approximate. Does it all mean that authors consider not a continuous operator learning problem but suppose that discretization (of the space of parameters and $\\mathcal{Y}$) is already performed ($\\mathcal{E}$ and $\\mathcal{D}$ are selected) and $\\widetilde{\\mathcal{E}}$ and $\\widetilde{\\mathcal{D}}$ approximate $\\mathcal{E}$ and $\\mathcal{D}$?\n   4. Why there are $\\widetilde{\\mathcal{E}}\\_{\\mathcal{X}}$ and $\\widetilde{\\mathcal{D}}\\_{\\mathcal{X}}$ but not the approximate encoder and decoder for $\\mathcal{Y}$?\n3. Several questions on Theorem 3.1:\n   1. Why $a_{\\mathcal{Y}}$ is called approximation error? What is the meaning of this term (it looks like some kind of volume)? In the ideal setting $\\mathcal{D}\\_{\\mathcal{Y}}\\circ\\mathcal{E}\\_{\\mathcal{Y}} = I$, so we have a perfect approximation that has nonzero approximation error.\n   2. Since $\\lim_{m\\rightarrow\\infty}m \\big/ L(m) = \\infty$ many terms in the upper bound will diverge unless the encoder and decoder are of sufficient quality. More specifically, the encoder and decoder should improve their accuracy as $\\sim m^{-1/2}$ for u.b. to be useful in case of $2$ norm and as $\\sim m^{-1}$ in case of Chebyshev norm. Can the authors please discuss this consequence of their result in more detail?\n   3. Can the authors also explain the consequence of the optimization error term? It seems that $\\tau$ that appears in it is to some extent beyond control and completely independent of $m$.\n4. Questions about experiments:\n   1. In experiments authors strive to confirm theoretical scaling $m^{-1}$. Since the encoder and decoder are perfect, and noise is absent, the only two sources of error are approximation error and optimization error. In my opinion, one can clearly see the signs of saturation for larger $m$. This is also clear from operator learning literature that relative error rarely fell below $10^{-3}$. Can the authors please comment on that? Is it because optimization error prevails for large $m$?\n   2. Dimensions $d=4,8$ in Figures 1, 2, 3 and in the text (e.g., lines 296, 297: \"PDE in $d=4$ and $d=8$ dimensions\") are slightly misleading because they refer to the dimension of the parameter space and can be confused with the number of spatial dimensions.\n   3. All results are given for perfect encoder and decoder. Can the authors share some thoughts on whether it is possible to test their upper bound with an imperfect decoder and encoder? What are those experiments might be? Is it possible to create a controllable setting to test terms in ub that contain encoder and decoder errors?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper addresses the challenge of learning operators between Banach spaces, focusing on holomorphic operators which have many applications. The authors employ standard (DNN) architectures with constant width greater than depth and $\ell_2$-loss minimization. They identify a family of DNNs that achieve optimal generalization bounds for such operators. For fully-connected architectures, they show there are infinitely many solutions that offer optimal performance. The results demonstrate that deep learning achieves the best possible generalization bounds, and no method can surpass these bounds, up to logarithmic terms.

STRENGTHS:
This work presents powerful and well-formulated theorems. The techniques used on holomorphic operators are interesting, and thanks to them, a generalization bound can be demonstrated. The work is novel, highly original, and I believe it could have a fruitful impact in the context of deep learning.

WEAKNESSES:
I think some of the figures of the experiments results are too small, making it a bit difficult to recognize the results. It would be good if the authors could review this issue.

QUESTIONS:
No questions.

LIMITATIONS:
Theorem 3.2 only asserts that some minimizers are ‘good’, not all.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12