Towards Size-Independent Generalization Bounds for Deep Operator Nets
Authors: Pulkit Gopalani
Sayar Karmakar
Dibyakanti Kumar
Anirbit Mukherjee
Venue: Accepted by TMLR ()

Abstract: In recent times machine learning methods have made significant advances in becoming a useful tool for analyzing physical systems. A particularly active area in this theme has been ``physics-informed machine learning'' which focuses on using neural nets for numerically solving differential equations. In this work, we aim to advance the theory of measuring out-of-sample error while training DeepONets -- which is among the most versatile ways to solve P.D.E systems in one-shot. Firstly, for a class of DeepONets, we prove a bound on their Rademacher complexity which does not explicitly scale with the width of the nets involved. Secondly, we use this to show how the Huber loss can be chosen so that for these DeepONet classes generalization error bounds can be obtained that have no explicit dependence on the size of the nets. The effective capacity measure for DeepONets that we thus derive is also shown to correlate with the behavior of generalization error in experiments.

Published in Transactions on Machine Learning Research (11/2024)
Towards Size-Independent Generalization Bounds for Deep Operator Nets
Pulkit Gopalani"
gopalani@umich..ee edu
Deportment of Computer Science Engineering University of Michigan
Sayar Karmakar
Snurrkarmakkrruuffr.
Departmend Statistics University of Florido.
Dibyakanti Kumar
Department of Computer Science The University of Manchester
Anirbit Mukherjee
Department of Computer Science The University of Manchester
Reviewed on OpenReview: https: // openreview. forum? id= 200.aaaalliiileeeeaaaea000000000
Abstract
In recent times machine learning methods have made significant advances in becoming useful tool for analyzing physical systems. A particularly active area in this theme has been "physics-informed machine learning" which focuses on using neural nets for numerically solving differential equations. In this work, we aim to advance the theory of measuring out- of-sample error while training DeepONets which is among the most versatile ways to solve P.D.E systems in one-shot. Firstly, for class of DeepONets, we prove a bound Rademacher complexity which does not explicitly scale with the width of the nets involved. Secondly, use this to show how the Huber loss be chosen that for these DeepONet classes generalization error bounds can be obtained that have no explicit dependence on the size of the nets. The effective capacity measure for DeepONets that we thus derive is also shown to correlate with the behavior of generalization error in experiments.
their
1 Introduction
Deep learning has recently emerged as competitive way to solve partial differential equations (P.D.Es) numerically. We note that the idca of using nets to solve P.D.Es dates back many decades Lagaris et al. (1998) Broomhead & Lowe (1988). In recent times this idea has gained significant momentum and "AI for Science" Karniadakis et al. (2021) has emerged as distinctive direction of research. Some of the methods at play for solving P.D.Es neurally E et al. (2021) are the Physics Informed Neural Networks (PINNs) paradigm Raissi et al. (2019) Lawal et al. (2022), "Deep Ritz Method" (DRM) Yu et al. (2018), "Deep Galerkin Method" (DGM) Sirignano Spiliopoulos (2018) and many further variations that have been developed of these ideas, Kaiser al. (2021); Erichson et al. (2019); Wandel et al. (2021); Li al. (2022); Salvi et al. (2022).
These different data-driven methods of solving the P.D.Es can broadly classified into two kinds, (1) ones which train a single neural net. to solve a specific P.D.E.. and (2) operator methods which train multiple nets in tandem be able to solve in "one shot" family of differential equations, with a fixed differential
"This work done when the author the Department of Electrical Engineering, Indian Institute Technology Kanpur (IIT Kanpur), India.

---

Published in Transactions on Machine Learning Research (11/2024)
operator and different "source/ / forcing" functions. The operator net was initiated in Chen & Chen (1995) and its most popular deep net version, the DeepONet, was introduced in Lu et al. (2021). An unsupervised form of this idea was introduced in Wang et al. (2021). We note that the operator methods are particularly useful in the supervised setup when the underlying physics is not known - as is the setup in this work - and stale-of-the-art approaches of this type, can be seen in works like Raonić et al. (2023).
As an explicit example of using DeepONet in the supervised setup, consider solving a pendulum O.D.E., d(v.u) (v, -k sin(y)+ f(t)) R for different forcing functions f(t). Then using sample measurements of de
valid (f)v) tuples, a DeepONet setup can be trained only once, and then repeatedly be used for inference on new forcing functions f to estimate their corresponding solutions y. This approach is fundamentally unlike traditional numerical methods where one needs to run the optimization algorithm afresh for every new source function. In a recent study Lu et al. (2022), the authors showed how the DeepONet setup - that we focus on in this work- has significant advantages over other competing neural operators, like the FNO Li et al. (2020), in solving various differential equations of popular industrial use.
As a testament to the foundational nature of the idea, we note that over the last couple of years, several variants of DeepONets have been introduced, like, Tripura & Chakraborty (2023); Goswami et al. (2022a); Zhang et al. (2022); Hadorn (2022-03-16); Park et al. (2023); de Sousa Almcida et al. (2023); Tan & Chen (2022); Xu et al. (2023); Lin et al. (2023). Different. implementations of neural operators have been demon- strated to be useful for various scientific applications, like for predicting crack shapes in brittle materials Goswami et al. (2022b), for fluid flow in porous materials Choubineh et al. (2023), for simulating plasmas Gopakumar & Samaddar (2020), for seismic wave propagation Lehmann et al. (2023), for weather modeling Kurth et al. (2023) etc.
In light of this burgeoning number of applications, we posit that it becomes critical that the out-of-sample performance of neural operators be mathematically understood. Towards this goal, in this work, we prove that the generalization error of one of the most versatile forms of operator learning i.e DeepONets, can become independent of the number of training parameters. We note that this is a significant improvement over the best known generalization bounds for DeepONets, Theorem 5.3 in Lanthaler et al. (2022), which grow exponentially in the number of paramcters. We are able to obtain this insight via analyzing the Rademacher complexity of DeepONets. That we can recover here the usual neural net like generalization bound of the form a capacity measure is particularly interesting because in here the predictor, the DeepONet, is ysample-size
a non-hipschitz function, while nets are Lipschitz functions and that is critical to how the standard analysis is carried out for obtaining generalization error bounds.
Motivations for Understanding Rademacher Complexity We recall that typically there is vast lack of information about the distribution of the obtained models in any stochastic training setup. Corresponding to a stochastic training algorithm, say ALG, that is using a m- sized dataset S,, to search over a class of predictors H. there is scarce mathematical control possible over the distribution of its outpul random variable the random predictor thus obtained, say ALG((S.. H). Hence there is an almost insurmountable challenge in being able to control the quantity we would ideally like to bound, the out-of-sample risk of the obtained predictor i.e. (Lo ALG(SM, H)) where R denotes the expectation w.r.t the true data distribution, say D. of a loss E evaluated on the predictor ALG(S..,HH H)).
The fundamental idea that takes us beyond this conundrum is to relax our goals from aiming to control the above to aiming to control only the data averaged worst (over all possible predictors) "generalization gap" between the population and the empirical risk ie, ESm-D" [supben (Re h) R(L o h))] where corresponding to a choice of loss E, (n h) is the empirical estimate over the data sample Sm, of the risk of and R(E h) is the population risk of ht over the data distribution D. The importance of the statistical quantity, Rademacher complexity Bartlett & Mendelson (2001) of the function class i H i.e. B.......--D.. [E, [supheR m Σ,- c,6 hh h(z)))) (where 6; =1 are Bernoulli random variables) stems from the fact that it is what bounds this aforementioned mcasure of the generalization gap.
It to be noted that trying to control the above frees us from the specifics of any particular M.L. training algorithm being used (of which there is a myriad of heuristically good options) to find arginf R(e h). But on the other hand, by analyzing the Rademacher complexity we gain insight into how the choice of the
2

---

Published in Transactions on Machine Learning Research (11/2024)
hypothesis class H. the choice of the loss L, and the data distribution D interact to determine the ability to find Re7t R((oh) via empirical estimates. Suppose that there exists some constant C(H, D) capacity measure such that Es,,.D" [suPheR (fm h) -((e is upper-bounded by 2C(H.D) then one can //
conclude that while training with n 2 (2((H.D) 2 for the predictor obtained is at most e, for any arbitrarily small >> 0. This motivates one of the uses of Rademacher complexity i.c to provide such estimates on the number of samples required to achieve a target lest accuracy. In particular, in cases where this capacity does not scale with the number of parameters in the hypothesis class H. one may begin to explain why certain over parameeeettee models might be good spaces for the learning task. Such understanding becomes immensely helpful when H very complicated, as the focus here, that of H being made of DeepONets. In Section 2, we review some of the recent advances in the Rademacher analysis of neural networks.
samples the data-averaged worst-case generalization error
We crucially note (Theorem 4.13, Ma (2021)) that any condition m, H and DD that makes Rademacher complexity small is condition which when true it becomnes reasonable to expect that the empirical and the population risk could also be close for any predictor in the class H.
In this work, we will compute the Rademacher complexity of appropriate classes of DeepONets and use it to give the first-ofits-kind bound on their generalization error which does not explicitly scale with the number of parameters. Generalization bounds that do not scale with the size of the nets can be seen as a step towards explaining the success of overparameterized architoctures for that learning task. Further, our experiments will demonstrate that the complexity measures of DeepONets as found by our Rademacher analysis indeed correlate to the true generalization gap, over varying sizes of the training data.
1.1 Overview of Training DeepONets & Our Main Results
Following Lu et al. (2021), we refer to the schematic in Fig.1 below for the DeepONet architecture,
X3 xf(f)
Branch Network
DON(x: (f), x7)
XT
Trunk Network
Figure 1: A Sketch of the DeepONet ("DON") Architecture
In the above diagram, the Branch Network and the Trunk Network are neural nets with a common output dimension. (f( e Rd, the input to the branch net is an "encoded" version of function f i.e. a discretization of onto di-sized grid of "sensor points" in its input domain. Ør Re is the trunk input. If the activation is at all layers and the branch net and the trunk net's layers are named B 1,2 and Tg, k 1,2,..., fr respectively, then the above architecture implements the map,
Rd. x R (I 2T) + (FF))Nc((2(())xx1))
Bap (BB((((
Toz (oTT (TT.(((((T)) ...))) (1)
=fa(#s(())
=fr(w))
In above the number of rows of the matrix Byn and Tgp noed be the same for the inner-produs to be defined. For a concrete example of using the above archilecture, consider the lask of solving the pendulum f()) R2. For a fixed initial condition, here the training/tor data sets would be 3-tuples of the form, (# ZT. where y é is the angular position of the pendulum at time i = for the forcing function f. Typically y standard O.D.F. solver's approximate
O.D.E from the previous section, d(p.)) dt (v,-k (t,-k-ssiny)) sin(y)
3

---

Published in Transactions on Machine Learning Research (11/2024)
solution. Given m such training data samples, the l empirical loss would be,
1 m L - Σ - (e))) 2m
2m Σ ())(::)) - (D))))))))))((()(())))))))))
(2)
In above G is the solution operator for this O.D.E. that the O.D.E. solver can be imagined to be simulating. The rationale for this loss function above originates from the universal approximation properly of DeepONets which we have reviewed as Theorem C.1. Going beyond such approximation theorems, we state the results that we prove here about the risk bounds in this novel learning setup.
Theorem (Informal Statement of Theorem 4.1). Consider a class of DeepONets, with absolute value acti- vation, whose branch and trunk networks are both of depth (i.e. QB = qr in equation 1) and suppose that the squared norms of the inputs to them are bounded in expectation by Ma,B B and Ma,T respectively, Then the average Rademacher complexity, for training with samples of size m, is bounded by,
2 Cn,u-1
n-1 [ j-2
Mz. M2. MM.
(3)
m
where the constants Cn.n-1 and C-R,-Ns k k 2. .n....................... n-1 are defined so that the weight matrices the Branch Network and the Trunk Network all the DeepONets in the class satisfy the following bounds,
Σ
Σ B
B
| 1TT Cn.x-1.
sup Σ (v.w) 1= 3pm1
Σ
(vw))
)in.all | P Tn-ki C-k.-b
=k= = .
(4)
where in each sum, above v, are on the unit spheres of the same dimensionality the number of rows in Ba-k and Tn-k respectively and b- represent the number of rows in the weight matrices Bn-* and Ta-k respectively. (For any branch weight matrix say Bp: in above we have denoted its j-th row as Bp.j and similarly for the trunk.)
Towards making the above measures computationally more tractable, in Appendix we have shown that Vk 2...., one can choose an upperbound Č_K,-N in place of C_ where,
Č_ = X|| for X Rl-xxt= with X X Ba-k.sh I |TT--.jall
(5)
In Section 4.3, we undertake empirical study particular component of the generalization bound, represented as Cn.--1 m (1)5). in relation to the generalization gap to assess the correlation between
these factors.
Having proven the key theorem above, the following generalization bound with modified loss function for DeepONets follows via standard arguments about Rademacher complexity,
Theorem (Informal Statement of Theorem 4.2). Considering the same class of DeepONets as in Theo- x
for | < with 6 = (1))^1 as the loss function. 8 (la) - for |
rem and using the Huber loss (H,(()) =
The expectation over data of the supremum of the generalization error over the above class of DeepONets can then be bounded by,
Cn,n-1
[I
M Ma,
mm
1-2
where Cn,n-1 and C-j.-j are the constants defined in Theorem
To the best of our knowledge, there is no general principle which determines when the worst case general- ization gap of learning scenario scales like c for some "capacity" function C which is entirely determined ym by the data distribution and the norms of the parameters allowed in the predictor class. We recall that this
4

---

Published in Transactions on Machine Learning Research (11/2024)
is the familiar form of this bound from calculations done with neural nets, as reviewed in Appendix 2. We posit that the reappearance of this form of the bound for DeepONets could not have been taken for granted without the involved calculation as done here to prove the above theorem.
Secondly, we note that there is no explicit dependence on the widths of the DeepONet of the effective capacity mcasure of C obtained above, once the depth has been fixed. In line with how terminology was developed for standard neural nets, as reviewed in Section 2, we can say that thus we have obtained size- independent generalization bounds even for DeepONets and also while not making any assumptions on the target operator G an instance of which was seen in equation 2.
Lastly, we note that in Pestourie et al. (2023), the authors had recently demonstrated empirically the effectiveness of using Huber loss for training neural networks to solve P.D.Es. Albeit for different setup, that of solving P.D.E. systems, via the above theorem, we provide a theoretical justification for why Huber losses could be well suited for such tasks, and in here we are also able to derive a good value of 5 -while this hyperparameter choice in Pestourie et al. (2023) was not grounded in any theory.
In Appendix B we give an experimental study of how lowering the value of indeed helps lower both the generalization error well as the test error. Further, will give experimental evidence (Figures 2 and 6) that the capacity function derived above indeed has high correlation with the variations in the generalization error over different experiments at different training data size.
In light of the above, we note the following practical applicability and qualitative significance of our bounds on the Rademacher complexity and generalization error of DeepONets.
Choosing the Loss Function Theorem 4.2 suggests that training on the Huber loss function could lead to better generalization. We give a performance demonstration in Appendix A for Huber loss function at 8 2--1 for DeepONets using tanh activation (although our above theorem does not capture this choice of activation) and we note that it gives 2 orders of magnitude lower test error than for the same experiment using l2 loss.
1
Additionally, note that, DeepONets without biases and with positive homogeneous activation, are invariant to the scaling of the branch net's layers, Bx,and trunk net's layers, Tk, by any μa and A > 0 respectively Vk s.t H& (14 =1. This is a larger symmetry than for usual nets but our complexity measure mentioned above is also invariant under this combined scaling. This can be seen as being strongly suggestive of our result being a step in the right direction.
Explaining Overparameterization Since generalization error bound has explicit dependence the width or the depth (or the number of parameters), it constitutes step towards explaining the benefits of overparameteriization in this new setup.
2 Related Works
Over the past few years, many novel generalization bounds for standard neural nets have been established. Many of these works have computed bounds on Rademacher complexity for various classes of nets, to show how different norm combinations of the involved weight matrices affect generalization performance, Sellke (2024), Golowich et al. (2018), Bartlett et al. (2017), Neyshabur et al. (2015). For shallow neural nets such methods have also been useful in explaining the benefits of dropout Arora et al. (2021) and overparameterization Neyshabur et al. (2019). However, for state-of-the-ee neural nets, other approaches Dziugaite & Roy (2017), Arora et al. (2018), Neyshabur et al. (2018), Mukherjee (2021) have sometimes given tighter bounds while using bounding expressions that are computationally very expensive. A thorough discussion of the reach and limitations of these various bounds can be found in Nagarajan (2021).
In Kumar& & Mukherjee (2024), the authors investigate the behavior of the lz-distance between the true solution and the PINN surrogate, building on the work of Mishra & Molinaro (2023). While the study establishes an upper bound on the fg-distance, this result alone is insufficient to guarantee successful training. For training to be effective, is also necessary demonstrate that the empirical risk converges to the population risk, which can be ensured using Rademacher complexity-based bounds.
5

---

Published in Transactions on Machine Learning Research (11/2024)
2.1 Comparison to Lanthaler et al. (2022)
Firslly, we note that none of the theorems presented here depend the results in Lanthaler et al. (2022). The proof strategies are entirely independent. Secondly, to the best of our knowledge, Theorem 5.3 in Lanthaler et al. (2022) is the only existing result on generalization error bounds for DeepONets. However, their bound has an explicit dependence on the total mumber of paramcters (the parameter de there) in the DeepONet. Such a bound is not. expected to explain the benefits of overparameterivaioon, which is one of the key features of modern deep learning (Dar al., 2021; Yang et al., 2020).
We note that for usual implementations for DeepONets, where depths are typically small and the layers are wide, for class of DeepONets at any fixed value of our complexity measure i.e. Cn (1)=2C-3->)) C-g-z), generalization error bound based on our Rademacher complexity bound in Theorem 4.1 will be smaller than the one in Lanthaler et al. (2022) that scales with the total number of parameters.
Further, as pointed out in our second main result, Theorem 4.2, our Rademacher bound lends itself to an entircly size-independent generalization bound when the loss chosen as a certain Huber loss.
Notation Given any U R", denote as (U) the of all functions f where μ is the standard Lebesgue measure RR. And we denote C(U), the set of all real valued continuous functions on U. The unit sphere in R is denoted by Sk-1 == {= e R 6 | = 1] For any matrix
s.t Se (x) du (x)
|Avlz denotes its spectral norm. For any bounded set S R", (SS) denotes the uniform A, ||||| sup [v| distribution over that set S.
3 The Mathematical Setup
Firstly, we recall the definition of the general DeepONet architecture from Lu et al. (2021), which will our focus in this work.
Definition 1 (A DeepONet (Version 1)). We continue in the setup specified in equation 1. Further, let p be the common output dimension of qa depth "branch net" fa and the qr depth "trunk net" fr.
When required to emphasize some constraint on the weights of DeepONet, we will denote ReepONet(xz) xT) by (eg.ONeet.(B..x)) where w collectively stands for all the weight matrices in branch and trunk networks.
Corresponding to the DeepONets defined above we define the following width parameters for the different weight matrices.
Definition 2. (Width Paramcters for DeepONets)
For k ......93. QB - and f= 1 1,..., 'r -r- 1, define b- and te to be the number of rows in the weight matrices B and -e respectively.
In the setup of Definition 1, we define the functions f' and fr s.t given any m inputs to a DeepONet as, ((.B,,, i m,...,m} we have the following equalities,
fB( By (Bon-I(((EB.)) fr(aT,)) Tyr 02 (Tg--1 (IT.)).
(6)
Next, we define certain smoothness condition that we need for the activation functions used.
Assumption 1. Let Φp,0Q R R be the activation functions for the branch net and the trunk net such that 3L > 0 s.t for any two sets of functions valued in R, say P and Q (pp.g), (p), E P x Q and V(, y) e (P x (Q), the following inequality holds,
((((p())))o(((()))) op (x) (oq( (v)) -
(7)
Our main results work under the above assumption and hence in particular, our results apply to the absolute value map, I x + |a| R, being the activation function both the branch and the trunk not. In Appendix we will indicate why this is sufficient to also capture DeepONets with ReLU activations and being trained bounded data.

---

Published in Transactions on Machine Learning Research (11/2024)
Towards formalizing the setup of the loss functions and the training data for training DeepONets recall the definition of the Huber loss Huber (19641).
Definition 3 (Huber Loss). For some ó 2 0 the Huber loss is defined as
for |ar] s for [x] >
(1))(()
6 (|r1 ()
Corresponding to a choice of any univariate loss function (f), like Huber loss as above, we define the DeepONet training loss follows.
Definition 4 (A Loss Function for DeepONets). Given D Rd, compact set with boundary, we define function class of allowed forcing functions F c C(D). Further, consider DeepONet maps as given in Definition 1, mapping as DeepONet: : IRd x R + R and consider an instance of the training data given as,
(T)) E F Rd - SS....S...
Then, for an operator g : C(D) - (U) where (U) is the I based Sobolev space for some 8 > 0, the corresponding DeepONet loss function is given by,
C Σ (((()))TT))) DeepONe((zB,..
(8)
where some loss function. Here we assume a fixed grid of size di on which the function Si gets discretized, to get B, IR.1
As the DeepONet function varies, corresponding to it class of loss functions gets defined via the above equation. This class of DeepONets can be seen as being parameterized by the different choices of weights on the chosen architecture.
Also, it can be seen that the loss function in the experiment described in Section 1.1 was a special case of the loss in equation 8 for {(x) 2: the squared loss and if we assume that the numerical solver exactly solved the forced pendulum O.D.E.
x²
Next, we recall the definition a key statistical quantity, Rademacher complexity, from Bartlett & Mendelson (2001), which we focus on as our chosen way to measure generalization error for DeepONets.
Definition (Empirical and Average Rademacher complexily). For function class K, suppose being given m-sized data-set of points (æ; For +1 with equal probability, the corresponding empirical Rademacher complexity is given by Rm (1C) = E. [supeex EEE.& <x(x;)] If the elements of this data-set above are sampled from distribution P, then the average Rademacher complexity given by,
1,...,m} m] in the domain of the elements in K.
1 m G (m)))))))
Rm (IC) Exg
P
E sup keR m el
Σ
The crux of our mathematical analysis will be to uncover recursive structure between the Rademacher complexity of DeepONets and certain DeepONets of one depth lower in the branch and the trunk network, and which would always have one-dimensional outputs for the branch and the trunk and which share weights with the original DeepONet in the corresponding layers.
4 Results
Our central result about Rademacher complexity of DeepONets will be stated in Theorem 4.1 Section 4.1 and consequent to that our result about bounding the generalization error of DecpONets will be stated in Theorem 4.2 of Section 4.2. In Section 4.3, we present an empirical study of our bound on Rademacher complexity for DeepONet trained to solve the Burgers' P.D.F.

---

Published in Transactions on Machine Learning Research (11/2024)
4.1 First Main Result Rademacher Complexity of DeepONets
In the result below we will see how for a certain class of activations, complexity of DeepONets which does not scale with the width of the nets for the branch and the trunk nets being of arbitrary equal depths.
get bound the Rademacher
Theorem 4.1 (Rademacher Complexity of Special Symmetric DeepONets). We consider a special case of DeepONets as given in Definition 1, with (a) QB qr = n, and (b) σ1, 02 satisfies Assumption for some constant L> 0, and they are positively homogeneous. Further, let b-; and t-; be the number of rows of the weight matrices Bn-s and Tn- respectively, as in Definition 2, for i and recall from Definition 1 that is the number of rows of B, and Tn Then given a class of DeepONet maps as above, we define the following n- 1 constants, Cn,x-1 0 and C-k,-k 0, k =2....., n DeepONet maps in the class satisfy the following bounds,
such that for S5 == 5h x 5t-x11 all the
b
E (B..,TT.)) f-1
Σ
Ba-1,A 1Ta-1
Cn,n-1
1
sup (u,w) Sk 52-1
Σ Σ
(vw)) )i
Bn-k T Taal S SC-R,-ks Vk = 2,....n
Then given training data as in Definition 4, the empirical Rademacher complexity of this class is bounded as,
Cn (2L)
m-1 C
Rm S
Σ I 12
m
j-2
Further assuming that the input distribution over F x Rd induces marginals distributions s.t E | [
2 B. E [ M2,r> we have the average Rademacher complexity of the same class bounded as,
Rmm S (2L) Cn.n-1
n-1 [ j-2
My. Ma,
mm
The proof for the above is outlined in Section 6. For (1(x) 02(x) = =|a| i.c. for DeepONets with absolute value activations, have T 1 and hence the subsequent simplification also happens in the result in Theorem 4.1.
Now we have all the necessary setup to state the final result on generalization bound for DeepONets.
4.2
Second Main Result: Independeend Generalization Error Bound for DeepONets Trained via Huber Loss Unbounded Data
Theorem 4.2 (Generalization Error Bound for DeepONet). We continue with the setup of the DeepONets as in Theorem 4.1 but with 01 (I) (2(2) = |z]. For an operator G C(D) (U) where H*(U) is the I²-based Sobolcv space for some s 0, a class of forcing functions F Fcc C(D) and W a set possible weights w for DeepONets (denoted as DeepONet..)) we define the function class as,
H ((fxx) (x))(x)) DeepONet, x)))))xx) |w e W, (f.xf))) e F Rd:
(L)(xx) (xB()), R
Then for Huber loss (- as in Definition 3 with 8= (2) n-1 where is the depth of the branch and the trunk nets have the following generalization bound,
1 m
B ((hh))))) (il...m)
sup weW
Σ
(10)
2C 2Cn.-1
S
C
Mr,T
Vm
J=2
where Cr and C-j-j are the constants defined in Theorem 4.1, and D distribution over F x R.dy
8

---

Published in Transactions on Machine Learning Research (11/2024)
The proof for the above theorem is given in Section 6.
4.3 An Experimental Exploration of the Proven Rademacher Complexity Bound for DeepONets
In the following sections, we present an experimental study on the complexity bounds for the Burgers' and 2D-Heat PDEs in Sections 4.3.1 and 4.3.2, respectively.
4.3.1 Burgers' PDE
Here, we consider the following specification the Burgers' P.D.E. with periodic boundary conditions,
Ös
Os
0²s
Ōt +
k Ox.²
(=,,))(=-,,)([00
dr
s(x- x,t) = s(x n. t), s ((.0)) u(x)
where, x denotes the fluid viscosity, for our experiments choose 0.01, and u(x) is 2n-periodic initial condition with zero mean i.e. u(=)dr 0 and Te R.
Hence it follows that the solution operator of Definition 4 corresponding to the above maps the initial condition u to the solution to the Burgers' P.D.E., Hence, we will approximate the implicit solution operator G with a DeepONet g which for this case would be a map as follows,
I x 2
> IR (v. v) ((v,)) == Σ NB)(w) NT.
where NB and NT are any two neural nets mapping R R' and R² + R respectively the Branch net and the Trunk net. above, q. the common output dimension of the branch and the trunk net is a hyperparameter for the experiment.
The data required to set up the training for the above map G can be summarized as a 3-tuple of tensors, (Beraining: Vtraining: Straining) of dimensions and descriptions as given below.
Utraining E R Neraining are evaluations of the Neraining number of input functions at the m sensors. Vi
We denote the 7t row of the above as, (.)).....,
are the Piraining uniformly sampled collocation points in I correspond- ing to cach of the Neraining number of input functions. And Vi {1...., Neraining), denote the sampled points as {(ZIJ,
Straining Ig-Neranng
uniform ([-T, [0, Piratming
Straining R.-Neratting Peratning are the values of the numerical solution of the P.D.E. at the respective collocation points. And the ti.j)th-enty of this would be denoted as si( 4zj.
In terms of the above training data, the corresponding training loss (Definition 4) is.
1
Neraining Paraininx
Lrraining (0) Nuratning Persining
Σ
Σ F=1
Σ [NeA(((E))) (((m)))),,,,,mm)) N NI
(11)
whereby in our experiments is either chosen as the l loss or the Huber loss and O denotes the vector of all trainable parameters over both the nets. Similarly as above the test loss L would be defined corresponding to a random sampling of Nust input functions and Ptest points in the domain of the P.D.E. where the truc solution is known corresponding each of these lest input functions. For our experiments, we fixed Ntraineng 400 and Piraining is chosen to be the following set of values (300, 400, 500, 750, 1000, 2000).
Further, for our experiments, we sampled the input functions u, by sampling functions of the form, (x)x) EM G. sin((n (n+ 1)x), where the coefficients were sampled as, G. ~N,0 A0)). with A and N being

---

Published in Transactions on Machine Learning Research (11/2024)
arbitrarily chosen constants. This way of sampling ensures that the functions chosen all 27-periodicc and will have zero mean in the interval [-x, T] as required in our setup.
In this section, we demonstrate our Rademacher complexity bound by training DeepONets on the above loss function and measuring the generalization gap of the trained DeepONet obtained and computing for this trained DeepONet an upperbound on the essential part of our Rademacher bound (i.e. Ca.. vm (IJ=2 11--2----7 C-)-s)
where Č=R,k is as defined equation 5). Then we vary the number of training data (i.e. Neraining x P Ptraining) and show that the two numbers computed above are significantly correlated as this training hyperparameter is varied. We use branch and trunk nets cach of which are of depth 3 and width 100.
The correlation plots shown in Figures 2a and 2b, correspond to training the Huber loss function, with two different values of 8 - and the former corresponds to the special value of 0= 4 where the size-independent generalization bound in Theorem 4.2 clicks for this setup.
In Figure of Appendix, we repeat the same experiments but for the & loss function and show that the required correlation persists even with this loss which is outside the scope of Theorem 4.2.
4.6-
Correlation Coefficient - 0.857
Correaaiio Confficier - 0.791
5.4
0e+05
05
52-
4.4
gl. 6e 05
4.3
(Trin-Erme
(Taan
0.8
0e-05
4.1
05
4.6
05
05-05
A De 404
4.0-
3.0
35
40
4.5
5.0
5.5
6.0
6.5
7.0
3
Generalization Enror Bound
Generalization Error Bound
x12
(a) 6=0.25, correlation 0.857
(b) 8=0.4, correlation 0.791
Figure 2: The above plot shows the behaviour of the measured generalization error with respect to Ca.c 2. for training DeepONets, to solve the Burgers' PDE, with empirical loss as given in equation 11, specialized to the Huber loss (Definition 3) for the stated values of and for the branch and the trunk nets being of depth 3. Each point is labelled by the number of training data used in that experiment.
ym
4.3.2 Heat PDE
Here, we consider the following specification the 2D-Heat P.D.E. (Altaisan, 2024),
Ou = u n² pu pu, Or2 0y²² ) (x,y) D, te [0,
a1
y, 0, (x,y) e OD, u(I,y, 0) = f(, (x
where, D is the 2-dimensional square [0, 1] x [0, 1], x > denotes the thermal diffusivity, for our experiments we choose x 1, and fax y) 2r-poriodic initial condition and T R.
In this case, the solution operator of Definition 4 corresponding to the above maps the initial condition f to the solution to the Heat P.D.E. Hence, we will approximate the implicit solution operator G with a DeepONet Č - which for this case would be a map as follows,
IR m x IR + IR, (v. y) - ((v : (B (N)(4)
where NB and N1 are any two neural nets mapping R R&. and R3- R respectively - the Branch net and the Trunk net. In above, g. the common output dimension of the branch and the trunk net is a hyperparameter for the experiment. We have chosen q = 128 and 100 for our experiments.
10

---

Published in Transactions on Machine Learning Research (11/2024)
The data required to set up the training for the above map G can be summarized a 3 B-tuple of tensors, (Jtraining: Ptraining, Wtraining) of dimensions and descriptions given below.
Straining E IR Veraining are evaluations of the Niraining number of input functions at the sensors. We denote the 8th Iow of the above as, [f( f1), (2), .2.,..., fi ym)] Vi
Puraiuing é IR IB-arratiingg Powsningxx are Puraining uniformly sampled collocation points in R corresponding to each of the Niraining number of input functions. And Vi (1,.... Ntraining), we denote the sampled uniform ([O, 1] x [0.1] x [O, Pieaininx
ii, points as {(11j (L.. .uraii,u
Vtraining E R N Prestning are the values of the numerical solution of the P.D.E. at the respective collocation points. And the (i.j)th th entrr of this would be denoted as us ui( ( 41.).
In terms of the above training data, the corresponding training loss (Definition 4) is,
Neratinng Peraining
Creiining (0)
N Neraining Ptraining
Σ
f. fa(Fm: ( (12)
whereby in our experiments & is either chosen as the &g loss or the Huber loss and denotes the vector of all trainable parameters over both the nets. Similarly as above the test loss L would be defined corresponding to a random sampling of Niet imput functions and Ptest points in the domain of the P.D.E. where the true solution is known corresponding each of these test input functions. For our experiments, we fixed Niraining to 64 and P Ptraining is chosen to be the following set of values (2 (29,36, ,...,,7777 78).
Further, for our experiments, we sampled the input functions by sampling functions of the form, M(x,y) = E Cmsin( m===) )}
mm [> dusin( po } where the coefficients were sampled as, Cm, dn ~ U[1000 200]), with N is fixed to 2. This way of sampling ensures that the functions chosen are all 2 2n-periodic and know the exact analytical solution for these initial conditions to be u(2,
EM Σ Amn sin maz leads to more controlled measure of the test error than in the previous setup.
sin pKy
where N = Amn = cm dn and zo. 30 = 1. This exactness
Zo
Here, we demonstrate Rademacher complexity bound by training DeepONets the above loss function and mcasuring the generalization gap of the trained DeepONet obtained and computing for this trained DeepONet an upper-bound on the essential part of our Rademacher bound (i.e.
vm (113-2 C-g.-)) where C_ is as defined in equation 5). Then we vary the number of training data (i.e. Ntraining x Persining) and show that the two numbers computed above are significantly correlated as this training hyperparameter is variod. We use branch and trunk nets each of which are of depth and width 128.
Cn.n
The correlation plots shown in Figures 3a and 3b, correspond to training on the Huber loss function, with two different values of 6 and the former corresponds to the special value of 8 where the size-independent generalization bound in Theorem 4.2 clicks for this setup.
Going beyond the ambit of the Rademacher bound proven, in Figure 7 in Appendix M, we present the plots demonstrating the solution of the Heat P.D.E. during training with lz and Huber loss, while allowing for ReLU activation and biases in the layers. It can be seen that Huber loss demonstrates noticeably better performance compared to & loss.
5 Discussion
The most immodiate research direction that being suggested by our generalization error bound to explore if the advantages shown here for Huber loss can make DeepONets competitive with CNO (Raonić et al., 2023).
11

---

Published in Transactions on Machine Learning Research (11/2024)
X101
x10
4 c108
Comelation Confficient 0.669
Corriationn Coofficient 0.717
[Paindrr Ted Tea--trori
(Tain Error Tesh Emori
4-
9e-06
fe+05
--66-00
7.5e+06
d.ee004
2 3e-00
g 0.00606
de Gemeralization
100
x10'
Generalization Ento
x12
(a) d=0.25, correlation = 0.669
(b) 6=0.4, correlation = 0.717
Figure 3: The above plot shows the behaviour the measured generalization error with respect to Ca.aC-a-22 for training DeepONets, to solve the 2D-Heat PDE, with empirical loss as given in equation 12, specialized to the Huber loss (Definition 3) for the stated values of and for the branch and the trunk nets being of depth 3. Each point is labelled by the number of training data used in that experiment.
vmn
Further, in Wang et al. (2021), unsupervised variation of the loss function of DeepONet shown to give better performance. In Goswami et al. (2022b), authors employ a variational framework for solv- ing differential equations using DeepONets, through a novel loss function. Understanding preciscly when these variations the loss function used give advantages over the basic DeepONet loss is yet another in- teresting direction for future research and the path towards such goals could be to explore if the met demonstrated here for computation of generalization bounds can be used for these novel setups too.
Also, it would be fruitful to understand how the generalization bounds obtained in this work can modified cater to the variations of the DeepONet architecture (Kontolati et al., 2023), (Bonev et al., 2023) Lu et al. (2022) that are getting deployed.
In the context of understanding the generalization error of Siamese networks, the authors in Dahiya et al. (2021) dealt with a certain product of neural outputs structure (where the nets share weights). In their analysis, the authors bound the Rademacher complexity via covering numbers. Since we try to directly bound the Rademacher complexity for DeepONets, would be interesting investigate if our methods be adapted to improve such results about Siamese nets.
Lastly, we note that the existing bounds on the Rademacher complexity of nets have typically been proven by making ingenious use of the algebraic identities satisfied by Rademacher complexity (Theorem 12 in Bartlett & Mendelson (2001)). But to the best of our knowledge, we are not aware of any general result on how Rademacher complexity of a product of function spaces can be written in terms of individual Rademacher complexitics. We posit that such a mathematical development, if achieved in gencrality, can be a significant advance affecting various fields.
6 Methods
In this section, we will outline the proofs of the main results, Theorems 4.1 and 4.2. We would like to emphasize that the subsequent propositions 6.2 and 6.3 hold in more generality than Theorem 4.1, because they do not need the branch and the trunk nets to be of equal depth.
Outline of the Proof Techniques Derivation of the first main result, Theorem 4.1, involves 3 key
steps: (a) formulating a variation of the standard Talagrand contraction (Lemma 6.1) (b) using this to bound the Rademacher complexity of class of DecpONets with certain activations (c.g. absolute value) by the Rademacher complexity for class of DeepONets having one less depth and 1-dimensional outputs, for both the branch and the trunk (Lemma 6.2) and lastly (c) uncovering a recursive structure for the
12

---

Published in Transactions on Machine Learning Research (11/2024)
Rademacher complexity across depths between special DeepONet classes having 1- 1-dimensional outputs for both the branch and the trunk. (Lemma 6.3).
Lemna 6.2 removes the last 4 matrices from the DeepONet (2 each from the branch and the trunk) leading to one-dimensional output branch and trunk nets. Lemma 6.3 removes 2 matrices (1 each from branch and trunk) - by an entirely different argument than needed in the former. Lemma 6.2 is invoked only once at the beginning, while Lemma 6.3 is repeatedly used for each remaining layer of the DeepONet.
We note that both our "peeling" lemmas above are structurally very different from the one in Golowich et al. (2018) - where the last layer of standard net gets poeled in every step.
Deriving the second main result, Theorem 4.2, involves the following key steps: (a) establishing a relationship between the Rademacher complexity of the loss class and that of the DeepONet when the loss function is Lipschitz (Proposition G.6) and (b) using this to upper bound the expectation over data of the supremum of generalization error in terms of the Rademacher complexity of the DeepONet.
Towards proving Theorem 4.1, we need the following lemmn which can be seen variation of the standard Talagrand contraction lemma,
Lemma 6.1. Let Φp, Qq R - R be two functions such that Assumption 1 holds and let P and Q be any two sets of real-valued functions. Then given any two sets of points (æ; = 1,...,m. m, and {y: i l,..,.,, in the domains of the functions in P and Q respectively, we have the following inequality of Rademacher complexities where both the sides are being evaluated on this same set of points,
R. (ōp o ΦQ o Q) Lm (P-Q) Q).
The above lemma has been proven in Appendix H.1.
Towards stating Propositions 6.2 and 6.3, we will need define certain classes of Sub-DeepONets s.t these sub-DeepONets would be one depth lower in the branch and trunk network, and would always have dimensional outputs for both the branch and trunk nets and would share weights below that with the corresponding layers in the original DeepONet.
Definition 6. (Classes of sub-DeepONets) Let Wrest be set of allowed matrices for nets f3 and fr as in Definition 2. Now, given a constant Cyg.er Cgg-rr-0=1 -ar---11.0-- > O we define the following set of 4- 4-tuples of outermost layer of matrices in the DeepONet as,
W(C 00 = (BqB B T Tmm-1)
b-1 f- B Bgg
Σ
1
Σ (B
T
(13)
kg=l
ki
Secondly, corresponding to the above we define the following class of DeepONets,
Wreat)
DeepONet (as in Definition 1) w (Wr W Wrest))
(14)
Lastly, we define the following class of DeepONets,
DeresN)t(Wress)) ( + v fx(())))))))))))o))) w f((x)) (eer) eR
the set of allowed matrices for nets fn and ftr are in Wrest (u, E 2h2--1 x st-z-1
(15)
13

---

Published in Transactions on Machine Learning Research (11/2024)
Proposition 6.2 (Removal of the Last 4 Matrices of DeepONet). We contimue to be in the setup of Definition 2 and assuming that 01,02 satisfies Assumption for some T. > 0, and are positively homogeneous. Then given the definitions of DeepONet(WC (r. Wrest) and DeepONet(Wrrss in Definition 6, we have the following upperbound on empirical Rademacher complexity of a DeepONet, (here S Sh-+-1 St==11
Rm Do 11, Wrest)
Σ e fEeEE) f(((,,)) w (F((T.)) i-1
22 -1 Bg
sup
(v.w))& Sa Wrest
2L 22L.CB...e Cqp.9r.98-1err1
Rm (DeepONet(Wiess))
Note that both sides of the above are computed for the same data ((BB,), Rd x IRd2 i 11=1..... 1=....,,m) m). The proof of the above proposition is given in Appendix E.
Referring to the definitions of the DeepONet classes on the L.H.S. and the R.H.S. of the above, as given in oquations 14 and 15 respectively, we see that the above lemma upperbounds the Rademacher complexity of a DeepONet class (whose individual nets can have multi-dimensional outputs) by the Rademacher complexity of simpler DeepONet class. The DeepONet class in the R.H.S. is simpler because the last layer of each of the individual nets therein is constrained to be a unit vector of appropriate dimensions (and thus the individual nets here always of 1 dimensional output) and whose both branch and the trunk are shorter in depth by 1 activation and 1 linear transform than in the T.H.S.
Proposition 6.3 (Peeling for DeepONets). We continue in the setup of Proposition 6.2 and define the functions f/ and f s.t we have the following equalities, f/B = σ1 (Bgn-2f%) and f/ 02 2f Further, given a constant C-2,-2 > O, we define W/est as the union of (a) the set weights that are allowed in the Wrest sot for the matrices Byn-3; Bgn-4...., B₁ and Typ-3, Tgr-4..... T2 and (b) the subset of the weights for Bgg-2 and Tør-2 that allowed by Wrest which also additionally satisfy the constraint, (here Sy S6-4-1 1 11-a-1)
sup (w.w) eS2
Σ (ow))iaaalBl 2.l 2 C_2.-2
(16)
22-11111 1==1
Then we get the following inequality between Rademacher complexities,
Σ e,o fB((D,)) w f(((TT)))
E,
sup E Wrast
Σ a( fH(,)))) H((r)))
22LC2,-2B
sup (u,w)) W
where b.R, L. as in Definition 2.
Proof of the above proposition is given in Appendix F, And now we have stated all the intermediate results nceded to prove our bounds on the Rademacher complexity of a cortain DecpONet class.
Proof of Theorem 4.1
Proof. For each & = 2.3...., 1, we define product of unit-spheres, S₁ 5 x sh-r-1 and let Sn = gih-1, x Sd2-1 Now we define,
b
W (B,, Bm- Tn. TT-1)
Σ
B An
[ 1a2l
Σ
B m.j (B..TTII
$ SCnnal
14

---

Published in Transactions on Machine Learning Research (11/2024)
Next, for each == ..,,..., 1, we define
b
W rest
Be Tis
sup Σ
Σ (vw )o.sal - BB
T C-ke-b
k=1
(v,w) di 52
Thus we have,
m (Bnor (Bn-Ifg(ZR,))) Taoz T (Ta- (TT
m
Be
sup W res1)
(Wil cctt
Then we can invoke Lemma 6.2 on the above to get,
mRn Ee
sup (Wi W
c (B (Bno (Bn-1fH(EB;)) Tho2 (Tn-1 (I--
m sup wa 02
$ 2LC m
Now, using Lemma 6.3 repeatedly on the R.H.S above, and defining in a natural fashion the subsequent branch and trunk sub-networks as (.) we have,
mRm $ ( n Cn, C-2,2B
sup
Woot (2)
:
i [
$ ((2L)
E.
sup W
( pl) (ans)) ( (xTW))
Σ
1-2
1=1
(v1,
$ (2 Cn.n-1
C
E.
sup (2)
j-2
Using Lemma 6.4, the final bound on the empirical Rademacher complexity becomes,
Rm S (2L)"=I Co,n-1
n-1 C 2--
m Σ i-1
1
Invoking the assumption that the input is bounded s.t E Rademacher complexity can be bounded
|
s M2 and E || || s M2 the average
Ry s (2T.)
I
Cn.n-1
n-1 Ma.T
C
mm
1=2
Lemma 6.4.
E,
sup Σ [e(o(=r))0baan))
Σ 1 2
(v) c.. 1-1
where c; = Fn, and S, = 8 x 5d2-1 and each 6, #1 uniformly.
Proof of the above lemma is given Appendix In Appendix we have setup a general framework for using Rademacher bounds as proven in the above theorem prove generalization error bounds for DeepONets. And will now invoke the result there to prove our main theorem about generalization error of DeepONets.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Proof of Theorem 4.2
Proof. We recall from Theorem 4.1 the following bound on Rademacher complexity for the given OeepONet class,
(2L) Cn
n- C- 5=2
Rm S
Mr, Mr,BMrr
m
Combining the above with Proposition G.6 we have,
E
Σ (h(fi,vrr))) -B -BBr ((()))(((((()))))))))))))))))))) ((ierr D (i=l...m)
sup NEE
n-1 C 7-2
2R.
Mn MaST
Vm
Note that Huber-loss LH, 8-Lipshitz and further setting 8 (2)=-1 and rewriting the supremum over all h H as supremum over all W can write the above inequality
1 m
Lastly, considering that the activation functions are (uu. obtained the claimed bound,
((1i00)))) SD [==1...
sup weW
i=1
2L) s 2)n=2) m
n-1 [ MATT
= |x| which makes L = in Lemma 6.1 we
1 Σ
E (1) jil...mm
sup weW
enss( - (Hn.)((())
m 1-1
2Cm,n-1
I 1-2
C-j-d
Mr. Me,T
mm
References
Mohammad Altaisan. Approximating solutions of 2d heat equation using deeponets & error estimates of deeponets for hölder contimuous operators. 2024.
Raman Arora, Peter Bartlett, Poorya Mianjy, and Nathan Srebro. Dropout: Explicit forms and capacity control. In Marina. Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine
Learning Research, pp. 351-361. PMLR, 2021. URL http://proceedinngs mlr (ress//v139/aroraa22 html.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via compression approach. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML, 2018, Stockholmemässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 254 263. PMLR, 2018. URL http::/proceedingg wtwp:/ceeedinnnmm......mmmmmm mlr. www..mt.e..../////mmmmm arora18b. html.
Peter Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. The Journal of Machine Learning Research, 3:224-24 240, 06 2001. doi: 10.1007/3-540-4445 15.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spertrally-normalized margin bounds for neu- ral wetworks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Sys- tems 30: Annual Conference on Newral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6240 6249, 2017. URL https://ppoont http//proeeenn...////// wetpeeeddinnnnnnnns..e... htt.:/gwww.....eee.p//////////// wwtpe/pppp///////aaeae foadddAbsractt.om html.
Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar. Spherical Fourier neural operators: Learning stable dynamics on the sphere. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 2806 2823. PMLR, 23 29 Jul 2023. URL https: / //proceedings. mlr. prrss//202/bonev222 html.
David Broomhead and David Lowe. Radial basis functions, multi-variable functional interpolation and adap- tive networks. RO ROYAL SIGNALS AND RADAR ESTABLISHMENT MALVERN (UNITED KINGDOM), RSRE-MEMO-4145 03 1988.
Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transaclions on Neural Networks, 6(4):911-99) 917, 1995. doi: 00.1109/72.399253
Abouzar Choubineh, Jie Chen, David A Wood, Frans Coenen, and Fei Ma. Fourier neural operator for fluid flow in small-shape 2d simulated porous media dataset. Algorithmss, 16(1):24, 2023.
Kunal Dahiya, Ananye Agarwal, Deepak Saini, Gururaj K, Jian Jiao, Amit Singh, Sumeet Agarwal, Pu- rushottam Kar, and Manik Varma. Siamesexml: Siamese networks meet extreme classifiers with 100m labels. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2330 2340. PMLR, 18 24 Jul 2021. URL ftaps://proceedingg mlr. www....re..........e..... press/v139/dee /aai///aahaaaaaaaa html.
Yehuda Dar, Vidya Muthukumar, and Richard G. Baraniuk. A farewell to the bias-variance tradeoff? an overview of the theory of overparameterized machine learning. CoRR, abs/2109.02355, 2021. URL https://arxiv: httpss//aaxxiv.orr/////////// org/abs/2109 02355.
Joao Lucas de Sousa Almeida, Pedro Rocha, Allan Carvalho, and Alberto Costa Nogueira Junior. A coupled variational ancoder-dccooee-ddccoonttt surrogate model for the rayleigh-bénard convection problem. In
AAAI Conference on Artificial Intelligence, 2023.
Beichuan Deng, Yeonjong Shin, Lu Lu, Zhongqiang Zhang, and George Em Karniadakis. Approximation rates of deeponets for learning operators arising from advection diffusion equatinns. Newral Networks, 153:411 426 2022. ISSN 0893-6080. doi: https////do..org100011///// www.et.2022..06.00.... 2022.066.19. URL https: nwwp:/.jjine....mmemeeem //www. www.ssciencedirect. wttp:///ttee//rrraaaa///////////
Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Gal Elidan, Kristian Kersting, and Alexander T. Ihler (eds.). Proceedings of the Thirty- Third Conference on Uncertainty in Artificial Intelligence, UAT 2017, Sydney, Australia, August 11-15, 2017. AUAT Press, 2017. URL http: pdf.
//auai.
Weinan E, Jiequn Han, and Arnulf Jentzen. Algorithms for solving high dimensional PDEs: from nonlinear monte carlo to machine learning. Nonlinearity, 35(1):278-311 dec 2021. doi: 10.1088/ (0008/1161--0 1133166444/a33/11 URL atttps///dol /tttpp://doi.orr/110.. (098//2F1331-654422228
N Benjamin Erichson, Michael Muehlebach, and Michael W Mahoney. Physics-informed auloencoders for Iyapumov-stable fluid flow prediction. arXie preprint arXiv:1905. 10866, 2019.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural net- works. In Conference On Learning Theory, pp. 297-299. PMI.R, 2018
17

---

Published in Transactions on Machine Learning Research (11/2024)
Vignesh Gopakumar and D Samaddar. Image mapping the temporal evolution of edge characteristics in tokamaks using neural networks. Machine Learning: Science and Technology, 1(1)001000, 2020.
Somdatta Goswami, Katiana Kontolati, Michael D Shields, and George Em Karniadakis. Deep transfer operator learning for partial differential equations under conditional shift. Nature Machine Intelligence, 4
(12):1155-1164, 2022a.
Somdatta Goswami, Minglang Yin, Yue Yu, and George Em Karniadakis. A physics-informed variational DeepONet for predicting crack path in quasi-brittle materials. Computer Methods in Applied Mechanics and Engineering, 391:114587, mar 2022b. doi: 2..101//.eema.2222.1455555 URL http////oo.or/...../ 1016/2Fj. cma 2022. 114587.
Patrik Simon Hadorn. Shift-deepont: Extending deep operator networks for discontinuous output functions. Master thesis, ETH Zurich, Zurich, 2022-03-16.
Peter J. Huber. Robust Estimation of a Location Parameter. The Annals of Mothemalical Stalislics, 35(1): http/aamms111177001
73 101, 1964. doi:
URL https://doi. http://ii.orr//////// 1211/2oms/1177785333
Eurika Kaiser, J Nathan Kutz, and Steven Brunton. Data-driven discovery of koopman eigenfunctions for control. Machine Learning: Science and Technology, 2(3):035023, 2021.
George Em Karniadakis, Toannis G. Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physiie-informed machine learning. Nature Reviews Physics, 3(6):422 440, Jun 2021. doi: 10.1038/ 542254--21-00331455 URL https://boi.orr/100 112///100000//422-00000000000000
Katiana Kontolati, Somdatta Goswami, George Em Karniadakis, and Michael Shields. Learning latent spaces improves the predictive accuracy of deep neural operators. arXiv preprint arXiv:2304.. 07599, 2023.
Dibyakanti Kumar and Anirbit Mukherjoc. Invostigating the ability of pinns to solve burgers' pdc near finite-time blowup. Machine Learning: Science and Technology, 5(2):025063, jun 2024. doi: 10.1088/ 2632- 212221555/dddcd..
Thorsten Kurth, Shashank Subramanian, Peter Harrington, Jaideep Pathak, Morteza Mardani, David Hall, Andrea Miele, Karthik Kashinath, and Anima Anandkumar. Fourcastnet: Accelerating global high- resolution weather forecasting using adaptive fourier neural operators. In Proceedings of the Platform for Advanced Scientific Compuling Conference, pp. 1 11, 2023.
L.E. Lagaris, A. Likas, and D.I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neurol Networks, 9(5):987-1000, 1998. doi: 10.1109/72.712178. URL wttp://0...1////7..11117177
Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis. Error estimates for DeepONets: deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6(1), 03 2022. ISSN 2398-4945. doi: 10.1093 10.1193/imatrm//tee /it/a//mmttmaco0...... URL posttp:://oi.ooo 1098//mattmm/naac00.. tnac001.
Zaharaddeen Lawal, Hayati Yassin, Daphne Tock Ching Lai, and Azam Idris. Physics-informed neural network (pinn) evolution and beyond: A systematic literature review and bibliometric analysis. Big Dala and Cognitive Computing, 11 2022. doi:
Fanny Lehmann, Filippo Gatti, Michaël Bertin, and Didier Clouteau. Fourier neural operator surrogate model to predict 3d scismic waves propagation. arXiv preprint Stl:2304.10244 2023.
Zongyi Lí, Nikola Borislavov Kovachki, Kamyar Aviwadennshelli, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar, al. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2020.
Zongyi Li, Miguel Liu-Schiaffini, Nikola Kovachki, Burigede Liu, Kamyar Avizzadenesheli, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Learning dissipative dynamics in chaotic systems. arXiv preprint ar Xic:21066 06898, 2022.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Guang Lin, Christian Moya, and Zecheng Zhang. B-deeponet: An enhanced bayesian deeponet for solving noisy parametric pdes using accelerated replica exchange sgld. Journal of Computational Physics, 473: 111713, 2023.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218 229,, mar 2021. 1038/2Fs42256 000002222660000002-----
doi: 101108//42255-22--03325, URL https://doi.org///
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393:114778, 2022. ISSN 0045-7825. doi: http:://dh..org//10001/////
science/artiillepiif /llemc//arrttccee///////82222210 210.:/0000000.....22211111111771 URL http:://ww.sctenceddaee
com/
Tengyu Ma. Lecture notes for machine learning theory. STATS214 / CS229M: Machine Learning Theory, 2021. URL nttp://web... stanford wdw/ httu/class/stats214///
Siddhartha Mishra and Roberto Molinaro. Estimates on the generalization error of physics-informed neural networks for approximating pdes. IMA Journal of Numerical Analysis, 43(1):1 2023.
Anirbit Mukherjee. A study of the mathematics of deep learning. CoRR, abs/ abs/2104.1100 12044.1433,, 2021. URL https://arxivvorr//b////// 14033.
Anirbit Mukherjee and Amartya Roy. Size lowerbounds for deep operator networks. Transactions on Machine Learning Research, 2024.
Vaishnavh Nagarajan. Explaining generalization in deep learning: progress and fundamental limits. CoRR, abs/2110.08922, 2021. URL https:://ar//tt wwwpa/wwwww.ra/////a/eaaa// 08922.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Peter Grünwald, Elad Hazan, and Satyen Kale (eds.), Proceedings of The 28th Conference on Learning Theory, COLT 2015, Paris, France, July 3-6, 2015, volume 40 of JMLR Workshop and Conference Proceed- ings, pp. 1376 1401. org,. 2015. URL http:/ http::/prroceeinng..oo wtwp:oceeedinn.mm.......m.. htt...ppeee///////eeaaabbbt.e 1.tmmmm
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach spectrally- normalized margin bounds for neural networks. In 6th International Conference on Learning Represen- lalions, ICLR 2018, Vancowver, BC, Canada, Ayril 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL
fott/formmiid-Sie WWbCZ.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-paramctrization in generalization of neural networks. In International Conference on Learning Rep- resenlations, 2019. URL
Jaewan Park, Shashank Kushwaha, Junyan He, Seid Koric, Diab Abueidda, and Twona Jasiuk. Sequential deep learning operator network (s-deeponet) for time-dependent loads. arXiv preprint arXiv:2306.0822 2023.
Raphaäl Postouric, Youssef Mrouch, Chris Rackauckas, Payel Das, and Steven Johnson. Physics-cnhancod deep surrogates for partial differential equations. Nature Machine Intelligence, 5, 12 2023. doi: 10.1038/ $42256-023-007611
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
neural networks: A deep learn-
Bogdan Raonić, Roberto Molinaro, Tobias Rohner, Siddhartha Mishra, and Emmanuel de Bezenac. Convo- lutional neural operators. Xiv preprint arXi0:2302.01178 2023.
19

---

Published in Transactions on Machine Learning Research (11/2024)
Cristopher Salvi, Mand Lemercier, and Andris Gerasimovies. Neural stochastic pdes: Resolution insolution-invariant learning of continuous spatiotemporal dynamics. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 1333 1344. Curran Associates, Inc., 2022. URL https::/prooceednggs hewpreeeedddnnnnn..errsss...... neurips....... www..ps..c/ppppr........... wttpppper_fille/pppppaeeeeeeeeee ht2p://2266666660444444665515122
pdf.
Mark Sellke. On sixe-independent sample complexity of relu networks. Information Processing Lellers, 186:106482, 2024. ISSN 0020-0190. doi: http://.....rr110...0..0/////222
URL https: /www.
sciencedirect. http:///eeeeeeerrraaaaaa//////0/
Justin Sirignano and Konstantinos Spilioponlos. Dgm: A deep learning algorithm for solving partial differ- ential equations. Journal of compulational physics, 375:1339 1364, 2018.
Losley Tan and Liang Chen. Enhanced deeponct for modcling partial differential operators considering multiple imput functions. arXiv preprinl arXic:2202. 08942, 2022.
Tapas Tripura and Souvik Chakraborty. Wavelet neural operator for solving parametric partial differential equations in computational mechanics problems. Computer Methods in Applied Mechanics and Engineer- ing, 404:115783, 2023.
Nils Wandel, Michael Weinmann, and Reinhard Klein. Teaching the incompressible savier-stokes equations to fast neural surrogate models in three dimensions. Physics of Fluids, 33(4):047117, apr 2021. doi: 10.1063/5.0047425.. URL https://doi org/10..o... 1063%/255.0044440
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed DeepONets. Science Advances, 7(40))eabi8605, 2021. doi: 20.1126/sciadv.abi88
Wuzhe Xu, Yulong Lu, and Li Wang. Transfer learning enhanced deeponet for long-time prediction of evolution equations. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 10629-10636, 2023.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. In Proceedings of the 37th International Conference on Machine Learning, 1CML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 10767-10777. PMLR, 2020. URL http://proceeedings mlr, gress/v119/yangDD0 .ttll.
Bing Yu et al. The deep ritz method: a deep learning- numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1 12, 2018
Jiahao Zhang, Shiqi Zhang, and Guang Lin. Multiautoodeepona::: multi-rosolution autocncoder deeponet for nonlinear dimension reduction, uncertainty quantification and operator learning of forward and inverse stochastic problems. arXiv preprint or Xxx::2204.03100 2022.
20

---

Published in Transactions on Machine Learning Research (11/2024)
Appendix
A Evidence for Training DeepONets with tanh Activation and Huber Loss
Similar to the experiments in Section 4.3, in this section too use DeepONets whose both branch and trunk networks are of depth of 3 and have a width of 100. However, unlike Section 4.3, the activation function used in the experiments displayed in Figure 4 is tanh. The chosen training loss is the same as in equation 11, where & set to the Huber loss with - 0.25, the value as would be inspired by Theorem 4.2.
Numerical Solution Burgers Equation
Predicted Solution using DeepONet
32
02
022
0.4
0.4
50
50
40
40
20
10
Numertical Solution Burgers' Equation
Predicted Solution using DeepONet
00
0.
Figure 4: This plot demonstrates the ability of Huber trained DeepONets to predict the solution Burgers' P.D.E. on an arbitrarily chosen inhomogeneons term which is different from the u's used for training the net.
The plots above motivate future studies exploring the possibility of Huber loss being good for training DeepONets.
B Empirical Study on the Behaviour of Generalization Error for Huber Loss at Different 5
10
Fayero
Prwar
1500
1503
0.5
2.
1.0
13
05 1.0
Figure 5: These plots demonstrate the behaviour of generalization error and test error for Huber loss trained DeepONets to predict the solution to Burgers' P.D.E.. varying values of for diffeernt dataset sizes.
21

---

Published in Transactions on Machine Learning Research (11/2024)
Similar to the experiments in Section 4.3, in this section too use DeepONets whose both branch and trunk networks are of depth 3 and have a width of 100. The selected training loss is identical to the one in Equation 11, with l set to the Huber loss. The generalization error is plotted for various values of delta at Ptraining = 200 and 1500, while maintaining Ntrainiug = 400. In this setup, in Figure 5 we plot the generalization error for 8 (v2)= for k 4,5, " 18 to show how the generalization error as well as the test error drops as 5 gets smaller.
C Review of the Universal Approximation Property of DeepONets
We follow the setup in Lanthaler et al. (2022) for brief review of its key theorem that motivates DeepONets.
Definition (Solution Operator). Given U c Rn and D c RRd, compact sets with boundaries, suppose we have a differential operator L: (U) - C(D), where (U) the 1,2basod Sobolev space for some Denote the functions in the range of L to be "forcing" "input" functions and those in the domain of L to be the "output" functions. Let OU denote the boundary of U. Then given g (OU) and f C(D) solution to the differential system f: is function u² e HU) s.t.
Lu = = 8.6 u² g on OU.
At fixed "boundary condition" g for such differential system, we assume that the solutions for the above system for different forcing functions f are given via operator/map, 9 s.t.
G - s.t Co = and g(f)= g on ØU.
Further, assume that f probability measure on C(D) s.t (1) iee. SCLD) (9( du (/)
< co. Multiple examples of g have been discussed in Lanthaler et al. (2022), and bounds on these operators evaluated - for instance, in Lemma 4.1 therein case of a forced pendulum that we used as demonstrative example in Section 1.1
the bounds the that corresponds to the the
Definition (DeepONet (Version 2)). Suppose A f C(D), #B()) is discretization of f. Further suppose A is branch net mapping to R and R= + IR+1 is trunk net. Then "DeepONet' (N C(D) - (U)) is defined as,
P
ERC", (()()))==((0)))) Σ Au(o (xr)) k=1
Theorem C.1 (Restatement of a Key Result from Lanthaler et al. (2022)). Let μ be as in Definition 7. Let G: : C(D) + LU) be Borel measurable mapping, with G E L2(),, then for every > 0, there exists an operator network N: C(D) + (?) as given above, such that
|g NNllee - (Seop, (9( -00 <c.
1/2
Remark. Henccforth øg == ==:= 2F(f) for any function f, and similarly æB, for function ft.
The above approximation guarantec between DeepONets (W) and solution operators of differential equations (G) clearly motivates the use of DeepONets for solving differential equations. In Deng et. al. (2022), the authors present specific scenarios wherein particularly small DeepONets satisfy Theorem C.1. And as a counterpoint, in Mukherjee & Roy (2024), scenarios are demonstrated where small DeepONets would not be able to reduce the empirical training error below certain threshold in the presence of noise in the data.
D Proof of Lemma 6.4
Proof. Denote by S, Sdi-1 5dh-1 Define W (wrw)) and a; (#B))) Then,
22

---

Published in Transactions on Machine Learning Research (11/2024)
E sup (v..0))
=E.
sup
Σ ci
Σ h1-1
U1 BBiik)
(e),
i=1
E
sup
Es (B,A)) (02,22
(11,22)
k1= 12=1
dg
E.
sup
11itg U2
Σ Es (EB (UT.Ait))) (
(m)
k1= kz-1
Note (ha v2) Sn
W 2
Σ
22 1,k7
2
1111122
Ri
Agml
Hence, can enlarge the domain to get,
d
d2
$ E
sup
Σ
Σ
W kk)
Σ Es x kr
W
كك R1-1
1z-1
The above sum can be viewed as an inner product of 2 vectors in Rdixd xds We have,
Ee sup (W. [W
(w.
XT T
where the inequality follows from Theorem 5.5 in Ma (2021).
E Proof of Proposition 6.2
Proof. We recall from the setup of Definition 2 that b_1 and f_ are the number of rows of the matrices Byn-1 and T and b-2 and f-2 are the output dimensions of the nets fü and fr. From Definition 6 we further recall the definitions of the 2 sets of matrices WCCgg.mm-a-- and Wrest and we simplify the required empirical Rademacher complexity as,
mk
Σ e (I
sup (WiC -2 Wien)
1=1
E,
sup 1o- -op -o Wiea)
Σ Gs (Byn (Bnn-1f((BB.)) Tay 02 (or-)))))))))))
(17)
Here, we have substituted the definitions of the functions f/ and the f as given in the statement of Lemma 6.2. To case notation define the following vectors,
b, a (B S'g (x to 0z (T (f
23

---

Published in Transactions on Machine Learning Research (11/2024)
Then we can rewrite the above as,
mRm R.
Σ ci B bi, Toz Torlii)
sup
(W(C ng 01-28-1-1r ee----------11-- Wone) )
E
sup [W(C 11 Wiece)
Σ
6j
B yBu bs T Terighi
j=1
E2g
sup
Σ
ti
B Dinks
Σ To 2,
(W(C g ww---------eee. Weot)
F=1
j=1
1==1
b
E2
sup
2
B BBs Tor
Σ
(W wot-gg/rww---.00 -ler Weest)
1-1k1-1k2-
p
sup (W(C wTw 1.er -1.er 1). Wecat)
Σ
B Banijkk T Ton
Σ
Eebi,ki 1i,k2
1-1
k1 kz
1-1
p b
E2
sup
Σ
Σ
B un T F k
W(C o up Ler 1) Wean
j-l k1-11==11
Σ c (B
02 (Tor-1f4 (F lo2
p b_ 4
E
sup
Σ
B
qB
T (o
-1 1| 1, 1-l 1,
WCC B or 18-
k1=1 h2-1
Wrest
Σ G0
B 1 (((B))) (B 00 (f
In the last line above we have invoked the positive homogeneity of a and Ø2.
Now assume 2 vectors u and un of dimensions b - x t - 1 s.t they are indexed by the tuple (k1, k2) for ky -1 +& b and kg -1,...,-1+t.. as follows,
(22 1 [T Σ
(B T Tor J)
1=1
k2) Σ E (B I J' fE((),)) (I Ir (xT,)))
Then we have,
Σ Σ (())) (((k) }
m R E,
sup
10----------------1 Wine)
11
Note that for any a, B R we have, (a, S (maxpet,....k 13pl) Σラ layl
max lucks luccr {
s E
sup
(W BB -o -p W.oal
1--1
Now we define set of weights W Wexcept-outer which an union of all possible weight matrices that are allowed in Wrest and all possible choices of (By-- 1, T 1 that allowed in the set W(Ca -11-------11)-
24

---

Published in Transactions on Machine Learning Research (11/2024)
Then we recall the definition of W(C (n to get,
Σ c (B 12-1. fb((),))) 02 (fo--1 Fr (((oass))
m
mRn
E.
sup
Cupver-B-Loon-1
max
-exnept-outer
k-
kg
where
((kg
ki= 12=1
b Σ
[ er 1,22
Σ
B
an T 9r
12=1
Note that any pair of row directions in the pair of matrices (Bog-1, Tor-1) is in 5b.2-1 x 2t So a sup over the set of Bgg-1 and T that is allowed by the constraint of Cqp Bhs.rr-ag aor-00-1..r and a subsequent max over the pairs row directions can be upperbounded by single sup over 50-2-1 St-2-1, Thus we have,
Σ cor (v' (I((B,)) 02 (wi ((TT))))
mRmm $ R.
Cgu 1aauerraa-1.g--1
sup (u.w)esth-2-
2- 2- Wrest
Invoking Lomma H.l, we have
qo- Σ aoo (o fe(BB.)) 02 (w (I }
RRm Be 2
sup
Wean
Now we invoke Lemma 6.1, assuming that Assumption 1 holds for σi and 02 for some L > 0, get,
Σ ao fa(#B,) w SA(TTT)) |
mRRm 2L -nn-er.nn--or-11 Eg
sup
(v.w)ess----ygs x
Weat
The above inequality exactly what we set out to prove.
F Proof of Proposition 6.3
Proof. We start with the expression in the R.H.S. of the Proposition 6.2 and simplify it similarly as was done in its proof the previous appendix. Denote S2 Sh-a-1 5h-z11
Σ S.v fb(BB.) www w F((ET.))
E,
sup ( Weant
E
sup
Σ 0 01 B (f ut 02 02202TTT (eTlo2TTpppppp (Pg-22(((Tr)))
(w)a0) S2 W Weost
b Σ
= E
sup
Σ (ow ) B 2.h T pr 2-22 Σ Gor [ - (o (
(w,w) e j==1 f==1 1==1
Wenet
Define
0 = (ww') ) .z Byn 2. T
Σ
c0 (
2,j J
)o (T 2 f*(TT,))
1-1
25

---

Published in Transactions on Machine Learning Research (11/2024)
Thus get,
Eg
sup (v,w) 52
Σ , fB((B,)) w fr (((T)) F=
= E.
sup
Σ
Σ
(v, Sq jr =1 j==l Wreet
Wraet
< E2
sup
Σ Σ ji 52m1
max 31
Weest
For ease of notation we define the set W-2.-2 as the set of matrices allowed in Wratt but with those with indices QB - and Gr 2 additionally satisfying the constraint in equation 16 in the statement of the lemma.
Σ e fa(wns)) w (((T)) 1=1
E
sup (v,20) E Weest
C- 2,
sup W-2.2
max
1 1...- 12..... 2r-2
2
sup
max
Σ 1
[ F
(Tr - ff((xT)))
W-2-2
f==1...9n
We note that any pair row directions in the pair of matrices (B Tor-2) ) is e Sh 3-1 St s-1 So a sup over the set of B and Tør-2 that allowed by the constraint of C-2,-2 and a subsequent max over the pairs of row directions can be upper-bounded by single sup over 5h x St-3-1
We recall the definition of W given in the lemma to conclude, (Sa = S a-1 x St 2-1)
Σ Go fa((B)) - w F(((T:)))
E sup (u (v.w) S2 Weant
$ 2
sup (v.a0) S W
Σ c fh(zz)))oz(w' i-
Invoking Lemma H.1,
$ 2C_ 2,
sup Σ (0 f"" fi(ar))) (w,w) 1= W Mest
m ff f+(2r)))
$ 2IC-2,-2E,
sup (n,w) #1 Wi
In the last step above we have invoked Lemma 6.1 using the fact that Assumption is true about a and d2.
G Generalization Bound For DeepONets For Unbounded Data
Definition 9 (dx Loss Function Class). Given H as defined in Theorem 4.2, we define the class dx associated with as,
Φu = (Fx R (f,x)) ((((f.)))) H}
where l is some non-negative univariate (loss) function
26

---

Published in Transactions on Machine Learning Research (11/2024)
Applying Theorem 4.13 of Ma (2021) to the above loss class get the following theorem Theorem G.1. Given on as in Definition 9 we have,
E ((D.ar:)
sup ke
(((h(f..T).))) -Br [((h(f,=))))
$ 2 (Φn)
where m is the number of samples used in the empirical risk, D is a distribution over F x R and R,m is Rademacher complexity as defined in Equation 9.
Lemma G.2 (Lipschitz Composition in Rademacher Complexity). Assume that Ø: IR is Lø- Lipschitz continuous function, such that (Ø(t) <(s) LØlt s|. Suppose V R". Then,
sup Σ E((((i) $ Lig fav m
sup Σ E fey m
Lemma G.3 (R(E(H)) in terms of R(H)).. Given H as defined in Theorem 4.2, if ((.) is R R-Tipschitz,
E sup he
R(((())) R R-R(H) R E, c ((,(s,
sup he? m
Σ 6 h (((.
Proof. Any function e H maps from F x Reda R. Hence, training data of size m for the DeepONet of the form {(f,, DT,) e Rdd , 1,2,...,
Consider the set K= = {(h(fi
T H) c IR. Then we can re-write,
R(E(H)) = Ee sup
- Σ a (((())))
1 m
kee m 1-1
Invoking that the Lipschitz constant for E(.) is R, and Lemma G.2 have,
1 sup Σ kek m=
R(((H)) R.E.
E
kg
which we recognize as the inequality we set out to prove,
R(E(H)) s R R(H)..
Lemma G.4 (R(H) in terms of RR ((DeepONet)). Recall that W is the set of allowed weights in the function class H. (as defined in Theorem 4.2). Corrresponding to it define the class DeepONet s.t DeepONet DeepONet denotes a DecpONet with weight w é W. Then the following holds,
1 m
R R(DeepONet) = E. sup = weW i-1
Σ
LeepONet,,,ff. (f.. R(H).
Proof. Realizing that taking a sup over h and w W are equivalent, we get,
1 E, sup Σ q (h((i, hew m [-1
2T,)
E.
sup E& (G(f). DeepONet, (e weW m im
27

---

Published in Transactions on Machine Learning Research (11/2024)
where we have used Z())) ((f.. Df - We can write R(H)
1
R(H) = E.
sup
Σ (G(f); - DeepONet,,tttt (f.
WEW m
m (T,
1 m
= E,
sup
Σ
Σ
DeepONel 20.,))
weW
i=
1
E,
Σ
c
sup
Σ (-c)
DeepONet, (f xT.1)
m 1=1
weW m 1=1
0+ E.
sup wew m 1=1
Σ
W (T,)
ReDeepONee).
since, -e in distribution.
Hence combining the above two lemmas we get, Lemma G.5 (R(E(H)) in terms of R(DeepONet)). Let e(-) be R-Tipschitz. Then,
R(((H)) & R R(DeeppNet).
Proposition G.6 (Generalization Error Bound for DeepONet). Given 6H in Definition 9 and let E(-) be R-Lisschizz. Then we have the following generalization bound,
E ((far,)) DD (i-1...)
sup heat
- ( $ Rm
where Rm as in Theorem 4.1, and is a distribution over Red
Proof. We recall from Theorem G.1 that,
E ((
sup he7 m71
Σ ((h(Ji, B( ((()([ 2 (pH)))
Now, invoking Lemma G.5 and using the fact that E(.) is R-Lipschitz get the claimed incquality.
H Contraction Lemmas
Lemma H.1. (From Ma (2021)) Let e Uniform( Uniform((1,-1]3 (1,-1)m") and suppose we have functions fo and go parameterized by 0 s.t, (I+)) 9 x (x)) (ge (æ1) ..... (xm))) E.. Suppose that for any (+1)""
supa (c)) 2.. 0. Then,
E. sup (c. fe(x))) 2E, sup (e, (e
Proof. Letting & be the ReLU function, the lemma's assumption implies that sup, di (le, fa(x))) supo (e.))()) for any e {+1}". Observing that
) s (snoo(ccff()))=sps(((((fff
= -upp fo(x)) sup fo
Taking the expectation over (and noting that and -e have the same distribution) we get the desired conclusion.
28

---

Published in Transactions on Machine Learning Research (11/2024)
H.1 Proof of Lemma 6.1
Proof.
Rm (0p o ...pp..p- 2) 1 ((-1((-))))))
Σ (=po))(()
sup
m
(x.))epx
We explicitly open up the expectation on 61 to get.
1
" (p(((=:)))o((()))) - (popeele))) fxoexx
sup
2m
sup P
1
(00()))))))0((()))))))))) op (=))))))) (d)()))
sup (()x)))
2m
m
By invoking the the assumption we get,
(L. (L-o(a))a(e) ( (()))d
1
B(ca...)
sup ((r:))
2m
s (or(0(=))) ( (w:) (v)))
Now we invoke the fact that inside supremum, an absolute value function redundant, since anyway the higher combination will get picked. Thus rearrange the above to get,
1
E
sup (PiA)EPeQ
(LP(Zz))((1))
Σ 4 op((lx:))
2m
- (L.p'(x))''())) (e
sup (P':)))PX
Σ (ple(x:)))
E, sup (p.9)ePxe
(.-L)(()))()))
Re-iterating the above argument for i= 2,3, we get,
E E.-U((-111)
sup
Σ L p(xs)a(v)) 1=
m
(P-
LRn (p.(pP.)) Q).
29

---

Published in Transactions on Machine Learning Research (11/2024)
H.2 A Contraction Lemma with Biased Absolute Functions
Suppose F is set of functions with common domain D and common range space, and B is subset of that range space. Then we denote new function space F B == (x+ f(x) b, Vx D e F, be B).
Lemma H.2. Let P and g be set of functions valued in R closed under negation. Given points {z; i= I,...,m), {3/ i ... in the domain the functions then we have the following inequality of Rademacher complexities where both the sides are being evaluated on this same set points,
R. (PP B1-- B2)) Rm
Proof. From the definition of Rademacher complexity it follows that,
Rm (mp 12 Bal) - -B-U((-111)")) sup P.gEPx
Σ 12
m
cs (p(x))))))))))))))())))) + Billu(y:) + Ba
We explicitly open up the expectation on to get,
[I((e)+ Bill((y)) (Billv)) B2l + Σ ==000)())))
sup p.pePx
2m
sup Px
+ B2l E (p(p((
(18)
sup p.p PPPPPPPPPPPPO (((a))(e))) (( (m)) Bellp(a:)- -Bil()))))) ((n))
$
E (ea 6m)
2m
4 ((le)tt)))d(())))))ooo00/00a11()
(19)
+ ))(u(a:)))o((())))
sup Px
2m
(B) ( (20)
sup s/ sx
5 m
Eg
sup
(21)
1
1-2
Iterating this, get
1 B..-(((111))m)) sup P.
Σ (p(2.)9(1)) BBBl((x,) +B:l)))))BBB(()))))
m
$ Rm (P. 2) + R (P) + Rm Q).
In equation 19, we have used the triangle inequality, followed by
[a(a))) Bil Bi) -
Bal (((t))) B2) (B) B2)
equation 20, we could open up the since the supremum would be positive. This follows from the fact that P and Q are closed under multiplication by 11, which implics sup f sup =usupper -f sup lff
30

---

Published in Transactions on Machine Learning Research (11/2024)
Converting - to abs -eet
Firstly, we recall that the map R 9 a - a E R is an exact depth 2 ReLU net - one which passes every coordinate of the input through the ReLU net R a 2 + [0, ma E R. Using this, given any ReLU DeepONet, we can symmetrize the depths wetween the branch and the trunk by attaching the required number of identity computing layers at the input to the shorter side.
Secondly, for typical DeepONet experiments, one can assume that the the set of all possible input data is bounded. Combining this with the assumption of boundedness of the allowed matrices, conclude that 3 B 0 s.t the input to any ReLU gate in any DeepONet in the given class is bounded by B. Now we observe that v|2| & B, we can rewrite the map H ma 2} as, 2 2 } BJ -l=
Hence, doing the above replacement at every gate we can rewrite any ReLU DeepONet (without biases) as a DeepONet using only absolute value activations but with biases and computing the same function on the same bounded domain, at the cost of increasing the size of the branch and the trunk net by a factor of 3. Thirdly, a similar result as in Lemma 6.1 continues
hold for this setup as given in Lemma H.2, Note that, Lemma H.2 bounds the Rademacher complexity of DeepONet class with the branch and trunk depths (k, k) by linear sum of that of - 1, 1) and sum of a (k, 0) and (0, DeepONet. While the first term can be recursed on again (using identical techniques as used to prove Theorem 4.1), for the last two one can invoke Theorem 1 of Golowich et al. (2018).
Thus, we observe that following the same arguments as in the proof for Theorem 4.1 we can derive an analogous bound for arbitrary ReLU DeepONet classes but with twice the depth of the DeepONet number of extra terms in the R.H.S. These extra terms would come in pairs each pair consisting of a Rademacher complexity bound standard net, from the branch side and one from the trunk side and of decreasing depths.
J Choosing Č-K,-b
Defining
= X Rloxx s.I. X = Ban | we can simplify as follows the
b.. kI=l..n expression defining C-k,-k in equation 4,
sup (v,w))s-1+b KxS-14 k 11=11gm 12=1
Σ
(vw))) )i 1I
sup (w,w)es-1+b 1+b 1xt J 1==1
| Bn-k.j |I| k.j 1,w
b Σ
sup (o,w)s 1 lickk
Σ
|0
X Xja.2" |ujall
sup (w,w)eS
f==1
11b-kx
1ikk
In the last line above we defined vectors ē and in of appropriate dimensions s.t. ū5 |oa] and ū; = |wsl- Then note that, XW | s [ XX|] |
In above | is the spectral norm of X. Therefore we have the following upperbound,
sup V X w/2 sup 1 sup |w|2 |X|
Since |||| = =|/i| S and || = -||o| s I we have,
sup ( g-its
x i
B TTll
Thus it follows that the RHS of the above inequality gives intuitive candidate for the quantity Č-R-K-
31

---

Published in Transactions on Machine Learning Research (11/2024)
K
Behaviour of Rademacher Bound for l--loss
X10
1.05
Correiation Confficiant
05050
0.95
3 0c105
0e+05
0.0e05
0.85
1.0e-06
0.75
2.4c006
2.0
2.5
3.0
Generaization Error Bound
X10
Figure 6: The above plot shows the behaviour of the measured generalization error with respect to C2,2 for training DeepONets
mm
solve Burgers' PDE using the empirical loss given in equation 11, specialized to the &g loss and for the branch and the trunk nets being of depth 3. Each point is labelled by the number of training data used in that experiment.
L Link to Code
Here is the link to the GitHlub repository containing our code for training a DeepONet for the Heat and Burgers' P.D.Es.
32

---

Published in Transactions on Machine Learning Research (11/2024)
M
Plot Showing the Evolution of Solution to the 2D Heat P.D.E. with Time
Prddcted Soution OwwzO att-332
Figure 7: This plot demonstrates the solution to 2D Heat P.D.E. at different times. In each figure, we have shown the analytical solution first followed by the prediction from the DeepONet trained with & and Huber loss at 0.25. Here, the DeesONets we used have ReLU activation functions and include bias terms.
33


Reviews:
[{"review_id": "taDRlqfQF5", "reviewer": "Reviewer_Vq8r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strong side of the paper is the presentation of the result and the application of the Huber loss to training\n\nMy concern is the formulation of parameter independence. The Huber loss is directly proportional only to the accuracy of the sequence of itself and the Rademacher complexity models a function as a random system learner.  So doesn't this mean that actually the complexity represents the learnability or generalization in terms of the function complexity projected to the unitary encoder, the random encoding? Therefore I am not sure if both this work and Lanthaler work are not equivalent", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a theoretical and experimentally verified bounds on the complexity of the o.d.e. learning using a specific type of artificial neural networks called the DeepONets. This type of networks train simultaneously two networks in parallel one for the functions it self and one for the time step. The networks then combine their outputs to generate a value and the time index of the o.d.e. The main work in this paper is purely theoretical and the authors demonstrate that unlike in Lanthaler et. al showing that the generalization grows proportionally to the exponential growth of learning parameters, here the results demonstrate that the generalization is proportional to the capacity of the network. While these two seems to be independent the capacity of the network is a less precise measure especially in the complex methods and thus the result is interesting. This is achieved in particular when the network is trained using a specific type of loss, notably the Huber loss. The result is a bound obtained by Rademacher complexity that describes the ability of representing a function by a random encoder and in particular on the number of samples in the training set. As such the authors are able to reduce the problem to a learner independent approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Explain in more details the actual difference between Lanthaller and this work", "broader_impact_concerns": "My only question is how is the Huber loss useful in the training of the o.d.e. That is is a standard approach or does it require specific set of preparatory steps to be applied?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yZI7SL5dk5", "reviewer": "Reviewer_mR3e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Although the techniques are simple, it is overall an interesting work.\n\nStrength: The logical flow and the motivation are clear.   Simple yet interesting and important contributions that study generalization bounds on a specific instance of neural network architectures.\n\nMajor weakness: \nThe main concern is its structure and writing in some parts.\nIt seems that several claims and descriptions are somewhat scattered around the paper.  And some sentences are a bit unclear to me.\nAlso, mathematical expressions in some parts are unclear as well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work shows Rademacher complexity of DeepONets which scales well by applying “peeling” lemma and a variation of the Talagrand contraction lemma.  And it suggests the use of Huber type loss.  Some of the theoretical claims are reflected by toy experimental examples.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. There are several comparisons to existing work; please summarize them concisely in one section or so so that the relation can be clear.  Especially, I would like to see a subsection for related work on sample complexity and another subsection for related work on mathematical proof techniques.\n\n2. It seems the authors separate meta summary of the work and mathematical setups, which looks good to me; however, if that is the case, meta summary part may need to be more intuitive so that there won’t be much overlap with the mathematical setup parts.\n\n3. Experimental sections are not really well structured; please make it flow well and add sufficient descriptions to each figure.  Also the texts in figures are a bit too small.\n\n4. It says the complexity does not (explicitly) depend on width and depth; how does the constants $C$ affect the complexity?  It seems exponential?\n\n5. Definition 6 is a bit hard to parse.\n\n6. page 28 bottom; “since the supremum would be positive…” can you elaborate more on this?\n\n7. $f$ and $f'$ are introduced; it seems $f'$ is used differently in different parts. (like in page 26)  Can you unify them?\n\nMinor changes\n\n1. maybe summarize the abstract as one paragraph (delete empty line)\n2. page 1 “deep neural version” → “deep net version”?\n3. P.D.E → “P.D.E.” ?\n4. $\\mathbf{R}^2 \\ni$ xxx $\\in \\mathbf{R}^2$ is strange; you don’t need double of this $\\in$.\n5. period in equation 1\n6. page 3; the pendulum at time $t = x_t$; this equal is weird.\n7. Informal theorem and formal theorem may want to have unified numbering or so; otherwise mentioning theorem 3.2 or so before introducing it is strange.\n8. page 4 last paragraph is a bit hard to parse.\n9. in Definition 2;  I would avoid using \\& symbol just to mean “and” in math environment.\n10. in Definition 2; the way f’ is defined is a bit strange; maybe it is ok but this is too intuitive definition that might cause misunderstanding.\n11. in page 6; what do you mean by “… and facing bounded data”?\n12. in page 8; the definition of $\\mathcal{H}$ is a bit unclear (you could put the set as a condition in the set)\n13. did you define $\\rho,T$ in page 8?\n14. $G$ and $\\mathcal{G}$\n15. in page 8 DeepONet (G) is confusing as it might mean that the DeepONet is a function taking G\n16. bullet points in page 8; equal $=$ is used in a weird way.\n17. there are may places where the parenthesis is not closed or overly closed; please check them.\n18. in page 27; first inquality no $L$ needed.  Third inquality is equality?\n19. the notation for uniform distribution is introduced before it is defined in page 28.\n20. please define $|\\mathcal{P}+B|$.\n21. second line of Lemma I.2.  “function in we” → “function then we”?  Please check other typos as well.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bT4igEla9r", "reviewer": "Reviewer_NK7K", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1.The paper is well-written and well-organized.\n2.The results presented effectively explain the training of DeepONets. For instance, the model-size-independent results improve upon previous work and may provide valuable insights (see below weakness 1).\n3.It discusses two distinct loss functions, $l_2$ loss and Huber loss, showcasing the extensibility of their work.\n4.The experiments conducted are good, validating their theoretical claims.\n\nWeaknesses:\n\n1.The discussion on their insight (how the choice of the hypothesis class, the choice of the loss function, and the data distribution interact to determine the ability to find optimal solutions through empirical estimates) is missing, despite being mentioned in the Introduction section.\n\n2.The explanation of their theoretical results is limited. The authors should elaborate on why the results are independent of widths, depths, and parameters. Are these terms synonymous? Could the summation of rows of the weight matrices be considered one of the parameters?\n\n3.Assumption 1 appears strong. Are there related works that use the same assumption? Please provide references. Furthermore, if the activation function satisfies Assumption 1, are there any other conditions it must meet, such as monotonicity?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a theoretical analysis of DeepONets, focusing on the generalization bound with two different loss functions: $l_2$ loss and Huber loss. The results surpass previous findings in several ways: 1. Their generalization bound is independent of the number of training parameters, explaining the success of overparameterized architectures in modern models. 2. They extended the results to Huber loss, demonstrating its empirical effectiveness in training neural networks to solve partial differential equations (PDEs). 3. Based on these findings, the paper potentially offers insights into how the choice of the hypothesis class, the choice of the loss function, and the data distribution interact to determine the ability to find optimal solutions through empirical estimates.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested Changes:\n\n1.Add the insights mentioned in the Introduction section and include experiments to validate them (related to Weakness 1).\n2.Provide more discussion on their generalization bounds and assumptions.", "broader_impact_concerns": "No impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Given that the three reviewers unanimously agree on both the claims and audience questions, it seems a clear accept.
Decision: Accept (poster)