Size Lowerbounds for Deep Operator Networks
Authors: Anirbit Mukherjee
Amartya Roy
Venue: Accepted by TMLR ()

Abstract: Deep Operator Networks are an increasingly popular paradigm for solving regression in infinite dimensions and hence solve families of PDEs in one shot. In this work, we aim to establish a first-of-its-kind data-dependent lowerbound on the size of DeepONets required for them to be able to reduce empirical error on noisy data. In particular, we show that for low training errors to be obtained on $n$ data points it is necessary that the common output dimension of the branch and the trunk net be scaling as $\Omega \left ( \sqrt[\leftroot{-1}\uproot{-1}4]{n} \right )$.

This inspires our experiments with DeepONets solving the advection-diffusion-reaction PDE, where we demonstrate the possibility that at a fixed model size, to leverage increase in this common output dimension and get monotonic lowering of training error, the size of the training data might necessarily need to scale at least quadratically with it.

Published in Transactions on Machine Learning Research (02/2024)
Size Lowerbounds for Deep Operator Networks
Anirbit Mukherjec
Department of Computer Science The University of Manchester
Amartya Roy"
Amartya. Roy@in, bosch. com
Robert Bosch GmblI, Coimbatore, India
Reviewed OpenReview: https: openreviee net/ formm??m id= RumWODTNFE
Abstract
Deep Operator Networks are an increasingly popular paradigm for solving regression in infinite dimensions and hence solve families of PDEs in one shot. In this work, we aim to establish a first-offits-kind data-dependent lowerbound on the size of DeepONets required for them to be able to reduce empirical error on noisy data. In particular, we show that for low training crrors to be obtained on n data points it necessary that the common output dimension of the branch and the trunk net. be scaling as Q (vn).
PDE, This inspires our experiments with DeepONets solving the craetiionddfffsioo---eee where we demonstrate the possibility that at a fixed model size, to leverage increase in this common output dimension and get monotonic lowering of training error, the size of the training data might necessarily need to scale at least quadratically with it.
1 Introduction
Data-driven approaches to analyze, model, and optimize complex physical systems becoming popular Machine Learning (ML) methodologies gaining prominence. Dynamic behaviour of such systems is frequently characterized using systems of Partial Differential Equations (PDFs). A large body of literature exists for using analytical or computational techniques to solve these equations under a variety of situations, such as varions domain geometrics, input parameters, and initial and boundary conditions. Very often one wants to solve a "parametric" family of PDEs i.e have a mechanism of quickly oblaining new solutons to the PDE upon variation of some parameter dynamics model. This is tantamount obtaining a mapping between the space of possible parameters and the corresponding solutions to the PDE. The cost of doing this task with conventional tools such as finite element methods (Brenner & Carstensen, 2004) is enormous since distinct simulations must be run for each unique value of the parameter, be it domain geometry or some input or boundary value. Fortuitously, in recent times there have risen a host of ML methods under the umbrella of "operator learning" to achieve this with more attractive speed- accuracy trade-offs than conventional methods, (Ray et al., 2023)
the PDE like say the viscosity in fluid
As reviewed in (Ray et al., 2023), recognize that operator learning is itself part of the larger program of rapidly increasing interest, "physics informed machine learning" (Karniadakis et al., 2021). This program encompasses all the techniques that are being developed to utilize machine lcarning methods, in particular neural networks, for the numerical solution of dynamics of physical systems, often described as differential equations. Notable methodologies that fall under this ambit are, Physics Informed Neural Nets (Raissi & Karniadakis, 2018), DeepONet (Lu et al., 2019), Fourier Neural Operator (Li et al., 2020b), Wavelet Neural Operator (Tripura & Chakraborty, 2022), Convolutional Neural Operators (Raonic al., 2023) etc.
Pnysics-Informel Neural Networks (PINNs) have emerged notable approach when there is specific PDE of interest that needs to be solved. To the best of our knowledge some of the earliest proposals of
part of the work done while the author was at the Jadaypur University

---

Published in Transactions on Machine Learning Research (02/2024)
this were made in, (Dissanayake & Phan-Thien, 1994; Lagaris et al., 1998; 2000). The modern avatar of this idea and the naming of PINNs happened in (Raissi et al., 2019). This learning framework involves minimizing the residual of the underlying partial differential equation (PDE) within the class of neural networks. Notably, PINNs are by definition an unsupervised learning method and hence they can solve PDEs with no need for knowing any sample solutions. They have demonstrated significant efficacy and computational efficiency in approximating solutions to PDEs, as evidenced by (Raissi et al.; 2018), (Iu et al., 2021), (Mao et al., 2020), (Pang et al., 2019), (Yang et al., 2021), (Jagtap & Karniadakis, 2021), (Jagtap et al., 2020), (Bai et al. 2021), A detailed review of this field can be seen at, (Cuomo et al., 2022).
As opposed to the question being solved by PINNs, Deep Operator Networks train pair of nets in landem to learn a (possibly nonlinear) operator mapping between infinite-dimensionaa Banach spaces - which de-facto then becomes a way to solve a family of parameteric PDEs in "one-shot". Its shallow version was proposed in (Chen & Chen, 1995b) and more recently its deeper versions were investigated in (Lu et al., 2019) and its theoretical foundations laid in (Tanthaler et al., 2022a).
Till date numerous variants of DeepONet models (Park et al., 2023), (Liu & Cai, 2021), (Hadorn, 2022), (Almeida al., 2022). (Lin et al., 2022), (Xu et al., 2022), (Tan & Chen, 2022), (Zhang et al., 2022), (Goswami et al., 2022) have been proposed and this training process takes place offline within predetermined input space. As result, the inference phase is rapid because no additional training is needed as long as the new conditions fall within the input space that was used during training.
Other such neural operators like FNO (Li et al., 2020b), WNO((Tripura & Chakraborty, 2022) enable effi- cient and accurate solutions to complex mathematical problems, opening up new possibilities for scientific computing and data-driven modeling. They have shown promise in various scientifie and engineering appli- cations including physics simulations (Choubineh et al., 2023), (Gopakumar et al., 2023), (Li et al., 2022b), (Lchmann et al., 2023), (Li et al., 2022a), image processing (Johnny et al., 2022), (Tripura et al., 2023), and weather-modellin (Kurth et al., 2022), (Pathak et al., 2022).
A deep mystery with neural nets is the effect of their size on their performance. On one hand, we know from various experiments as well as theory that the asymptotically wide nets are significantly weaker than actual neural nets and they have very different training dynamics than what is true for practically relevant nets. But, it is also known that there are specific ranges of overaarametrization at which the neural net performs better than at any lower size. Modern learning architectures exploit. this possibility and they are almost always designed with a large number of training parameters than the size of the training set. It seems to be surprisingly easy to find overparametrized architectures which generalize well. This contradicts the traditional understanding of the trade-off between approximation and generalization, which suggests that the generalization error initially decreases but then increases due to overfitting as the number of parameters increases (forming a U-shaped curve). However, recent research has revealed puzzling non-monotonic dependency on model size of the generalization error at the empirical risk minimum of neural networks. This curious pattern is referred to as the "double-descent" curve,(Bella,, et al., 2019). Some of the current authors had pointed out (Gopalani & Mukherjee, 2021), that the nature of this double-descent curve might be milder (and hence the classical region exists for much large range of model sizes) for DeepONets- - which is the focus of this current study.
It is worth noting that this phenomenon has been observed in decision trees and random features and in various kinds of deep neural networks such as ResNets, CNNs, and Transformers (Nakkiran et al., 2021). Also, various theoretical approaches have been suggested towards deriving the double-descent risk curve, (Belkin et al., 2018a), (Belkin et al., 2018b), (Deng et al., 2022), (Kini & Thrampoulidis, 2020).
In recent times, many kinds of generalization bounds for neural nets have also been derived, like those based on Rademacher complexity (Sellke, 2023), (Golowich et al., 2018), (Bartlett et al., 2017) which are uniform convergence bounds independent of the trained predictor or results as in (Li et al., 2020a) and (Muthukumar & Sulam, 2023) which have developed data-dependent non-uniform bounds. These help explain how the gencralization error of deep neural nets might not explicitly scale with the size of the nets. Some of the current authors had previously shown (Gopalani et al., 2022) the kirst-of-is--innd Rademacher complexity bounds for DeepONets which does not explicitly scale with the width (and hence the number of trainable
2

---

Published in Transactions on Machine Learning Research (02/2024)
parameters) of the nets involved. Despite all these efforts, to the best of our knowledge, it has generally remained unclear to how one might explain the necessity for overparameteriaation for good performance in any such neural system.
In light of this, a key advancement was made in, (Bubeck & Sellke, 2023). They showed, that with high probability over sampling n training data in d dimensions, if there has to exist neural net f of depth D and parameters such that it has empirical squared-loss error below a measure of the noise in the labels then it must be true that, Lip(f) 2 2 (V Dp) This can be interpreted as an indicator of why large models might be necessary to get low training error real world data. Building this work, prove the following result (stated informally) for the specific instance of operator learning as we consider,
Theorem 1.1 (Informal Statement of Theorem 4.2). Suppose one considers a DeepONet function class at a fixed bound on the weights and the total number of parometers and both the branch and the trunk nels ending in a layer of sigmoid gates. Then with high probability over sampling a n-sized training data set, if this class has to have predictor which can achicve empirical training error below label noise dependent threshold, then necessarily the common output dimension of the branch and the trunk must be lower bounded Q Vn). And notably, the prefactors suppressed by Q scale inversely with the bound on the weighls and the size of the model.
Thus, to the best of our knowledge, our result here makes a first-of-is--kind progress with explaining the size roquirement for DecpONets and in particular how that is related the available size of the training data. Further, motivated by the above, we shall give experiments to demonstrate that at fixed model size, for DeepONets to leverage increase in the size of the common output dimension of branch and trunk, the size of the training data might need to be scaled at least quadratically with that.
The proof in (Bubeck & Sellke, 2023) critically uses the Tipschitzness condition of the predictors to leverage isoperimetry of the data distribution. And that a fundamental mismatch with the setup of operator learning since DeepONets are not Lipschitz functions. Thus our work embarks on a program to look for analogous insight as in (Bubeck & Sellke, 2023) that applies to DeepONets.
1.1
The Formal Setup of DeepONets
We recall the formal setup of DeepONets (Ryck & Mishra, 2022). Given T>> and Rd compact, consider functions u: [0, 7] R for 2 1, that solve the following time-dependent PDE,
(t), =0 and u(x,0) u4 (tt
This abstracts out the use case of wanting to find the time evolution of k dimensional vector fields on d dimensional space and them being governed by a specific P.D.E. Further, let H be the function space of PDE solutions of the above. Define a function space y s.t uo y c (D) i.e the space of initial conditions and then the differential operator above can be chosen to map as, La : H L ['O, x D) and these operators are indexed by function E Z (D))
Corresponding to the above we have the solution operator ; + H: f- u. where f {wo, a} X e { 2} where the two choices correspond to the two natural settings one might consider, that of wanting to solve the PDE for various initial conditions at a fixed differential operator or solve for a fixed initial condition at different parameter values for the differential operator.
The DeepONet architecture shown in Figure 1 consists of two nets, namely Branch Net and a Trunk Net. The former is denoted by WB that maps Rdh + IR which in use will take as input a d1 point discretization of a real valued function ʃ as a vector, s of "sensor points" {z, 1 J j S di} D. The Trunk Net, denoted by Nr, maps R + R2 which takes as input any point in the domain of the functions the solution space of the PDE. (In the context of the above PDE we would have d2 11 + d). Note that in above g is an arbitrary constant. Then the DeepONet with parameters 0 (inclusive of both its nets) can be defined as the following map,
)) corresponding to some arbitrary choice
3

---

Published in Transactions on Machine Learning Research (02/2024)
Go
f (o (p) (N)(s))
(1)
One would often want to constraint p e U where U compact domain in Rd Given the setup as described above, the objective of a DeepONet to approximate the value (p) by Go (S (I1), (x02), , (xa,) (p).
nodes
Branch-Net: RR% + RR99
DeepONet(E,pp
Product
Trank-Net: 224 R23
Figure 1: Sketch of the DeepONet Architecture
Review of the Universal Approximation Property of DeepONets An universal approximation theorem for shallow DeepONets was established in (Chen & Chen, 1995a). A more general version of it established in (Lanthaler et al., 2022b) which we shall now briefly review.
Consider two compact domains, D c Ied and U a R", and two compact subsets of infinite dimensional Banach spaces, K1 c (DD) and K₂ c C(U), where C(D) represents the collection of all continuous functions defined on the domain and similarly for C(U). We then define (possibly nonlinear) continuous operator 9: K1 K2.
Theorem 1.2. (Restatement of key result from (Lanthaler et al., 2022b) on Generalisod Universal Ap- proximation for Operators). Let μ. P(C(D)) be probability measure on C(D). Assume that the mapping G: C(D) (U) is Borel measurable and satisfies G & (P)).. Then, for any positive value there erists an operator 9 C(D) (U) (a DeepONet composed with a discretization map for functions in C(D)), such that
1/2
19- (6-1))))))) = (Cop,11(0 ((W)))
<8
In other words, 9 can approximate the original operator & arbitrarily close in the ( with respect the measure μ. The above approximation guarantee between DeepONets and solution operators of differential equations (9) clearly motivates the of DeepONets for solving differential ial equations.
4

---

Published in Transactions on Machine Learning Research (02/2024)
2
Related Works
In (Lanthaler et al., 2022b) the authors have defined the DeepONet approximation error as follows,
dy du(u)) 1/2
E ((WW))
where the DeepONet approximates the underlying operator g C(D) C(U) and j being as defined previously and u a fixed finite grid discretizalion of To the best of knowledge, the following is the only Nee size lowerbound proven previously,
Theorem 2.1. Let μ ((L)(T)) Let ((u) denote the operator, mapping initial data u(x) to the sobulion at lime 1 = 2/2 the Burgers' PDE (Hon & Mao, 1998). Then there erists universal constont C>0 (depending only on μ. but independent of the neural network architecture), such that the Deep ONet approzimation error & is,
E> C
Vq
where, is the common output dimension of the branch and the trunk net.
Firslly, from above it does not seem possible to infer any relationship between the size of the neural architec- ture required for any specified level of performance and the amount of training data that is available to use. And that is a key connection that is being established in our work. Secondly, the above lowerbound is in the setting where there is no noise in the training labels while our bound specifically targets to understand how the architecture constrained when trying to get the empirical error to below measure of noise in the labels i.e our setup that of solving PDEs in supervised way while accounting for uncertainty in data. Thirdly, and most critically, the lower bound above is specific to Burger's PDE, while our thcorem is PDB-independent.
Organization Starting in the next section we shall give the formal setup of our theory, In Section 4 we shall give the full statement of our theorem. In Section 5 we shall state all the intermediate lemmas that we need and their proofs. In Section 6 we give the proof of our main theorem. Motivated by the theoretical results, in Section 7 we give an experimental demonstration revealing a property of DeepONets about how much training data is required to leverage any increase the common output dimension of the branch and the trunk We conclude in Section 8 delineating some open questions.
3
Our Setup
In this section, we will give all the definitions about the training data and the function spaces that we shall need to state our main results. As a specific illustration of the definitions, we will also give an explicit example of DeepONet loss function.
Definition Training Datasets
Let (yi, (si,P:) be i..dd sampled input-output pairs from distribution on [-B,B]x DxU where D and U are compact subsets of Rdi and Rd2 respectively and define the conditional random variable g(s, p.) = Eyy (s).
Definition 2. Branch Functions & Trunk Functions
B = {Bur function with s da parameters Bu Rd R3, Lip(Bw) s La & |wlp 2 s Wn & |Bwll o0 sC}
T (Tw function with s dr parameters Tw d2 RS, Lip(Tw) s LT |wll, s WT & (wwll S SC}
The functions in the set B shall be called the "Branch Functions" and the functions in the set would be called the "Trunk Functions".
5

---

Published in Transactions on Machine Learning Research (02/2024)
The bound of in the above definitions abstracts out the model of the branch and the trunk functions being nets having a layer of bounded activation functions in their output layer while they can have any other activation (like ReLU) in the previous layers.
Definition 3. DeepONets
Given the function classes in Definition 2, we define the corresponding class of ONepONets as,
(hws,.w ((w),w.) Rdd R2 (s. p) in hw (s. p) (Ber, (8); T (P)) E TR. Bw, eB T Tw, TT}
Further, note that A0 > O 3 "0-cover" of this function space Ho such that, v Vew,,, E eH ale , Ho
s.t [w - w. & and |,o, - w s and w.g and w being elements of the 2 covering space of the set of branch and trunk weights respectively.
It's easy to see how the above definition of H includes functions representable by the architecture given in Figure 1. Now we recall the following result about neural nets from (Bubeck & Sellke, 2023).
Lemma 3.1. Let fw a neural network of depth D, mapping into R with the vector of parameters being w e and all the paramelers being bounded in magnilude by W ie the set of neural networks porametrized by E [-W, W]P Let Q be the marimum number of matriz or bias terms that are tied to single porameter where W, is the malrix
Wa for some E [p]. Corresponding to il we define, B(w)= = [j{[D] max (1WWlllop
in the -lyer of the net.
Let x e Rdd such that |z||| s R, and w1, W2 E Rp such that B (wi), B (wx2) s B. Then one has
1s (x) fws (a)) B²QR wi w2ll
Moreover for any W. with W 1, one has, B(w) (W VTQ)D
In light of the above, define J as follows,
Definition 4 (Defining J). Given any two valid weight. vectors w and wy for a "branch function" B assume to have the following inequolity for some fired. J
sup Bw, (s) - Bww (s) s J- J-1lw1 w2
DED
And similarry for the trunk functions over the compact domain U.
One can see that the above inequality is easy satisfy if the space of inputs the branch or the trunk is bounded. Thus invocation of this incquality implicitly constraints the support of the data distribution.
An Example of DeepONet Loss To put the above definitions in context, let us consider an explicit example of using a DeepONet solve the forced pendulum PDE, R² d(y.)) -inkk-sinny)) f(t) R² at dr
different forcings at a given initial condition. Corresponding to this, the training data DeepONet would need would be 3-tuples of the form, (#B(S),FT.b) where (J') is discretization of a forcing function f onto a grid of "sensor points", y R is the angular position of the pendulum at time t = Fr for f. Typically is a standard O.D.E. solver's approximate solution. It's clear that here being an angle is bounded, as was the setup in Definition 1.
Referring Figure 1, we note that If()). di point discrotization of a forcing function f would be the input to the branch net, p or d2 R, would be the trunk input iee the time instant where we have the location (y) of the pendulum. Then the output of the architecture in Figure 1 is the evaluation of the following inner-product, Rd x Rd ( + DeepONet(5,, p) == (Branch-Net(3 Trunk-Net(p)). And given such data above, the l empirical loss would be, L 2n ΣL (y: (eepONe((zn()),xr))) This

---

Published in Transactions on Machine Learning Research (02/2024)
empirical loss when minimized would yield a trained architecture of form as in Figure 1, which when queried for new forcing functions and time instants would yield accurate estimates of the corresponding pendulum locations.
4
The Main Theorem
In the setup of the definitions given above, now we can state our main result as follows,
Theorem 4.1. Vo 10 and an arbitrury posilive constand > 0 and for C 1 (from Definition 2), if with probability at least 1 1 with respect to the sampling of the data {(Vi, (s. i == n.....n. 3 hws. 6 H. s.t
Vi bws, wa (s., so² (1+ (B+2
then,
22
9 nt.
(2)
288 B
In 4min(dg.dd] dstdr
-Wevdds) dn -Wrydd) dr +2
In( T-8
and g(s. = E[y (s. ().. where 22:= Ei E (y: - (0ui-9(s1) (
The proof of the above can be seen in Section 6. Note that the lowerbound proven here for q is a necessary (and not a sufficient) condition for the required high probability (over data sampling) of the existence of a DeepONet with empirical risk below the threshold given above. For further insight, we now specialize our Theorem 4.1 to using
which then encompasses the case that we shall do experiments with, that of having DeepONets whose branch and trunk nets end in a sigmoid gate. Also, towards the following weakened bound for more intuitive presentation parameter vector for the branch and the trunk net and define dB dr as the upperbound on the total number of parameters in the predictor being trained.
common upperbound of W the 2- 2-norm of the
Theorem 4.2. (Lowerbounds for DeepONets Whose Branch and Trunk End in Sigmoid Gates) Let C 1 and constants W and s be bounds on the norms of the weights of the branch and the trunk and the total number of trainable parameters respectively. Then Vo e (0, 1), and any arbitrary positive constant e 0 if with probability at least 1 with respect to the sampling of the data {( (8 p.)) . 3 wa, H s.
Σ (1) ww (8),)))) 50 (I+J.(++2)
n
then,
q 2 n
(3)
288 B In on (4-iinfda (dl dr W WW))") In ln((22))
where 0= + E B(( E
g (s), and g(s.p) = E [sy (s,p)) and if the branch net has o- o-fractioo of the training parameters then a' a In I + 1-a In I-a'
2
2
To interpret the above theorem consider a sequence of DeepONet training being done for fixed training data (and hence a fixed n) and on different architectures but having the same weight bound and the same number of parameters and the same q, the common output dimension of the branch and the trunk functions. Now we how the above theorem reveals largeness requirement for DeepONets that if there has to

---

Published in Transactions on Machine Learning Research (02/2024)
exist an architecture which get the training error below certain label-noise dependent threshold then necessarily the branch/ ttunnk output dimension q has to be
Later, in Section 7, we shall conduct an experimental study motivated by the above and reveal something more than what the above theorem guarantecs. We will see that over a sequence of training being donc on different DeepONet architectures (and fixed PDE) having nearly the same number of parameters, one can get monotonic improvement in performance upon increasing training data size increase in q s.t yn is nearly constant. We also show that slightly smaller rate of growth for n for the same sequence of qs would break this monotonicity. Thus reveals a "scaling law" for DeepONets which is not yet within the ambit of our theoretical analysis.
is accompanied by an
5 Lemmas Towards Proving Theorem 4.1
Lemma 5.1. For any space X with Euclidean metric, denote (0, the covering number of it at scale 0. Further recall from Definition 2, that dg and dr the total number of parometers in any function in the sets B and 'T respectively. Let WB TRda, WT E RRd and WH. WB WT denote the sets of allowed weights of B, T, and H (Definition 3), respectively. Then the following three bounds hold for any A > 0,
dg
dr
2W 2WBVd dB
2W dr
N(0,WB)
N(O,
0
0
NO (w, N(0/2, NN(02,
The proof of the above Lemma is given in Section 5.1.1
Lemma 5.2. We recall the defivition of H from Definition 3, B as given in Desontion 1 & from Defoniton 4. Further for any and any training data of the form as given in Theorem 4.1, we denote the corresponding empirical risk as, I((h) E Ei (()). Then, v0 we have,
R( RR ;) .)) R RR (w),. qC.JC (B 29(²)
and w and w be s.l. wr, wo. o 5 and |w, w g 5
Thus we see that is quantifiable as to how much the increment in the empirical risk when for a given training data a DeepONet replaced by another with weights within a distance from the original and that this increment is parametric in The proof of the above lemma is given in Scction 5.1.2.
Lemma 5.3. We recall the definition of Ho from Definition 3; dB, dr, WB, WT, C 8 from Defintion 2 and as given in Defintion 1. Then A > 0, and for 2, == ys - (s.,,pp) P:);
PP 3 h(w, m.
Hol
- i-l
hw m. (8,, Pi) 24 2 1
1
22(d
do (WrVdr) dr exp
2n02
g(da+dr)
84. ( 922)
-
2exp
83. B2, 92 C1
The proof of the above lemma given in Section 5.1.3

---

Published in Transactions on Machine Learning Research (02/2024)
Lemma 5.4. We continue in the same setup in the previous lemma and further recall the definition of in Theorem 4.1. Then vO O
P 3 Awrgor wr H Σ (y: hwr, (Bt.p:))² s 0² 00² 2exp
nd²
+1P 3 huwg.ar H
1
Σ h (8;
(s
0
288
72
The above lemma reveals intimate connection between the empirical error of DeepONets and the corre- lation of its output with label noise. The proof of the above lemma is given in Section 5.1.4
5.1 Proofs of the Lemmas
In the following sections we give the proofs of the above listed lemmas.
5.1.1
Proof of Lemma 5.1
Proof. The first. two equations are standard results, Example 27.1 of (Shalev-Shwartz Ben-David, 2014)
Further define d(x,y) || yll2. Then, let S. R be witness for N(0/2, that is, for all ws WB. there is some s S such that d(w,, s) s 0/2. Similarly, let P c Rd be witness for N(0/2, Then for all wy E W3, w. Wr, there exist a corresponding cover point S S and p P. And since (w,, wi) WHz
d(( (ws. wi), (s,p) s d( (ws, wr). ; (s, wi)) d((s,
(by triangle inequality)
= d( wp. s) d(w,,p) (under d (2-mo
s (by definition of S and P)
Hence, Sx T is an f-cover of Wy.
5.1.2 Proof of Lemma 5.2
Proof. Given an 8 0 and a h(mm..).) H. let w and w, be s.t. Iwo - 1 s and ww w
Then from the definition of in Definition 4, the following inequalities hold,
sup Bung (8) Bwg (8) J.z and sup Tun (p) Twn (p)
1
J. 0
Further, we can simplify follows, for any valid (s,p) input the function Mwn. similarly for hw, ".
(Bur: Twi)) and
Bw, (8), (p)) (wu,g (8), (())l
Buw, (s), Tw, (p)) - (uw, (s), Tw, (p)) (s), Tw,, (p)) (p), Bwo (s) Bu 1f (8)
Buw. (s), (p)
Buo (s), (p) Two (p) +
(Tue [0]l/+[Tug]
To upperbound the above recall (a) the definition of C from Definitions 2 and (b) that for any two q- g-dimensional vectors a and b we have, (a, b)l s Σx_ |ai|[ba] s (max....... (b:l) Eg lail. Thus we have,
(ssp)),
(Bur, (s), (p)) Bu (s), Tw,, (p) 52. q. c)
( J
(4)
((s p),
Nwn. - hes w, (s.p) s C- J8
(5)

---

Published in Transactions on Machine Learning Research (02/2024)
Define, 11,1 == y/ h(w,, g.w.. 1. g)), and r2 == (yr 1 (B..
Now,
r1, 22 (1e a w, (): h(w,, w.) (8:, 2y/i (B.. - liw,, u. (s...))))
(mng (m (ot. N( 3.m. m. (8) (8) nim g.". ..m.)))s)))))) +.. B.B.
wo) Mym,, g. , () (s. 9. JJ B. C.
q C. J0. ( (Me hw & u m ()(
Averaging the above over all training data we get,
1 1 1
1 q. C. J0. ((: + h(w, ". (),, B
(6)
Using Cauchy-Schwarz over the inner-product in the definition of h, get,
((s / VCC s q. C² = ((0 h(m " sm (8). .Q.C²
(7)
Substituting the above into equation 6 and invoking the definition of R
R( g, ,,,g)) R( (((
Σ q. C- J0 (hi + new w. g) (s.,)))) pi)) B) 11 1=1
1
R ()(ws)))))
(q.C.
C. J0 B) C--C- J0) (2q. C²)
< R( R(((w...)) cCJ0 (B +2(C²
The above what we out prove.
5.1.3 Proof of Lemma 5.3
Proof. Recall that for each data i, we had defined the random variable, =; y: 9 (s:.p.). Since g(s.p)
E[y (s,p)], we can note that [[2]] = O. Further,
2f (v: (s:, s vi 2. vi p(st,p)) ((s..p.) 2
(8)
Recall from equation that | i. ", ()..)))))) C2
For each data i, we further define the random variable, Yo, == (hrma,g "m. (s:.)) (s....) p:) B[h ,, 1)
Now note that,
10

---

Published in Transactions on Machine Learning Research (02/2024)
B[Yo,,] E (h(w,,g u. .. (si.Pi) P.) -E[h(w, g. g.w. )))
E (mw,.g.u. (st, y/
E8 hew g. , (.). ....),.
Next, use the tower property of conditional expectation to expand the first term,
E /. 4.4 (s,, P:) Y/ E El Bl ..))) (s p.) (st. B (l(,, ,, (.).P)) Eb p:)]]
E he m. (s. g (s..p.)
Substituting this back into the previous equation, we get,
Yo, =0
Further,
[Yo h(w, g, ,. (81,p)) 18[ 10 g. w.,g 111 -2al (( w. ()
B Bo w. gill) 2B
C2. B.
Applying Hoeffding's inequality on Yo,s, we will get,
2nt?
P
( m. m..g) (st.P.) E[h(m, mm (m..g))aa) 2t s exp
(9)
(88 B qC2)2
We choose t= 0 to get,
(s.. P Σ (hiws w. (s:. B[h , (..g)) g)1)2s)
0
2 exp
2n02
84 (8t. (B.(BBg))
n
We define two events,
Theorem 5.5. (Hoeffding's 's inequality). where be oc, Then
Let Zi...., Zn be independent bounded random variables with Z, [a,b] for all i,
1
2nt
P
Σ (2.
B(2,)) 2 s exp
(b ((-a)²² a)
and
P
1 n
Σ (2, B[2,)) s s-
supp s
2nt²
(b a)
[-]
all 2
11

---

Published in Transactions on Machine Learning Research (02/2024)
- Σ 2 2
1
1
Eg
& Es == {3 hews, , 8) E Ho
Σ
El
E[ (m, _,, 12 2
8- gC²
n2 2=1
Recalling the bound the h function have, pC= [ 4 ,. e)ll E [0, 1], we have that if Eğ happens then for such sample of (z, n,...,n),
A Vh
10
E Ho.
1
23
[ [m m
)]
n Σ 2
Σ
[m & .m).))a
8 G0
71
q q.C²
n gc
i-1
Hence Eg
E and hence IP (Ea) s IP (Eg) ie
n
0 (i , g)) 2
0
P 3 3
how, a , e H
Σ
B
8) SP
zx
2
(10)
12
8 8.9.C2
Recalling that (/2/ s 2B), by Hoeffding's inequality we have,
0
-n0² 88. B2. B2.92 92 C1
P
2/
s 22exp
(11)
q
C2
Now we define three events E,, Eg and Eg follows,
1 - Σ He, n
Mw w. (BL. (Bt, B[h w. )1)))) 0 8
E, y h(w, g. w)
[[h Ffb g) 1 8
Eg
h(w, w E Ho:
B Σ
n 1=1
n
(w q m g) (s.. 2, S 4 2
Eg hew, m. g) E Ho.
Σ
ml
Observe that, if E, and Eg hold then Eg will also hold.
Hence,
P(E, n E&) s P(E))
P(E,) s PPEE)) P(E))
Thus, we can invoke equations 10 and 11 to get,
1
h e _0 g)) (s.. 2 0
3h g. g E Ho!
Σ
n
12

---

Published in Transactions on Machine Learning Research (02/2024)
< P
3 v.
w
)
Ho
lw v. 4)) (8;, p:) - E Bl , {)))
24
2
+
Σ
11
2
8 q C
i=1
-
P
(10 3 .. (s. (s..P.) E (wm, .. 1)))2
2 8
+ 2 exp
83 B2 q2 Ca
JeHl
-ng?
s
Σ
P
h (w0 3 2. ( ((..,.) p.)- 12 (w 1. 1)) Σ
+ 2 exp
hi a... y 1.
8
83 B² q.. C4
Hence,
1
w .)) (s:. 21 2 0
E Hol
- Σ
h
41
n i=
d2(d=+dT)
de
(WWydr) dr
2n0²
$ ((dx+dr)
(WBVdB)
2. 2xx
81. (B. q
-n 83. B² g2. C4
+ exp
And the above what we set out to prove.
5.1.4 Proof of Lemma 5.4
Proof. Recall the definition of 24 from the previous proof and and from the assumptions in Theorem 4.2 we have, E/ E. [=+]] =0² Recalling that 2; E [-2B, 2B] and they are i.i.d. we can invoke Hoeffding's Lemma 8
5.5 (with t= 6. bb 5. 1=2Baa -BB) to get,
1 n
0)
n0²
P
2
0²
exp
(12)
6
288 B2
Further note that, z g Pt) is i.i.d with mean 0 since [[2;] (8i, P:)] = 0 and Applying Hoeffding's inequality again,
Izc (s)) 2B
9 s exp 1 Σ 2g (s: p.) s -
ndf2
(13)
288B2
Given a ww,, H. we define the following vector random variables,
Z =
Vn (21, 22,
(14)
(81, (81. (82. .. px))
(15)
F
(hrw,, (81,P1) ,, (82.p2) , Nwn wr (8n Pa))
(16)
Vn
13

---

Published in Transactions on Machine Learning Research (02/2024)
Note that,
(g (r), (sn, Vn (21,..., 2n) vn ww (81, wu (8n, Pn)) 11? -- Vn
(17)
Recalling that 2; g (8,,, Pi) and the definition of the empirical risk of the predictor, W(wws.w. = 1 Σi (3/i ww,. (st. p;))², we realize that,
1z G- F1 G-FFI R( R(w
Suppose, |2| -² and (Z, G) - Then we have,
1Z+G- (G +2((3)) 2) -
20² 0 0
0 F(
2 2(2 F) 20
If further have, | G F F & 0²
8 then we have from above, (F,
Motivated by the above, we define the following 4 events, namely E, i 1, 1....,, 4
0 Eg (Z, 2 2- E3 {3 hwr EH E 0² -2 & E, ( hwm, eH 0 Tj E :- 11 o²
0
Thus our above argument can be summarized to say that if the events E,, Eg and Eş hold, then E, will also hold. This we can write as, P(E₁ n Eg n Eg) s P(E.). This implies, P (E) 2 1 I-(((E, n E₂ n Eg) But, by union bounding, P (E E E s (EE) + PEE)) + P(E)) s 3 (P((E)) + P(E2) + P(E))). Hence combining we have, ((E4) 2
2(E2) P(E3))
From equations 12 and 13 obtaín, that, (1 (1-PP (EE)) s exp (-
nd² 28882 ) and similarly for (1 (1-P(Ez)).
Thus substituting in above we get,
ng2
P(Ea) s 2eep
+P(EE,)
288
Thus we have proven what had out to prove,
6 Proof of the Theorem 4.1
A careful study of the proof of Lemma 5.4 would reveal that it can well be invoked Ha.
And by doing we get.
14

---

Published in Transactions on Machine Learning Research (02/2024)
12
1e ". (s). pi)) s so²-0)
P
3l(uug & (.3)) E Ho
-
-
1/5
i=1
n02
(8: Ho Σ h(m g. m) (s:, Pi))) 2s 3 9)
22exp
IP
288
B2
Using Lemma
5.3,
nl²
22(dg+dr)+1
(WBVd8) dn (Wrrdrr) dr -exp 81. (B g22)3
2n0²
exxp
288 B2
Olds+dr)
nd?
+ 2 exp
83. (B q. C2)2
(18)
Invoking Lemma 5.2 at 0= fo and recalling that the q 1 (by definition) we have,
R( R(h
s (w((ss J. (B+2-C3)
(B
(19)
With respect to random sampling of the training data we define two events E (corresponding to the function class H) and E2 (corresponding to the 24 -cover of H),
E == (ww.w)))) w.) H R ((h(ws, s
(B+2.
E2 ==
33
wi E Ha R( (R((Wws.e (3))))
Thus if Ej is true, we can invoke the above inequality get,
R(h 2 n 252
)) S R
(h( +C. (..
$ 0
Thus we observe that, E₁ = Eg and thus P(E,) P(Ez) we can invoke equation 18 to get,
P 3 h(ws w. E H
Σ :=1
(ys
hwn, wa (st. s0² - -I
12( ((
< P
3 1w.. 347
U1 h(w,, zيي w. size (8i.
s s
we
71
R2 Z5z 10 2002
nc²
2d(dn=dr)+l
(WaVds) da (WoVdr) dr
2nc
$ 2eexp
288 B qd)
(dn+dr)
eexp
84. (B g²
(20)
n
+ eexp
83. B go Ca
Hence if the required probability has to be at least 1 it's necessary that have,
15

---

Published in Transactions on Machine Learning Research (02/2024)
ne 288 B² .pt)
22(d24dr)=1
(Wovds) da (WrVar)) exp (dæ+dr)
2n
(1 (1-6) s2exp
84. B² 9o C4
n
eexp
B2. 98
Note that, c5.B a necessary condition to satisfy the above inequality is obtained by weakening the above inequality by replacing all the three exponentials in there with the smallest of them,
ne B4-BagghE Further recalling that g,C 2 we have, 288- Ba. Thus
2ne
-E
)(222)
)
4q² datdy
(WWadds) do (WrrVrr) dr +2
1 1-8 exp
288 13 gz)
And the necessary condition above leads to the lower bound,
2²
q 2 n
(21)
288
B
4g²
dgtdr
(Wavda) du (WrVdr) dr + +2) + In 1-8
The claimed lower bound in the theorem statement follows by further weakening the inequality above recalling that by definition have, 9 min(dp,dr)
7 The Experiment Set-up
In this section, we shall demonstrate that at a fixed number of total parameters, increasing the output dimension(4) and increasing the training data as q² can cause a monotonic decrease the training error of DeepONet.
The adlvetiin-diifnsion--eadd plays crucial role in modeling various physical, chemical, and biological processes and that shall be example for the experimental studies. More specifically, given function f this PDE is specified as follows,
P.D.F. (Rahaman al., 2022) (referred the ADR PDE from here on)
du - D 8u Øt dx² with sero initial and boundary conditions, and for our preliminary experiments we shall use D= 0.01 as the diffusion coefficient, and & 0.01 as the reaction rate, We use DeepONets to learn the operator mapping from f()) to the PDE solution (xx,t). In this case the operator Go will map the source terms fax to the PDE solution u(x,1). Given choice of m sensor points, we shall denote a discretization of f onto the sensor points as the vector f & R Recalling the DeepONet operator loss, realize that minimizing that is trying to induce, (f( t)) G(((ztt)..
ku²+ fx),, xE [0, [0.1]
(22)
For sampling f we have considered the Gaussian random field(GRF) distribution. Here we have used the mean-zero GRF, f ~ & (0, ky (21, where the covariance kernel by (21, exp =211² /21²) is the radial-basis function (RBF) kernel with length-scale parameter > O. For our experiments we have taken = 10-3. After sampling f from the chosen function spaces, we solve the PDE by second-order finite difference method to obtain the reference ions.
For n training data samples, the E2 empirical loss being minimized is, DeeepoNet = 1 Σn, (y: Go (f+) (p:))² where P: is a randomly sampled point in the (x,t) space and ys is the approximate PDE solution p corresponding to fü which recall was obtained from conventional solver.
16

---

Published in Transactions on Machine Learning Research (02/2024)
7.1 Implementations & Results
We created 10 DeepOnet models in each experimental setting such that each model has depth of 5 and width varying between 24 and 50 for each layer while keeping the total number of training parameters approximately equal for each of those 10 models. For each case the branch input dimension is 40(i.e number of sensor points), and trunk input dimension is 2. The smallest number of training data (n) we use is 10* and twice we make a choice of 10 different. (q,n) values parameterizing the learning setups, once keeping the ratio i approximately constant and then holding the ratio were trained by the stochastic Adam optimizer at its default parameters.
3 almost fixed. All the DeepONet models n
The code for this experiment can be found in our GitHub repository (link).
Experiments in the fixed S setting. In this setting, the value was varied from to 50, in increments of 5. We have laken the starting value of n as 10 In Figure we have plotted the training loss dynamics for these 10 models being trained over 120 epochs.
Experiments in the fixed 3 setting. We repeat the above experiment but while appproximately fixing the value of 9 The corresponding plots are shown in Figure 3. nd
Mudld (2=3)m=21101 0.00
Mould 1 12--2.n-2100" --000
107
Model 1--10-000000. -0.05)
10
Medel 21 21 5-8.00
sedd 18--2.---8--00----000 3
Model 4 (2-20..1-10 105 -e-
Model 4 4pp:200 5.0.00
Hadel 05m25,0m2 1101 3+ 0.800
101
Miviel 105- -nmm
Nodel G 30-30, 10" - 6.05)
Model 6 6pmm20
=0.0
Nodel (2-23,11-6 12" (0-005))
L000
Model 25 2-00
Tresing
Model 10" (0000))
Model 8 1p-40 252222
(0-(5,n-0 -000))))
Modal o m45-4 3020000 0-0.0
Madel 10 10-
Mide (0-30,n- 353153
2214
10-2
39
200
230
1/5
400
60
300
120
Epocha
Epochs
Figure 2: Training Loss Epoch in fixed setting Figure 3: Training Loss Epoch in fixed 2 setting
Va
Further experiments of the above kinds al other values of D and k, for orders of magnitude above and below what's considered here, can be seen in Appendix A. In Appendix A.1 we shall show that the emergence of a scaling law as demonstrated for fixed g experiments persists even for fixed n be expected the amount of available data increases for each model considered. A summary table of all parameters studied for the ADR PDE can be seen in Section A.2.
y experiments as is
We draw two primary conclusions from the above results. Firstly, from Figure 2, we can observe that if and n increase at a fixed - then performance increases almost monotonically. Secondly, from the Figure 3 it is clearly visible that the previous monotonicity is breaking that is the rate of increase of data size in the later experiment was not sufficient to leverage the increase in the output dimension size of the branch and the trunk as was happening in the first figure.
v
8 Discussion
Our key result Theorem 4.1 shows that a certain data size dependent largeness of is needed if there has to exist a bounded weight DeepONet at that which can have their empirical error below the label noise threshold. From our experiments, we have shown that there some non-trivial range of q (the common output dimension) along which empirical risk improves with g for a fixed model size if the amount of training data is scaled quadratically with q. We envisage that trying to prove this "scaling law" can be a very interesting direction for future exploration in theory.
17

---

Published in Transactions on Machine Learning Research (02/2024)
Secondly, note that result hasn't yet fully exploited the structure of the neural nets used in the branch and the trunk. Also, it would be interesting to understand how to tune the argument specifically for the different variations of this architecture (Kontolati et al., 2023), (Bonev et al., 2023) that are getting deployod, Lastly, we note that our result is currently agnostic to the PDE being attempted to be solved. There is a lantalizing possibility, that methods this proof could be extended to derive bounds which can distinguish PDEs that are significantly hard for operator learning.
References
J. Almeida, P. R. B. Rocha, Allan Morcira De Carvalho, and A. C. Nogucira. A coupled variational encoder- decoder-deeponet surrogate model for the rayleigh-benard convection problem. null, 2022. doi: null.
Genming Bai, Ujjwal Koley, Siddhartha Mishra, and Roberto Molinaro. Physics informed neural networks (pinns) for approximating nonlinear dispersive pdes. arXiv preprint wxXi:2104.05584, 2021.
Peter Bartlett, Dylan J Foster, and Matus J Telgarsky. penetrally-normaliie margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Mikhail Belkin, Daniel J Hsu, and Partha Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. Advances in neural information processing systems, 31, 2018a.
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning, pp. 541-549. PMLR, 2018b.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849- 15854, 2019.
Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar. Spherical fourier neural operators: Learning stable dynamics on the sphere. arXiv preprint Stff::2366008833 2023.
Susanne C. Brenner and Carsten Carstensen. Finite Element Methods, nov 15 2004. URL //tt:://dx.. doi. 0rg/10.1002/0470001112 ecm003.
Sébastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. Journal of the ACM, 70 (2):1 18, 2023.
Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911 917, 1995a.
Tianping Chen and Hong Chen. Universal approximation to nonlincar operators by noural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911-99)) 1995b. doi: 10. 10.1109/72.392253.
A. Choubineh, Jie Chen, David A. Wood, Frans Coenen, and Fei Ma. Fourier neural operator for fluid flow in small-shape 2d simulated porous media datasct. Algorithms, 2023. doi: 10.3390 10.3390/a16010024 a16010024
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics-informed informed neural networks: Where we are and what's next. Journal of Scientific Computing, 92(3):88, 2022.
Zeyu Deng, Abla Kammoun, and Christos Thrampoulidis. A model of double descent for high nigh-dimensional binary linear classification. Information and Inference: A Journal of the IMA, 11(2):435 495, 2022.
MWMG Dissanayake and Nhan Thinn. Neural-network-basse approximations for solving partial dif- ferential equations. communicaatinss in Numerical Melhods in Engineering, 10(3):195-200 201, 1994.
18

---

Published in Transactions on Machine Learning Research (02/2024)
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural net- works. In Conference On Learning Theory, pp. 297-299. PMLRR 2018.
Vignesh Gopakumar, S. Pamela, L. Zanisi, Zong- Yi Li, Anima Anandkumar, and Mast Team. Fourier neural operator for plasma modelling. null, 2023. doi: null.
Pulkit Gopalani and Anirbit Mukherjee. Investigating the Role of Overparameterization While Solving the Pendulum with DeepONets. In The Symbiosis of Deep Learning and Differential Equations, NewrIPS Workshop, 2021. URL sestps://pperreee http://ppaeeeieeeeeeettrrri/aiea wtt.:///oooomm//////tttt/0000000
Pulkit Gopalani, Sayar Karmakar, and Anirbit Mukherjee. Capacity Bounds for the DeepONet Method of Solving Differential Equations. 2022. URL /ttpps:/doo..org/ 10w.00.555000r..... 2205. 11359.
S. Goswami, Katiana Kontolati, M. Shields, and G. Karniadakis. Deep transfer learning for partial differential equations under conditional shift with deeponct. arXiv.org, 2022. doi: 10.48550/ //////xxxi22201
Patrik Hadorn. Shift-deepone:: Extending deep operator networks for discontinuous output function sunctonnpattris hadorn. null, 2022. doi: mull.
Yiu-Chung Hon and XZ Mao. An efficient numerical scheme for burgers' equation. Applied Mathematics and Computation, 95(1):37-50 1998.
Ameya D Jagtap and George E Karniadakis. Extended physics-informed neural networks (xpinns): A gen- eralized space-time domain decomposition based deep learning framework for nonlinear partial differential equations. In AAAI Spring Symposium: MLPS, pp. 2002-2041, 2021.
Ameya D Jagtap, Ehsan Kharazmi, and George Em Karniadakis. Conservative physics- neural net- works on discrete domains for conservation laws: Applications to forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 365:113028, 2020.
Williamson Johnny, Hatzinakis Brigido, Marcelo Ladeira, and Joao Carlos Felix Souza. Fourier neural operator for image classification. Iberian Conference on Information Systems and Technologies, 2022. doi: 10.28919/cistti49922 2w..4449......92202..1
George Em Karniadakis, Ioamnis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics- informed machine learning. Nature Reviews Physics, 3(6):422-44)) 2021.
Ganesh Ramachandra Kini and Christos Thrampoulidis. Analytic study of double descent in binary classi- fication: The impact of loss. In 2020 IEEE International Symposium on Information Theory (ISIT), pp. 2527-2532. IEEE, 2020.
Katiana Kontolati, Somdatta Goswami, George Em Karniadakis, and Michael D Shields. Learning in latent spaces improves the predictive accuracy of deep neural operators. arXiv preprint 2XXu:2304.07599 2023.
T. Kurth, Shashank Subramanian, P. Harrington, Jaideep Pathak, M. Mardani, D. Hall, Andrea Miele, K. Kashinath, and Anima Anandkumar. Fourcastnet: Accelerating global high-resolution weather fore- casting using adaptive fourier neural operators. Platform for Advanced Scientific Computing Conference, 2022. doi:
Isaac E Lagaris, Aristidis Likas, and Dimitrios Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE trunsactions on neural networks, 9(5):987 1000, 1998.
Isaac E Lagaris, Aristidis C Likas, and Dimitris G Papageorgion. Neural-network methods for boundary value problems with irregular boundaries. IEFE Transactions on Neural Networks, 11 1144)10011,104, 2000.
Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis, Error estimates for DeepONets: a deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6 (1), 03 2022a. ISSN 2398-4945. doi: 10093/maatm//tnae URL https: /ttp::///ooooo.o.../m 1199//mmatrrr tnac001. tnac001.
19

---

Published in Transactions on Machine Learning Research (02/2024)
Samuel Lanthaler, Siddhartha. Mishra, and George E Karniadakis. Error estimates for deeponets: A deep learning framework in infinite dimensions. Transactions of Mathemalics and Hts Applications, 6(1):tnac001, 2022b.
F. Lehmann, F. Gatti, M. Bertin, and D. Cloutcau. Fourier neural operator surrogate model to predict 3d seismic waves propagation. or oiv.orr, 2023. doi: 10.48550/arsa 23.500/arxi.2233 2ew..2334..22..
Jingling Li, Yanchao Sun, Jiahao Su, Taiji Suzuki, and Furong Huang- Understanding generalization in doep learning via tensor methods. In International Conference on Artificial Intelligence and Statistics, pp. 504 515. PMLR, 2020a.
Zhijie Li, Wenhui Peng, Zelong Yuan, and Jianchun Wang. Fourier neural operator approach to large eddy simulation of three-dimensional turbulence. Theoretical and Applied Mechanics Letters, 2022a. doi: 10.1016/j. 2022..00030203
Zongyi Li, Nikola B. Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhaltacharya, Andrew M. Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arviv: Learning, 2020b. doi: null.
Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. Cornell Universily arXiv, 2022b. doi: 10.48550/ arxiv. 0t..222270000000
Guang Lin, Christian Moya, and Zecheng Zhang. B-deeponet: An enhanced bayesian deeponet for solving noisy parametric pdes using accelerated replica exchange sgld. Journal of Compulational Physics, 2022. doi: (ttpi//..cpp.222.111111
Lizuo Liu and Wei Cai. Multiscale deeponet for nonlinear operators oscillatory function spaces for building seismic wave responses. arXiv.org, 2021. doi: null.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv: Learning, 2019. doi: null.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. Deepxde: A deep learning library for solving differential equations. SIAM review, 63(1):208 20888288 2021.
Zhiping Mao, Ameya D Jagtap, and George Fm Karniadakis. Physics-informed neural networks for high-
speed flows. Computer Methods in Applied Mechanics and Engineering, 360:112789) 2020.
Ramchandran Muthukumar and Jeremias Sulam. Sparsity-aware generalization theory for deep neural net- works. In The Thirly Sirth Annual Conference on Learning Theory, pp- 5311-5342 PMLR, 2023.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Esperiment, 2021 (12):124003, 2021.
Guofei Pang, Lu Lu, and George Em Karniadakis. fpinns: Fractional physics-informed neural networks. SIAM Journal on Scientific Computing, 41 (0)0))2666322222 2019.
Jacwan Park, Shashank Kushwaha, Junyan Hc, S. Koric, D. Abucidda, and I. Jasink. Sequential deep learning operator network (s-deeponet) for time-dependent loads. arXiv. org, 2023. doi: 10.48550/ (0.48550/arxiv... arxiv.23006 2306.08218.
Jaideep Pathak, Shashank Subramanian, P. Harrington, S. Raja, A. Chattopadhyay, M. Mardani, Thorsten Kurth, D. Hall, Zong-Yi Yi Li, K. Azizzddnnshheii, P. Hassanzadch, K. Kashinath, and Anima Anandkumar. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arxiv.org, 2022. doi: null.
20

---

Published in Transactions on Machine Learning Research (02/2024)
Muhammad Masudur Rahaman, Humaira Takia, Md Kamrul Hasan, Md Bellal Hossain, Shamim Mia, and Khokon Hossen. Application of advection diffusion equation for determination of contaminants in aqueous solution: A mathematical analysis. Applied Mathematics, 10(1):24 31, 2022.
Maziar Raissi and George Em Karniadakis. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics, 357:125-144 2018.
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: navier-stokes in- formed deep learning framework for assimilating flow visualization data. arXiv preprint arXiv:1808.00 049777 2018.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural nelworks: deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Bogdan Raonic, Roberto Molinaro, Tobias Rohner, Siddhartha Mishra, and Emmanuel de Bezenac. Convo- lutional neural operators. In ICLR 2023 Workshop on Physics for Machine Learning, 2023.
Deep Ray, Orazio Pinti, and Assad A Oberai. Deep learning and computational physics (lecture notes). arXiv preprinl Xrxi::301.0000
2023.
Tim De Ryck and Siddhartha Mishra. Generic bounds on the approximation error for physics-informed (and) operator learning. ArXiv, abs/2205. 11393, 2022.
Mark Sellke. On size-independent sample complexity of relu networks. arXio preprint arXiv:2306.01999, 2023.
Shai Shalev-Shwart and Shai Bev-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Lesley Tan and Liang Chen. Enhanced deeponet for modeling partial differential operators considering multiple input functions. arXiv.org, 2022. doi: mull.
Tapas Tripura and S. Chakraborty. Wavelet neural operator: a neural operator for parametric partial differential equations. arxiv.org. 2022. doi: 10.48550/arx 22150//axxiv.2000211909
Tapas Tripura, Abhilash Awasthi, Sitikantha Roy, and Souvik Chakraborty. A wavelet neural operator based clastography for localization and quantification of tumors. Comput. Methods Programs Biomed., 2023. doi: 10.1016/j.cmpb..023. 107443..
Wuzhe Xu, Yulong Lu, and Li Wang. Transfer learning enhanced deeponet for long-time prediction of evolution equations. arXiv.org, 2022. doi: 2203550/axxiv2222..00
Liu Yang, Xuhui Meng, and George Em Karniadakis. B-pinns: Bayesian physics-informed neural networks for forward and inverse pde problems with noisy data. Journal of Computational Physics, 425:109913, 2021.
Jiahao Zhang, Shiqi Zhang, and Guang Lin. Multiauto-deeponee::: multi-resolution autoencoder deeponet for nonlinear dimension reduction, uncertainty quantification and operator learning of forward and inverse stochastic problems. arXiv. org, 2022. doi: 10.48550/ 10..500/axxv.2244033199
21

---

Published in Transactions on Machine Learning Research (02/2024)
A Appendix
In this section extended the scope of our demonstration by revisiting the experiments detailed in Section 7.1 and redoing them at further more values of the diffusion coefficient (D) and the reaction rate (k), as specified in Equation 22. This time we go a couple of orders of magnilude above as well as below the value of D = chosen in Section 7.1.
We conducted four sets of experiments, at different common values for D and k, namely 1) 1 in Figure 4, 2) 0.1 in Figure 5, 3) 0.001 in Figure and 4) 0.0001 in Figure 7.
Our findings indicate that for any given pair of (D,k) value, at a fixed A setting, performance keeps increasing monotonically i. the best empirical loss obtained monotonically falls with increasing q. However, for the fixed - setting, this monotonicity breaks, particularly for higher values of D = k.
n
Thus the key insights about possible scaling law for DeepONets continues to hold as was motivated earlier in Section 7.1.
(g 5.n- 1-10f)
Modil 2 -04,6-31621
Macil lo 13 in 10f)
Nodil (q -3,n
i0 12%)
Model 6g =40 252902)
Hotel 1g 10-45,00 -101))
Maccel lo 50,00 100-1001
Noodie 0g 50.0 3535530
2214
2012
22
20
of
Figure 4: (D & k value as 1) Left: Training Loss vs Epoch in fixed Epoch in fixed o setting.
2 -
setting. Right: Training Loss vs
Nonwl 1-1)))
Widde 109
167
Hosel l0 10.0 qi Juf)
Moddd 4 51023)
lg 10")
Modd 4 -4, 252080
Morel 1a 45, at 104)
Modd l 14,45, 5028701
04gg
Up n- 100
107
banngla
22
2012
20-2
25
120
22
100
320
No oo
ofochh
Figure 5: (D & & value as 0.1) Left: Training Loss vs Epoch in fixed In setting. Right: Training Loss vs Epoch in fixed o setting.
22

---

Published in Transactions on Machine Learning Research (02/2024)
201)
3 3-3-- 107
207
107-
Modd 2 -2
Mocel lo 10,0 10f)
Nodel 40 252982
Model 6 2555531
moccal ta 30, 100 2011
Therang
20-4
Figure 6: (D & k value as 0.001) Left: Training Loss vs Epoch in fixed Epoch in fixed setting
v setting. Right: Training Loss vs
n3
io 5, 20f)
Medil (1/-10,n-310000000
Havrel oo 11. 1n1)
Mocel ia 40 am 64 184)
Model 4 m40
Noudal 0g 50.5- 3535539
Mozil ii 50 a- 100 20"1
101
Saasing Losa
Teaniig
2271
2014
10-1
/0
Figure 7: (D & & value as 0.0001) Left: Training Loss vs Epoch in fixed vs Epoch in fixed 2 setting
setting. Right: Training Loss
v
23

---

Published in Transactions on Machine Learning Research (02/2024)
A.1 Experiment at fixed + setting the ADR PDE
In here we conducted two sets of experiments, at q and settings more closely inspired by Theorem 4.2 i.c. we chose a set of nets of inercasing such that the size of the nets and o are almost fixed. We do nt experiments different common values for D and k, namely 0.0001 & in Figure 8.
Nodel 1 18200
64 - 11850)
Model 2 Ra 25331)
Node 2g m 603313
Mudel 0
12.0 240215)
Nodel (q 12,n 746235)
Leaa
10"2
20:1
Figure 8: (D & k value as 0.0001) Left: Training Loss Epoch in fixed 4 setting, (D & k value as 1)Right: Training Loss Epoch in fixed setting.
The above plots suggest that the monotonic performance improvement with increasing & as seen in all earlier experiments continue to hold even at the scaling as chosen here.
A.2 Table of Data for All Experiments the ADR PDE
A.2.1
Experiments where (and size of the operator net) kept nearly constant
# Trainable Parameters in the DeepONet
71, D = k
10
n1, D =
18112
6
11650
11650
18316
8
65511
65511
18520
10
249906
249906
18724
12
746215
746215
Table 1: The last columns correspond Figure 8
A.2.2 Experiments where A (and size of the operator net) are kept nearly constant
Trainable Parameters in the DeepONet
R,
A-
10
n. D
10"
D, k
13, A-
D.D-
10000
10000
10000
10000
10000
40000
40000
18568
90000
90000
90000
18719
640000
640000
640000
640000
640000
18714
45
810000
810000
870000
810000
810000
18760
50
1000000
1000000
1000000
1000000
1000000
Table 2: The column for D k= 10 corresponds to Figure 2 and the rest of the last columns, starting from the rightmost above, correspond to the left column of Figures 4, 5, 6, and 7
24

---

Published in Transactions on Machine Learning Research (02/2024)
A.2.3
Experiments where
(and size of the operator net) kept nearly constant
nf
T'aiinable Parameters the DeepONet
D k
10
=)=k
10
D= == 10
D k= 10"
==
18010
10000
10000
10000
10000
10000
18520
10
31623
31623
31623
31623
31623
18568
15
58000
58000
58000
58000
58000
18719
40
252982
262982
252982
252982
252982
18714
45
301870
301870
301870
301870
301870
18760
50
353533
353553
353553
353553
353553
Table 3: The column for D == 10-2 corresponds to Figure 3 and the rest the last columns, starting from the rightmost above, correspond the right. column of Figures 4, 5, 6, and
25


Reviews:
[{"review_id": "VCEFjmzDhh", "reviewer": "Reviewer_qWBR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe proofs in the paper is cleanly set out and reasonable to follow. Splitting Theorem 4.1 into four smaller steps makes it easier to identify how the relationship between $q$ and the dataset size is derived, and the splitting of events $E_i$ make this proof well structured. The paper also clearly lays out previous work and restates relevant theorems pertaining to the approximation theory of DeepONets. This paper is strong in completing its main objective of identifying how to connect the parameter $q$ with the dataset size $n$. \n\nWeaknesses:\nThe computational study in this paper is limited compared to the theoretical analysis. Whilst the authors present two settings, one for the case $q/\\sqrt{n}$ is fixed and one whilst $q/n^{2/3}$ is fixed, neither of these correspond to the rate that they derive in their analysis. Whilst, from what I gather, their computational evidence here is stronger than what they prove, for completeness the paper should have a small experiment for the rate they derive. The authors also only consider one test problem of a single PDE, with variations of the coefficients in the appendix. Whilst a computational study is not the focal point of the paper, it would be interesting to know if this monotonic convergence for $q/\\sqrt{n}$ occurs elsewhere, or if this is just relevant to the PDE the authors consider. Currently, it is unclear to me whether the authors are trying to convey a conjecture that the rate $q/\\sqrt{n}$ might be expected in practice generally or if this rate could be given from specific additional structure of the particular PDE investigated.\n\nThe paper also has many grammatical inconsistencies and mistakes. These would be easy to correct, but should be done to improve the readability and flow of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper analyses Deep Operator Networks and gives a bound which relates the choice of the $q$ parameter (the dimension of the latent space connecting the 'branch' and 'trunk' networks in a DeepONet ) to the size $n$ of the empirical data set. To my knowledge, the relationship derived is novel and of interest to the operator learning literature for PDE surrogate modelling. \n\nFrom my understanding, I see two main contributions of this paper. \n1) How to derive a bound which relates the choice of $q$ with the choice of $n$; and,\n2) Empirical evidence to show that monotonic convergence can be achieved by keeping $q/\\sqrt{n}$ constant with a fixed parameter budget on a certain PDE. \n\nFor the first point, deriving this relationship is the main contribution of the paper, with Lemma 5.1-5.4 necessary to pave the path for achieving the bound given in Theorem 4.1. As the authors note, there is a discrepancy between the bound they derive and what achieves monotonic convergence in their experiments \"Later, in Section 8, we shall conduct an experimental study motivated by the above and reveal something more than what the above theorem guarantees.\". From this, I see this paper as a first step towards quantifying an optimal scaling relationship between $q$ and $n$ in DeepONets, but there is still future work to possibly sharpen this result. Because of this, I find what I consider their second contribution to be rather interesting and understated compared to point 1) . Motivated by the relationship they derive, the authors conduct a numerical convergence test and identify monotonic convergence at a rate different to what they found in Theorem 4.1. Whilst welcomed, I view this result as separate to their theoretical analysis and possibly suggestive of a stronger result yet to be proved.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Suggested changes:\n\nI might be incorrect in my understanding here, but currently I see that the experiments run show a sharper rate than what is proven in the main theorem. If this is the case, I believe the text could be improved with the addition of a simple table of $q$ and $n$ comparing the values used in the experiments and the rate required from the theory to make this clear. I also think having a small example of the rate predicted by the theory, even if it is a weaker experiment, would help in the presentation of the results. Currently it is a little difficult to immediately understand how the current bound influences the choice of $q$ and $n$ in the experiments. \n\nI also recommend reordering some of the sections to improve the flow of the paper. I currently do not understand why the proofs of the lemmas 5.1-5.4 are listed in section 7 when section 5 states all of the lemmas and section 6 begins with:\n\"A careful study of the proof of Lemma 5.4 would reveal that it can as well be invoked on Hθ.\".\nFor the readability of the document, I would recommend just putting the proofs of the lemmas with their statements in section 5, especially seeing as the current section 6 is also proof based and follows on from the proofs of the lemmas. \n\nMinor comments:\n\nThe authors swap between PDE, PDEs and P.D.E in various sections (see Introduction and Section 8). This type of inconsistency is present throughout the document, for instance the authors swap between Physics Informed Neural Networks and Physics Inspired Neural Networks in an introductory paragraph. Making sure that abbreviations are consistent would aide the reader throughout the document. \n\nThere are several spelling/grammar mistakes throughout the current document which can be straightforwardly fixed, the ones I can easily garner are: \"oftenn\" and \"reviewed\" (instead of review I think) in the second paragraph of the introduction; Equation 1 and the equation at the beginning of Section 1 are missing punctuation; Definitions 1 and 3 are not complete sentences. I would request that the authors ensure that the sentences are complete, especially around the mathematics and definitions, to help the flow of the document.", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9WN29OhTfL", "reviewer": "Reviewer_pViX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\n- S1) Theory for operator learning has mainly focused on providing upper bounds on the network size in terms of the error tolerance. It is a very interesting topic to investigate whether corresponding lower bounds can also be proved, and how these depend on all parameters. This work could be a valuable contribution in this area. \n- S2) Although the evidence is rather limited, the scaling law that the experiment shows is very interesting and would be useful for practitioners. \n\n*Weaknesses*\n- W1) Just like the results of Bubeck and Sellke (2023), the lower bounds are only relevant in the case of noisy training data. In the context of DeepONets, noise in the training data can occur when physical measurements are used, or (more commonly) when the training data set is obtained through a numerical solver. Suppose one wants to guarantee an accuracy of $\\sigma^2/2$ in the error bound of Theorem 4.1 (bottom of page 6), then one can either (1) use a large enough DeepONet following Theorem 4.1 or (2) one can simply use a slightly better numerical solver such that the variance of the noise is $\\sigma^2/2$ rather than $\\sigma^2$. I believe that such considerations are important and put the proposed results in a different context, as results such as Theorem 2.1 do not depend on data noise. \n- W2) Notation throughout the paper is very inconsistent and therefore hard to follow. The true operator is denoted by both $\\mathcal{G}$ and $g$, the input function of the DeepONet is denoted by either $f$ or $s$, whereas $s$ is at the same time the total number of parameters in the DeepONet. The number of branch/trunk nets is denoted by both $p$ and $q$. \n- W3) There sometimes is a lack of mathematical rigour, which again makes the paper harder to follow. For instance, in Definition 1 it is unclear where the $(y_i, (s_i,p_i))$ is sampled from (which space, which distribution). In Section 1.1 one considers a compact domain $D$ yet in Definition 2 the trunk and branch nets are considered over $\\mathbb{R}^{d_i}$. It is unclear which setting is considered in the main results. In Definition 3, which vector norm is used? In Definition 4, over which set is the supremum taken? In the main theorem, $\\theta$ is defined in terms of $q$ but $q$ is undefined at that point and then only later a lower bound for $q$ is introduced. \n- W4) The authors state that the downside of Theorem 2.1 is that it doesn't make a connection between the lower bound and the size of the branch net, but also main theorem only gives a lower bound on the number of trunk nets, and not (at least not explicitly) on the size of the branch net. \n- W5) Finally, the main result is stated in a way that is very hard to interpret. In particular, there are three equations that link $q$, $n$ and $\\epsilon$: namely $\\theta \\leq \\epsilon/q^2$, the assumption on the training set size $n\\geq C_1 / \\theta^2$ and the final lower bound $q\\geq C_2 \\theta \\sqrt{n}$ (simplified). Rearranging these equations leads to $q^2 \\lesssim \\epsilon n$. This can either be interpreted as  (1) an upper bound on $q$ that needs to hold if the lower bound is valid (which is a bit weird as the goal is to obtain a *lower* bound), or (2) a lower bound for $\\epsilon$ (which is hopefully smaller than $(1+\\mathcal{C} J(B+2\\mathcal{C}^2)^{-1}$ as otherwise the assumption of the theorem is invalid). Finally, the total number of parameters of the DeepONet $d_B$ and $d_T$ (and their sum $s$) also depend on $q$. This means that the current upper bound on $q$ is more of an implicit relationship between various quantities, rather than an explicit upper bound. All of this makes it currently very hard to get a clean understanding of the meaning of the main result.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a lower bound on the size of DeepONets (meaning the number of trunk and branch nets), mainly as a function of the number of samples in the training data set $n$, with a resulting lower bound $\\Omega(\\sqrt{n})$. Inspiration is drawn from the work of Bubeck and Sellke (2023). The theory is supplemented by an experiment where DeepONets were trained for fixed ratios of $q/n^\\gamma$, suggesting a sort of scaling law.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be great if the authors can address or clarify the weaknesses mentioned above. Some additional remarks: \n- RC1) Regarding W1: I would suggest to clarify further that the obtained lower bounds are more of an artefact from the way the training data is constructed, rather than an intrinsic property of DeepONets or the underlying PDE (as was the case in Theorem 2.1). \n- RC2) Regarding W4: I would suggest to clarify how Theorem 4.1 addresses the second weakness mentioned after Theorem 2.1. \n- RC3) In Lemma 5.1 one uses the notation $N$ before defining it at the end of the lemma, which is very confusing. Similar in Lemma 5.2 with $\\hat{\\mathcal{R}}$. \n- RC4) In the beginning of page 10 (proof of Theorem 4.1) it is written that one requires that the probability is at least $1-\\delta$. In the proof of the main result of Bubeck and Sellke (2023) one requires that the probability is at most $\\delta$. Why is it different here?\n- RC5) Experiments could be repeated with different constants, and perhaps different PDEs.", "broader_impact_concerns": "/", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2fXEpqSgmS", "reviewer": "Reviewer_mYsr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths. Proving complexity lower bounds of DeepONet is an important question for operator learning while has less been explored. The paper proved an interesting  lower bound for learning a generic operator with DeepONet. To the best of my knowledge, this result seems to be new and relies on the recent result on robustness of neural networks by  Bubeck \\& Selker 2023. \n\n- Weaknesses. I do not have much to comment on the weakness, but I do have some concerns and questions on the statements of main results, which can lead to a major mistake/weakness of the paper. See the Questions below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the necessary lower bound on the size of DeepONet required to attain a given empirical training error. It is shown that the output dimension of the DeepONet needs to scale $O(\\sqrt{n})$ where $n$ denotes the number of training data when the training error is achieved below a label noise dependent threshold. Some numerical experiments are provided to verify the necessary growth of output dimension as a function of the size of training data to monotonically decrease the training error.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- One major concern I have is about the statements of the main results. For example, in Theorem 4.1, I am confused by which parameter is fixed (and which needs to be bounded). If I understand it correctly, given a fixed sample size $n$ the goal is to obtain a lower bound on the output dim $q$ as a function of $n$. However, the introduction of parameter $\\theta$  complicates the process and makes me really confused. In the preample of Theorem 4.1, $\\theta \\leq \\epsilon/q^2$ and $n \\geq 288B^2/\\theta^2 \\ln{4/(1-\\delta)}$. This means that $n \\gtrsim q^4/\\epsilon^2$ or equivalently $q \\lesssim n^{1/4}$. Would not that contradict with the lower bound requirement $q\\gtrsim O(n^{1/2})$? The author should elaborate and clarify this in the revision.\n\n- I strongly recommend the authors elaborate more on the Setup, especially the discussion on the training datasets. For readers who do not know the background of operator learning, the current description of training dataset is way too sketchy. If I understand it correctly, the data set should be the input and output functions. But what is the meaning of $y_i$? \n\n-In Theorem 2.1 and several other places, the authors mention that $p$ is the size of the trunk net. Is this the total number of neurons or just the width of the network? \n\nThe authors are recommended to submit a major revision of the paper addressing the comments above before it is considered being published in TMLR.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has been reviewed by 3 reviewers: two recommend "leaning accept" (qWBR and mYsr) and one recommends "learning reject" (reviewer pViX). It provides a lower bound o the size of DeepONets, mainly as a function of the number N of training data. Some numerical results have also been provided to verify experimentally the necessary growth of output as a function of N.

During the rebuttal phase, the paper was significantly improved but the authors erroneously did not make the answers visible to reviewer pViX. This has been corrected and there has been a lively discussion to clarify the remaining issues this reviewer had. 

I recommend acceptance of the paper subject to the final comments of reviewer pViX being taken into account.
Decision: Accept (poster)