Learning Deep Operator Networks: The Benefits of Over-Parameterization
Authors: Bhavesh Shrimali
Arindam Banerjee
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Operator Networks
Optimization
Neural Tangent Kernel

Abstract: Neural Operators that directly learn mappings between function spaces have received considerable recent attention. Deep Operator Networks (DeepONets), a popular recent class of neural operators have shown promising preliminary results in approximating solution operators of parametric differential equations. Despite the universal approximation guarantees, there is yet no optimization convergence guarantee for DeepONets based on gradient descent (GD). In this paper, we establish such guarantees and show that over-parameterization based on wide layers provably helps. In particular, we present two types of optimization convergence analysis: first, for smooth activations, we bound the spectral norm of the Hessian of DeepONets and use the bound to show geometric convergence of GD based on restricted strong convexity (RSC); and second, for ReLU activations, we show the neural tangent kernel (NTK) of DeepONets at initialization is positive definite, which can be used with the standard NTK analysis to imply geometric convergence. Further, we present empirical results on three canonical operator learning problems: Antiderivative, Diffusion-Reaction equation, and Burger’s equation, and show that wider DeepONets lead to lower training loss on all the problems, thereby supporting the theoretical results

Under review as a conference paper at ICLR 2023
LEARNING DEEP OPERATOR NETWORKS: THE BENEFITS OF OVER-PARAMETERRIATIOO
Anonymous authors Paper under double-blind review
ABSTRACT
Neural Operators that directly learn mappings between function spaces have re- ceived considerable recent attention. Deep Operator Networks (DeepONets) (Lu et al., 2021), a popular recent class of operator networks have shown promising preliminary results in approximating solution operators of parametric partial dif- ferential equations. Despite the universal approximation guarantees (Lu et al., 2021; Chen & Chen, 1995) there is yet no optimization convergence guarantee for DeepONets based on gradient descent (GD). In this paper, we establish such guar- antees and show that over-parameterization based on wide layers provably helps. In particular, we present two types of optimization convergence analysis: first, for smooth activations, use the bound to show geometric convergence of GD based on restricted strong convexity (RSC); and second, for ReLU activations, we show the neural tangent kernel (NTK) of DeepONets at initialization is positive definite, which can be used with the standard NTK analysis to imply geometric convergence. Further, we present empirical results on three canonical operator learning problems: Antideriva- tive, Diffusion- -eaction equation, and Burger's equation, and show that wider DeepONets lead to lower training loss on all the problems, thereby supporting the theoretical results.
bound the spectral norm of the Hessian of DeepONets and
INTRODUCTION
Replicating the success of Deep Learning in scientific computing such as developing Neural PDE solvers, constructing surrogate models and developing hybrid numerical solvers has recently captured interest of the broader scientific community. Neural Operators (Lii et al., 2021a;b) and Deep Operator Networks (DeepONets) (Lu et al., 2021; Wang et al., 2021) encompass two recent approaches aimed learning mappings between function spaces. Contrary to a classical supervised learning setup which aims at learning mappings between two finite-dimensional vector spaces, these neural operators/operator networks aim to learn mappings between infimite-dimensionall function spaces. The key underlying idea in both the approaches is to parameterize the solution operator neural network and proceed with learning as in a standard supervised learning setup. Since a neural operator directly learns the mapping between the input and output function spaces, it is natural choice for learning solution operators of parametric PDE's where the PDE solution needs to be inferred for multiple instances of these "input parameters" or in the case of inverse problems when the forward problem needs to be solved multiple times to optimize given functional. While there exist results on the approximation properties and convergence of DeepONets; see, e.g., (Deng et al., 2021) for convergence analysis of DeepONets vis-a-vis their approximation guarantees- for the advection- -iffuuiion equation, there do not exist any optimization results on when and why GD converges during the optimization of the DeepONet loss.
deep
In this work we put forth theoretical convergence guarantees for DeepONets centered around over- parameterization and show that prreeeeramttiinitioonn based on wider layers (for both branch and trunk net) provably helps in DeepONet convergence. This is reflected in Figure I which summarizes an empirical evaluation of wee-paraaeterrize DeepONets with ReLU and smooth activations prototypical operator learning problem. In order to complement our theoretical results, we present empirical evaluation of our guarantees on three template operator learning problems: (i) Antideriva- tive operator, (ii) Reacttonn PDE, and (ii) Burger's 's equation and demonstrate that wider DeepONets lead to overall lower training loss at the end of the training process.

---

Under review as a conference paper at ICLR 2023
10
200
100
101
10
10
10
311
10m
10-10
600
Epocha % 100
Epocite
(a) ReLU Activation
(b) Smooth Activation
Figure 1: Benefits of over- wwerpaaamtteriitioonn on learning of DeepONets for the Antiderivative Operator: Go (o(w)) (x) Je¹ uu{) dg. In both cases m denotes the width of the branch net and the trunk net. For both ReI.U and smooth activations, increasing the width m leads to much lower losses. Note that the y-axis is in log-scale.
The rest of the paper is organized as follows. Section 2 we review the existing literature on neural operators, operator networks and over-parameterization based approaches for establishing convergence guarantees for deep networks. Next, we devote Section 3 to briefly outline the the DeepONet model, the learning problem and the corresponding architecture. Section 4 contains the first technical result of the paper. In Section 4 we establish convergence guarantees for DeepONets with smooth activations (for both branch and trunk net) based on the Restricted Strong Convexity (RSC) of the loss. Next, in Section 5, we present the second technical result of the paper where establish optimization guarantees for DeepONets with ReLU activations by showing that the Neural Tangent Kernel (NTK) of the DeepONet is positive definite at initialization. In Section 6 we present simple empirical evaluations of the main results by carrying out parametric study based on increasing the DeepONet width and noting its effect on the total loss during training. We finally conclude by summarizing the main contributions Section 7.
2 RELATED WORK
2.1 LEARNING OPERATOR NETWORKS
Constructing operator networks for ordinary differential equations (ODE's) using learning- based approaches was first studied in (Chen & Chen, 1995) where the authors showed that a neural network with single hidden layer can approximate nonlinear continuous functional to arbitrary accuracy. This was, in essence, akin to the Universal Approximation Theorem for classical neural networks (see, c.g., (Cybenko, 1989; Hornik al., 1989; Hornik, 1991; Lu et al., 2017)). While the theorem only guaranteed the existence of neural architecture, it was not practically realized until (Lu et al., 2021) which also provided an extension of the theorem to deep networks. Since then number of works have pursued applications of DeepONets to different problems (see, e.g. (Goswami et al., 2022; Wang al., 2021; Wang Perdikaris. 2021)). Recently (Kontolati al., 2022) studied the influence of over-par parameterizatioon on neural surrogates based on DeepONets in context of dynamical systems. While their paper studies the effects of over-parane peer-arrameterzation on the generalization properties of DeepONets, optimization analysis of DeepONets is a largely open problem.
2.2 OPTIMIZATION: NTK, ETC.
Optimization of over--arameee over-parameterized deep networks have been studied extensively (see, e.g., (Du et al., 2019; Arora et al., 2019b;a; Allen-Zhu et al., 2019; Liu et al., 2021a)). In particular, (Jacot et al., 2018) showed that the neural tangent kernel (NTK) of a deep network converges to an explicit kernel in the limit of infinite network width and stays constant during training. (Liu et al., 202 1a) showed that this constancy arises due to the scaling properties the hessian the predictor as a function of network width. (Du et al., 2019; Allen-Zhu et al., 2019) showed that gradient descent
2

---

Under review as a conference paper at ICLR 2023
converges zero training enror in polynomial time for deep over- parameterize model, with (Du et al., 2019) showing it for a deep model with residual connections (ResNet) while (Allen-Zh Zhu et al., 2019) showed in context of geed-forward models, CNNs and ResNets. (Karimi et al., 2016) showed that the Dolyak-Lojasiewte used explain the linear convergence of gradient- based methods.
(PI.) condition, a much weaker condition than strong convexity can be
LEARNING DEEP OPERATOR NETWORKS
Learning neural operators (Li et al., 2020; Lu et al., 2021) is fundamentally challenging problem as it requires learning parametric maps between two sharp contrast to classical deep learning which learns parametric maps between two fiite-dimensional vector spaces. A succinct review of learning in infinite dimensions is provided in Section A. in the Appendix. Here we focus on the DeepONet model and outline its main features.
function spaces which is in
3.1 DEEPONET SETUP
In what follows, denotes the input and O the learnable parameters. A DeepONet an operator network that learns parametric map Go such that Go(u) G (u). where u denotes the input function, and Gt denotes the "truc" operator whose approximation predictor can defined as the inner product two deep networks: f = [fx net and = {g the trunk net, namely
the notation p(0;; to denote a deep network Se, 1m + R" where
wish to learn. Following (Lu et al., 2021) DeepONet
1K known as the branch k=
Golu)(v) = Σ
(1)
k=1
where u E Rdd is the input function and y dom(Go(u)) C Rd the output locations'. The training data comprises of input functions, that is {u(0) Ji=1 and q output locations for each G(u((),, i.e.
Hu (3) 19 } with yj denoting the j-th output location for Ga(u())). The branch net f has parameters , 12y and the trunk net g has parameters 0g E RP,, The entire set of parameters for the DeepONet is given by - 10, 0,1T= IRP Further, let { dom(u) IRd Vr [R] and u (xx) E Rd. For scalar functions (+) E R the branch net takes input {u()) (z, (x which implies R + R Similarly, for scalar output locations v/ E R we have g IR R The DeepONet learning problem can then be cast as the minimization of the following empirical risk:
a
0t arg min L (C)(()) 8ee
Σ (G (m())))))))
(2)
qi
==1
with
K
Go (o(u = Σ fo (v: (x.)))))))))) gk (0gi ())
(3)
and ((=) : 1/2(=)2 denoting the mean-squared error (MSE) loss. Note that the "true" operator Gt whose approximation is sought in (2) can either be explicit, c.g. integral of a function, or implicit, c.g. the solution nonlinear partial differential equation (PDE).
3.2 DEEPONET ARCHITECTURE
We now briefly outline the architecture used throughout the analysis in Sections 4 and 5 and in the numerical experiments in Section 6. We adopt fully connected feedforward neural networks (FNNs) for both the branch and trunk nets which is also the baseline DeepONet model in (Lu et al., 2021). Figure in the Appendix shows schematic of the architecture and the notation used throughout this paper. For the architecture we adopt the unstacked configuration (see, Fig 1d in (Lu et al., 2021)).
'The original DeepONet paper puts forth the above model and another one with bias term added to the inner product. For definiteness, we restrict our attention to the model without bias
3

---

Under review as a conference paper at ICLR 2023
Remark 3.1 (DeepONet Training Tuple). As mentioned in the prequel, each DeepONet training data
comprises of the tuple D :- (ful)) (x.))= " ( (uj (1) 19:
1z-1)
Remark 3.2 (Training Dataset), The entire training dataset then simply given by the collection of all tuples (D())
Remark 3.3 (Widths my and my). We denote the width of the branch net by my and the trunk net by
my and use my my = m interchangeably through Sections 4-5. Similarly, for the experiments we
use my = my unless otherwise stated. For the latter (i.e. when my # mg) the analysis in Section 4
still holds with m min (m,, mg)
Remark 3.4 (Last layer of the branch and trunk net), Note that the last layer of the branch and trunk
net is simply linear layer and does not have any nonlinearity/aciiationn
4 OPTIMIZATION GUARANTEES FOR DEEPONETS: SMOOTH ACTIVATIONS
In this section, we focus on DeepONets based on smooth activation functions. To build up to the optimization analysis, we first establish a bound on the spectral norm of the DeepONet predictor, in particular showing that || (1) (z)]l2 = O fm) where, again, m mf mg-- The spectral norm bound is then used to establish form of Restricted Strong Convexity (RSC) of the DeepONet loss (2), which in turn is used to establish geometric convergence of gradient descent (GD). For the analysis, analogous (Liu et al., 2021b), we consider FNN for the branch net:
w 8f1 8 u. 8 Φi (vmm
= = 9(L-1)
VI VE [L 11.
1].
(4)
f== - f.k.
[ [K]
where my = m and L denote the width and depth of the branch net respectively, [K] := KK. di is the activation function layer I, 88)) are the outputs layer and w w denote the weight matrices at layer 1. Similarly, we consider a fully connected feedforward network for the trunk net:
(1)
((0)
y, 1 Φ1 1
(Vm
1 1 mg 9
AI [T. 1],
.(()) (
(5)
gk = 1 Vk [K].
where, again, myg = m and L denote the width and depth of the trunk net respectively and W!!h = (1) denote the weight matrices layer 1 of the trunk net. In order aid our analysis, we make the following assumptions on the activations, the loss and the weights:
Assumption 1 (Activation functions). The activation functions Φt at each layer are 1-Lipschitz and 3a-smooth (i.e. ø" S (g)) for some B& > O.
Assumption (Loss function). We assume the loss Eig is (i) strongly convex, i.e., EY, 2 a > (ii) smooth, ie.., e < b, and (iii) Lipschitz, i.e.,
|6tsl S X., where we make use of the following notation (0) - Gl(( (o)))) Eas = (Go(u ( -6- (3,()),),)
Cas = 1 Go (1 (w)))) Gg 8" (Go(u()))()))) ( (uo))))))
Assumption 3 (Initialization of Weights). All weights of the branch and trunk net are initialized as (I) Fa t = w fo. ~ (0, of) and w = w "5 ~ (O, oZ) for E [L-1] and some constant oo > 0 respectively. Furthermore, in order for the model (1) to have a suitable scaling in our analysis. we initialize the weights in the last layer of the branch and trunk nets w ~ ~N 1/ (m 1/0011mmm1 K)) and 1o
(!)
is ~ (0, (/mm K)) respectively.
Remark 4.1. For Assumption 1, our analysis straightforwardly extends to general <-Lipschitz activations, with constants depending s. For Assumption 2, the Lipschitz loss assumption can be dropped by assuming that the true responses G (w)) are bounded and showing that the prediction responses Go Go()(()) are bounded with high probability.
4

---

Under review as a conference paper at ICLR 2023
Definition 1 (Norm ball). Our analysis focuses on the standard Euclidean norm ball around the initialization o, i.e. B B 0n). where
Euc (0) == {0 PR 11 0112 S sp
4.1 SPECTRAL NORM OF THE HESSIAN OF BRANCH AND TRUNK NETS
The convergence analysis makes use of the gradients and hessians of the total loss (2) and the predictor (1) with respect to the parameters 9, namely,
HJ5 Hof
Hig
VeC(O) Vo, -
f) and = H (0)
(8)
Hag
where Ve, (()) E RP' and Ve, ((0) E R Note that we make use of the notation Vo, (-) to denote the derivative wrt the parameters 0, and this not a functional gradient. Similarly, the individual blocks in the 2 x 2 block hessian (0) are given by
SL H 002²'
OC ,0,
OC ,0
and Hgg = 8C
H
Hos H
(9)
00, 2
where Hs E X Hgg E RPa*Ps, Hig E RRI*Ps, Hos E RPsxPy and the argument 0 is ignored for clarity of exposition. Using (2) and rewriting the derivatives in (8) and (9), we get
OC ;0;
-
Jh.
(1)
and
OC 00g
9
0 9h. (10)
4i
1=1
i=
k=1
for the gradients, and
OC
(
1
"
95
Σ
K
i (
Vo, Jk
Vo,
(i)
00,2
11
4b
k
1=1
k,k=1
2
96
8
(
4.
Σ
jk k
(2) (2)
V Vo, 00
( Vo.kk (T
00,
2
YI
4i
j=1
1,k=1
8
91
K
41
K Σ
Σ 9:
Σ
Σ
No,
Vo
IR.s &
( (1)
Vo,
(1) Vo ())
00,00,
7
qi
5=1
j=1
(6,kml
(11)
for the individual blocks of the hessian (8) where, we make use of the notation and f( = (l))
Kk.j gv (0:
Lemma 4.1. Under Assumptions 1, 2 and 3, and for 0 E (3Euc(00)) with high-probability have for all & [K]
( and
cls)
V2 (0)
Jk
max max
V 0
(12)
iEin
xE(m] [q:)
m
V fx 11 S e(f) and
Vo, 9u|| S gl5)
(13)
where c()) cla) e(
(lا) suitable constants,
(.) = a(-)/0,, Vo,() a(-)/00..
Vi ()) = ()) and V2 (.) 82 (-)/
Proof. The proof follows directly from Theorem 3.2 in (Liu et al., 2021a).
5

---

Under review as a conference paper at ICLR 2023
4.2 GEOMETRIC CONVERGENCE BASED ON RESTRICTED STRONG CONVEXITY
We now focus establishing the convergence of gradient descent (GD) by using the Hessian spectral norm bound. The convergence analysis is based on a generalization of the notion of restricted strong convexity (RSC) of the loss (2) (see (Negahban et al., 2012; Negahban & Wainwright, 2012; Zhang Cheng, 2015; Zhang Yin, 2013; Lai & Yin, 2013) for a review RSC and its applications in high--dmeensional statistics for linear models). In order further aid clarity of exposition, state the main results along with their implications in this section and leave the details the proofs to Section A.3 in the Appendix.
Definition (Restricted Strong Convexity (RSC)). A function said to satisfy a-restricted strong convexity (a-RSC) with respect the tuple 0) for any 0' Q S IPP and some fixed 0 E IR,, have C (0') 2 ((0) (0' 0, Vo)(0)) + 0' 012 with 0.
Note that L being a- a-RSC w.r.t (5, 0) does not need C to be convex on Rº. Further, let {0th22 denote sequence of iterates obtained from GD, i.c.,
0++1 0. T/ VoL
(14)
0_ T we will use dynamic restricted sets Q% C R,, where p Py + Py- to show RSC of the DeepONet loss. Note that these sets are parameterized by a constant x. which measures the absolute cosine similarity between suitable vectors (see Section A.3 and specifically Proposition 1 in the Appendix for detailed discussion on these sets). Further, our optimization for DeepONets is based on second order Taylor expansion the loss (2) w.r.t. O. = 10,, 0,,1
Since - 10
9.4.
1 (0 0.) 0.) (() 0.), [()) [(0.) + (0 - 0,, VoC(O.))
(15)
2
where, VoC(O.) and H are in (8), (9), (10), and (11), and Ô -0 + (1 -) for some TE 1].
Definition 3 (Q% sets.). For an iterate O. -0, 0g, consider the singular value decomposition
4
Vo,
Vo JeJ
Σ
(16)
k=1
where q S gk, and 0, E R by E IR' respectively denote the singular values, left singular vectors, and right singular vectors. Further, let
Go
1
Σ
n2
9
Go(u
(17)
n
Then, for some suitable k E (0, }11 we define the set:
0% = -:=0-- = =10';:0 0,] cos(O' O,, VoGoo)1
(18)
o 18 S an) Gy.1, bn)) 2 0, vh E [g]
Remark 4.2. Note that Py. Py respectively the number of parameters in the branch and trunk nets and the models can be sufficiently werrparameterriee such that they are larger than the number of training examples q Σ 9:
Remark 4.3 (Q' sets). The specifics of the restricted set Q PR stem from technicalities in the analysis. For detailed outline of these sets we refer the reader to Section A.3 and specifically Proposition in the Appendix.

---

Under review as a conference paper at ICLR 2023
Theorem 4.2 (RSC of the loss). Under the assumptions Assumptions 1, and 3, 40' 0% D B Rac (Go) with high probability we have
B
[(0)) 2 (0.) (0' O2 VoL (0,)) -2 110 -0.12 where az a= ci vil
C2
vm (19)
where G; = + JE Go (u )0 and C1 > 0 are constants. Thus, the loss C()) satisfies RSC w.r.t (B 0,) whenever a, 0.
Proof. A detailed proof is presented in Section A.3 in the Appendix.
Theorem 4.3 (Smoothness of Loss). Under the assumptions Assumptions 2 and 3, with high probability, for 0 E B (6o), (()) 3-smooth with 3 = bb² + evā with c= max(c ( (()))))))))))))))))))))))))))( o= max( (6 with clf), ,l,), glf), g as Lemma 4.1.
Proof. We again refer the reader Section A.3 in the Appendix for detailed proof.
Lemma 4.4 (RSC = Restricted PL). The RSC and Smoothness of the Loss together imply form of Dlyaa-Lojasiewiczz (PL) condition w.r.t. the tuple (B,, 0₂), unlike standard PL which holds without restrictions (Karimi al., 2016).
Proof. For a detailed outline of the proof, we refer the reader to Section A.3 in the Appendix.
Theorem 4.5 (Global Loss Reduction). Consider the same conditions in Theorem 4.3 and Or
for E [T] for a gradient descent update 0.11 = 0, - n VoC(O,) with π we for some We E (0, 2), where is defined as in Theorem 4.3. Then, with high probability, v0 arginf L(0) with BE
((0.+1)-200 2(0) and O.41 rnginneeqq:BBe0r0.. L(0), we have ((0.)-((0)
0 7t
((0++1)- - C(0)) s (1 (1-awi(1 74) (2 - wx) (C((.) - C()))
aww,(1
(20)
B
Proof. We refer the reader to Section A.3 and in particular Lemma 1 in the Appendix for the proof.
L
Remark 4.4. A direct consequence of Theorem 4.5 is global reduction in the loss in RP and thus the convergence of gradient descent for the DeepONet optimization problem.
5 OPTIMIZATION GUARANTEES FOR DEEPONETS: RELU ACTIVATIONS
We now present an alternative optimization analysis based on the Neural Tangent Kernel (NTK) (Ja- cot et al., 2018 8), establish guarantees for the convergence of gradient descent and its variants for DeepONets. Although the NTK convergence theory holds for both smooth and ReLU activations (see, e.g. (Allen-Zhu al., 2019; Du et al., 2019)), present this in context of the latter. Further, in this work we only present a proof for the positive definiteness of the DeepONet NTK at initialization. This allows to readily transeribe analogous existing arguments for deep networks, for the convergence of gradient descent, to the DeepONet model. (Allen-Zhu et al., 2019; Du et al., 2019; Nguyen et al., 2021) Now, recall the DeepONet predictor
K
Go (1())
(3)
Σ fa (0):u(()) gk (0g: (0):
(21)
"The hessian-basse analysis presented in Section 4 constitutes a sufficient condition and is not directly applicable for the case of ReI.U activations. In contrast, the NTK analysis presented here applies to both smooth and ReI. activations.
7

---

Under review as a conference paper at ICLR 2023
and its corresponding gradient with respect to the parameters 0,
9k Ug Vo, Su (0;:u(())
VoCio(u Σ
(22)
ffr (Oy; ult)) Vo,gt (0g: yj (6)
This allows us to write the NTK (( of the DeepONet, which is a q matrix as:
( [VooG(((()) VoGo
(23)
where, again, 7 y= -i= 9:- Given the branch and trunk nets are initialized using standard initial- ization techniques, as typically done in practice for deep networks, (Arora et al., 2019b; Du al., 2019) with the addition of Assumption 3, the resulting branch net NTK KG and trunk net NTK Kg namely
Kyf,. (V Ja (0r:
Vora (0ri (or:
nxn
(24)
Kg.. = ( (Vo,,.9 9h (0givg)) Vo,.tk (0gi M/
(8')
9x9
can be shown to be positive definite with high probability for each i E [K] (Nguyen, 2021; Liu et al., 2021a; Du et al., 2019). In particular, with high probability, for is E [K], we have
Amin (KG,,)) 2 Ao.f
and Amin (Kg,)) 2 Ao.a
V E
(25)
where Amin (AC,,.) and Amin Amin(K.g... (K.g,s) are the minimum eigenvalues of the branch and trunk net NTKs respectively, and Ao, Ao, > Ô are positive constants (see, e.g., Theorem 4.1 in (Nguyen, 2021)).
Theorem 5.1 (KC(0) is positive definite at initialization). Given standard initialization for the branch and trunk nets, and granted that the individual branch and trunk net NTKs are positive definite (24)-(25), the NTK of DeepONet is positive definite at initialization, i.c.
T [(K( 2 <>>0
(26)
where a denotes an arbitrary block unit vector with n blocks, and on corresponds to the j-th entry in the i-th block and denotes a positive constant.
Proof. We refer the reader to Section A.4 and specifically Propositions 2, 3 and 4
Remark 5.1. direct consequence of Theorem 5.1 is that the DeepONet NTK is positive definite at initialization. It is then straightforward to invoke standard NTK analysis to show convergence of gradient descent during training, see, e.g. (Jacot et al., 2018; Du et al., 2019; Arora al., 2019b;a; Allen-Zhu et al., 2019; Nguyen et al., 2021).
6 EXPERIMENTS
We now turn to an empirical evaluation of the effect of over-parameeerizzaion on the training perfor- mance of DeepONets, as measured by the empirical risk over a mini-batch / of the training dataset (3.2), for three canonical operator- leaanin problems. The results for smooth activations empirically verify the analysis in Section 4 whereas the ones for ReLU activations verify the analysis in Section 5. In all the examples described below, we consider FNN architectures for both branch and trunk nets which are similar to the ones chosen in (Lu et al., 2021). For definiteness, we set the width each layer of the branch and trunk net to be the same (i.c. my my m) and then increase it uniformly from m 10 to m = 500. We monitor the training process over 80, 000 training epochs and report the resulting average loss over each mini-batch with size n.
4:
Gt (f
LD.
(27)
where ng denotes the number of input training functions (w²) in the batch B. We refer the reader to Section A.5 in the Appendix for specific details of the setup. Figure 2 shows the training loss (27) as a function of the epochs for DeepONets with smooth activations and Figure shows the same for ReLU activations.
8

---

Under review as a conference paper at ICLR 2023
08
201
31
8
(a) Antiderivative: selu
(b) Diffusion Reaction: selu
(c) Burgers: selu
Figure 2: Training progress DeepONet with a smooth activation function selu (Klambauer et al.,
2017) 7) for (a) Antiderivative Operator, (b) Diffusion- Reactiion Equation and (c) Burger's Equation. We plot the training loss (27) as a function of the epochs with the y-axis on a log-scale to clearly distinguish the effect of increasing width (m1) Increasing the width of the branch and trunk net leads to lower losses for all the problems.
3
1:3
Fachs
Epoow 100
(a) Antiderivative: relu
(b) Diffusion Reaction: relu
(c) Burgers: relu
Figure 3: Training progress of DeepONet with ReLU activations for (a) Antiderivative Operator, (b) Diffusion- -eacttonn Equation and (c) Burger's Equation. The y-axis is again plotted on a log-scale to clearly demarcate the effect of increasing width. Increasing the width again leads to lower training losses.
Remark 6.1. We store the mini-batch training loss at every 100-th training epoch and observe that the training loss measured over the mini-batch is lower for wider DeepONets. This observation is consistent for both smooth u) and non-smooth (relu) activations.
Remark 6.2 (Antiderivative Operator). The Antiderivative operator is linear operator and hence is learned very accurately especially for wider DeepONets (LDg ~ 102 at the end training). C
Remark 6.3 (Diffusion Reaction). The Diffusion reaction equation also demonstrates lower loss
with increasing width, albeit less markedly than the antiderivative operator. This can attributed in part the fact that the operator is inherently nonlinear.
Remark 6.4 (Burger's equation). The operator corresponding to Burger's equation more intricate with the added periodicity constraints the solution (see Section A.5.3 in the appendix for details on the problem). We remark the distinction from the operator learning problem for Burger's 's equation studied in (Li et al., 2021a) where the operator only sought to learn the mapping from the input (initial condition L = 0) to the final output = and not the entire solution space (ær, () E [0, 1] x [0,
7 DISCUSSION AND CONCLUSION
We present two novel optimization analyses for DeepONets (Lu et al., 2021) based on over- parameterization and establish convergence guarantees for the DeepONet models with smooth and ReLU activations. The analysis for smooth activations is built on top of the restricted strong convexity of the DeepONet loss whereas the one for ReLU activations is based on the positive definiteness of the NTK at initialization. To the best of our knowledge, this is the first work to mathematically and empirically show the benefits of over preepearatttiiiiaioon on the the learning of DeepONets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A Convergence Theory for Deep Learning via Pareeearaamttiiiaiionn.. Technical Report
arXiv, June 2019. URL http: 1111. 03962. arXiv: 1ete:88111..0900 1.03662 [cs, math, stat] type: article.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. International Conference on Machine Learning, pp. 322 -32...2.22 PMLR, 2019a.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infimitely wide neural net. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019b. URL https: httpss////pproeeeeeeeeeee https/////p.ceeccgg..... paper/201 9 file/ /ttp://0110/111a/aacae---bbb-eee wttp://eebbfffeeeeaaaaaaaaeeeaaa
wwwp:/ewwiiiinnnnnnneeeeeeeeeeee neurips cc/
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http:// http://githbb com/ comm/goole//////// jax.
Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application dynamical systems. IEEE Transactions on Neural Networks, 6(4):911-917, 1995.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303- -14, 1989.
Beichuan Deng, Yeonjong Shin, Lu Lu, Zhongqiang Zhang, and George Em Karniadakis. Conver- gence rate of deeponets for learning operators arising from advection-diffusson equations. arXiv preprint arXiv:2 10621, 2021.
Tobin A Driscoll, Nicholas Hale, and Lloyd N Trefethen. Chebfun guide, 2014.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima deep neural networks. International conference on machine learning, pp. 1675-1685. PMLR, 2019.
Somdatta Goswami, Minglang Yin, Yue Yu, and George Karniadakis. A physics- infoice- variational DeepONet for predicting the crack path brittle materials. Computer Methods Applied Mechanics and Engineering, 391: 114577, March 2022. ISSN 00457825. doi: 10. 10.1016/j.cma.022. 114587. URL http://t/// /ttt:://araae http:a/axvvvv..////////////aa wwwp:iii.rrg/aaaaaaaeeaeaaeao 2208.. 06905. arXiv: 2108.06905.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2): 251-257, 1991.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks universal approximators. Neural Networks, 2(5):359 366, January 1989. ISSN 08936080. doi: 10. 101 10166/0893-6000088990000-- URL https: htw.//iinnnnubbb.///// elsevier com/ retrieve/ p1i/089360808850
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal- gradient methods under the polyak-ojasiewier condition. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 795-811. Springer, 2016.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self- Self-Normalizing Neural Networks. 1rXiv:1706.022 [cs, stat], September 2017. URL http: http/arxvvvv..o/../...... abs/1706 02515.
arXiv: 1706.02515.
10

---

Under review as a conference paper at ICLR 2023
Katiana Kontolati, Somdatta Goswami, Michael D Shields, and George Em Karniadakis. On the influence oaraterrrattiinitaooonn in manifold based surrogates and deep neural operators. arXiv preprint wwxXiv2203.0507// 2022.
Ming-Jun Lai and Wotao Yin. Augmented $telL_S and nuclear-norm models with a globally linearly convergent algorithm. SIAM Journal on Imaging Sciences, 602):1059-1091, jan 2013. doi: 10.1137/120863299. URL https::///// //te:////ooiio/ wwt.oooooooooooooo 111...1112222288229893
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural Operator: Graph Kernel Network for Partial Differential Equations. arXivv2003. 03445 fes, math, stat], March 2020. URL 2003. 03485.
wrg. abs/
arXiv: 2003.03485.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier Neural Operator for Parametric Partial Differential Eguations. arXiv:2010.08895 [cs, math), May 2021a. URL URRhttp: http://aaxvvvo//../.../ 2000.. 08895. arXiv: 2010.08895.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadeneshelii Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Markov Neural Operators for Learning Chaotic Systems. arXiv:2106. 201.69006.08888 [cs, math], June 2021b. URL arXiv:
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant. Technical Report arXiv:2010.01092, arXiv, February 2021a. URL http::tt//// /ttpvvawivvv///////////.././.av. org/ abs/2010 01092, arXiv:2010.01092 [cs, stat] type: article.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over- parameterieed non-linear systems and neural networks. Technical Report arXiv:2003 000007 arXiv, May 2021b. URL
org/ /200//200... 00307. arXiv:2003.00307 [cs, math, stat] type: arti-
cle.
Lu Lu, Pengzhan Jin, Guofei Pang. Zhongqiang Zhang, and George Em Karniadakis. Learning nonlin- ear operators via deeponet based on the universal approximation theorem of operators. Nature Ma- chine Intelligence, 3(3):218 229, Mar 2021. ISSN 2522-5839. doi: 10. 00.10088/2225--02-00000000 URL http:////// / / dx www.ddiw..ooo...oooooooooo...... 10.0000.00000/ 1038 s/28//4225555 020022660202000002--22200
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The Expressive Power of Neural Networks: A View from the Width.
Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: htttt:::/pppcccediiee http://pccceee..ccccoppcoo www.siiiiinnnnnnnnuneeeee scc/paper2011 / e e/ cb 68 788888181111
Sahand Negahban and Martin Wainwright. Restricted strong convexity and weighted matrix completion: Optimal bounds with noise. The Journal of Machine Learning Research, 13(1): 1665-1697, 2012.
Sahand N. Negahban, Pradeep Ravikumar, Martin J. Wainwright, and Bin Yu. A unified framework for high-dimensional analysis of $mS-estimators with decomposable regularizers. Statistical Science, 27(4), nov 2012. doi: 10.1214/12-sts400, URL https:// //ooi. /0../.1..111111 2F12-sts400
Quynh Nguyen. On the proof of global convergence of gradient descent for deep relu networks with linear widths. In International Conference on Machine Learning. pp. 8056-8066 PMLR, 2021.
Quynh Nguyen, Marco Mondelli, and Guido Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In International Conference Machine Learning, pp. 8119-8129, PMLR, 2021.
Sifan Wang and Paris Perdikaris. Long-time integration of parametric evolution equations with physics-informed DeepONets. arXiv:2106.08. 05384 [physics], June 2021. URL http: //arxiv. org//bss/206 05384. arXiv: 2106.05384.
11

---

Under review as a conference paper at ICLR 2023
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics informmed DeepOnets. arXiv:2103. 10974 [cs, math, stat], March 2021. URL
org/ abs abtp//b..........1 arXiv: 2103.10974.
Hui Zhang and Lizhi Cheng. Restricted strong convexity and its applications convergence analysis of gradient-type methods in convex optimization. Optimization Letters, 95):961-979, 2015.
Hui Zhang and Wotao Yin. Gradient methods for convex minimization: better rates under weaker conditions, 2013. URL https:://axxi..ceeett http:///aaww...ooo////////..... 4645.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 LEARNING OPERATORS
Here we briefly outline the notion of learning for neural operators (Li et al., 2021a; 2020; Lu et al., 2021). The standard operator learning problem seeks to approximate a possibly nonlinear operator the learnable parameters 0. The goal is to learn an optimal set of parameters 0 such that Ggt se G Given observations {ul) }3=1 U and {Gt (G
G1 U + V by parametric operator Gege U + V that depends
V where w() y is an i.i.d sequence from the probability
measure μ supported on U and G(u()) is possibly corrupted with noise, the objective is to find Øt as the solution of the minimization problem
ft arg min Buw [C (Go(u)), G (u))) oc
(28)
where U4 and are separable Banach spaces and a suitable cost functional. This is analogous to the notion of learning finite dimensions, which is precisely the setup classical deep learning used for.
A.2 DEEPONET ARCHITECTURE
Branch Net: f(0
my, width branch
Toput Functions
mh width of the trunk not
"inner product"
M
Go(u)(v))
ER!
[ kal
Oj
0 CR
Output Locations
0, ER*
Trunk Net: s(O,:v)
Figure 4: A schematic of the unstacked DeepONet architecture (Lu et al., 2021) used in this study, Note that the input functions need not be sampled on a structured grid of points in general.
A.3 OPTIMIZATION GUARANTEES FOR DEEPONETS: SMOOTH ACTIVATIONS
Theorem 4.2 (RSC of the loss). Under the assumptions Assumptions 1, 2 and 3, 4O' Q% n BEac (On) with high probability we have
B =
( 2 (0.) + (0' O., (O.( + at 0 2
0, 0 where a; v.=
C2
m
(19)
where G. n E 1 E Go. (u )1))) and C1. 02 > are constants. Thus, the loss L(0)
satisfies RSC w.r.t (B 0₂) whenever a > 0.
Proof. From the Taylor expansion in (15), establish (19) it suffices to focus on the second order term and for 0' E B¹ show
0.) (( 0,) 0 0.113
(29)
13

---

Under review as a conference paper at ICLR 2023
Given the 2 x 2 block structure of the Hessian as in (8), denoting 60 := 0' 0g for compactness, note that
S0 H 50, H 2 H 1 + 80 1g
(30)
Ti
T2
Focusing on Ti and using the exact form of Hs as in (11), we have
T
1 n 98
50, f.
9k.j
Vo
1
9k.
5k350 Vo, Sk 50,
(1)
50
V
(0)
72
91
n
q8
k=1
k=1
())
50,, Vo, Go (o(
2
Acpe!
||50,113
(31)
Vm
j=1
where (a) follows from Assumption 2 and Lemma 4.1. The analysis for T' is similar, and we get
(00g. Vo, Co(ul))) 2
Acgele) mg
T3
150
(32)
4i
Focusing on T2 and using the exact forms in terms of H16 and H(()) as in (11), we have
98
K
- TT =50
2
q1
V
e
k
T oo
T
50,
1=1
/=1
k=
+ 500
K Σ
I
Vo
(0)
)
Σ
K
f
Vo 80,
(T)
qi j=1
k=1
k==
n
( 80]
Σ
b bJ
80, + +80]
d Vo, Go, Vo, ( 80,
i.
j=1
Th (50,, an) (50 bn) + Σ 21 1=1
1
Σ
4 500 Vo, Go.( (50g. Vo, (o,
h=1
13 5 Vo, Go (u
(50g Vo, (o(
(33)
96
where (a) follows from SVD as in (16). (b) follows since by Definition 3, (80 and > 0, (80 g: bn) N 0. Combining (31), (33), (32), using m mgg my and max c(0)), we have
1
% ((50), Vo, (o (()))) + (50g) Vo, )() ((,)))))) 2 Acoei
50 T (0)50
|100
n
Σ
a
Σ 800 Go( + (50s. Vo, (o( ())
2
Acpe1
||50|1
Vm.
(b)
2a((00, Vo, Go Co(u()) (o)))) + + 80,, Vo, Go (o))))))))))))))
Acyei |000|3
2
m
=a (60. VoGo)² 2 Acoc1
|1801
vm
( < 1 G
Acae1
1001
||00||
Vm
601
where (a) follows from Assumption 2, (b) follows from Jensen's inequality and with Go Eッ 1
qi E Go (1))) as in Definition 3. (c) follows from Definition 3, and a;
ax Vo vm That completes the proof.
14

---

Under review as a conference paper at ICLR 2023
Proposition 1 (Q% is non-empty). For over-parametterized branch and trunk nets with Ps.Pg gk. the restricted set Or is non-empry.
Proof. We simply construct a 0' 0 0 T E Q along with the value of k. Without loss of generality, we make θ, the origin of the coordinate system and work with the unit vector g = VaCo, \o Further, we assume O' also to be unit vector. Then, our problem reduces to feasibility of the following system of two quadratic equations over 0 R 6, R ,
g 8g
07'
Σ h=1
The On b/g 0'2 2 0
(10 (0 Bol) k2
where Chi. The O. an E R are orthogonal unit vectors, bn IRPo are orthogonal unit vectors, = f 8g E RP/+Pp a'd [0'; g. RP are unit vectors, and we can choose E (0, 1]. Without loss of generality, assume | la 2 1B,122 Then, set 0 so that our feasibility condition
reduces (0)) 2 x² for some suitably chosen & E (0, 1]. Finally, set 0' |
81
so that
gi
(0),
8f
ggrl =
||g
so that i. E (0 (in fact, x. 2 1/v2) as desired. That completes the proof.
Theorem 4.3 (Smoothness Loss). Under the assumptions Assumptions 1, and 3, with high probability, for 0 E BBEcC(()) [(0) is 3-smooth with 8 = bo² + evl with max((l)) clo)), vm
max with clf), e g (19) as in Lemma 4.1.
Proof. By the second order Taylor expansion about 0, have L( [(() + (0' 1000-00 0, VoL(O))
(0' 0) 8 (())
O0' 0). where 0 50' + (1 E)O for some & E [0, 1]. Then,
30
0 C(
91
(0' 0) 002
(0' 0) (0' 0)T
VG
(C
11
v
G
(0' 0) (Q
1
0
46
1m1
j=1
q
Eg (0' 0) (0)))(("))))) 0)
1=1
1=1
Now, note that
1. 0 0, {o)((()))))}
T
qi
i=1
(a)
4
11 10' 0
bo 0112
where (a) follows by the Cauchy-Schwartz inequality and (b) from Lemma 4.1.
15

---

Under review as a conference paper at ICLR 2023
For 12, with Qu(ss)) (0' 0) (0 (10) -0), have
(Q(()))) - 110 012 FCg(a()) - 011
vm
Then, we have
"
9
2
(
(0'
0) (""oo(u("))) 0)
71
j=1
1/2
1
Qu
2i
1=1
Y c
2
where (a) follows by Cauchy-Schwartz. Putting the upper bounds on T and I2 back, we have
ev A
00' - 0) G'g
0) be²
0 012
vm
This completes the proof.
Lemma 4.4 (RSC = Restricted PL). The RSC and Smoothness of the Loss together imply (PL) condition w.r.t the tuple (Br, 0z), unlike standard PL which holds
form of without restrictions (Karimi et al., 2016).
Proof. Define
(0) :- [(0,) + O., VoL(O.)) +
0t *10 0, 2
2
By Theorem A.3, v0' B₁, have
[(0') 2 Lo.("))
(34)
Further, note that Lo, (0) minimized at 0.41 = 0: VoL(O,)/ an and the minimum value is:
inf Lo, (0) (0.+2) L(O:) 1
VoC(0.))
2a
Then, we have
inf C()) inf (0) 2 inf Lo, (0) = L(0,) 20z VoL(0))
OEB,
OEB.
0
where (a) follows from (34). Rearranging terms completes the proof.
Lemma (Local Loss Reduction B.). Let oie, 3 be as Theorems A.3 and A.3 respectively, and
B. = Q n BBe (06) n BBuc(( Under assumptions Assumptions 1, and 3, for gradient descent with step size nte
We wi 2), any B.+1 E arginf [(0), we have with high probability D
L(0,+1) L(0 (1- aywe
(2 we) (C(O.) L(0r+1))-
(35)
3
Proof. Since C is 3-smooth by Theorem A.3, we have
C(0++1) s L(O.) + (0.41 Or. VoL(O.)) + 0 0.11
L().) nrl /l(Vee((0.)) +
VoL(O.)13
(36)
2
3m
=((O) 1 2
||oo((0))1
16

---

Under review as a conference paper at ICLR 2023
Since 01+1 arginf@EB, ((0) and az > 0 by assumption, from Lemma A.3 we obtain
- -IVoC((,.) (C(((((O.)) ((O++1)).
Hence
C(0++1) C(0++1) S ((0,) [(0++1) 718 (1- 8T
(Vo((.))
(a) S ((0,) [(0(+1)
1 5mp
(ar(C(())) - -(0(+1))
71
2
1 20 (1-
31pe
(C(O,) ((0++1))
2
where (a) follows for any S 3 because this implies 1
Byg 2
2 0. Choosing T/2 = J we E (0,2),
L(0++1) L(0++1)
(1-10 agwt
(2 wi) (C(O.) L(0
3
This completes the proof.
Theorem 4.5 (Global Loss Reduction). Consider the same conditions as in Theorem 4.3 and a₁ > fort E [T] for gradient descent update 0(+1 0, 77. VeC(O,) with n for some Wi E (0,2),
where 3 defined as in Theorem 4.3. Then, with high probability, v0 arginf L(0) with BEBT
0 S " = -(0++1)-210) ((0)
< and 0,41 E arginf (eg: ((0), we have
-(0++1)-(()) [()) S (1-0r01)
awwr(1 -1) a (2 wi) (C(0))-C(0)) C()))
(20)
3
Proof. We start by showing " 1(0,11)-2100 satisfies 0 S 7 1. Since 0 arginf L(0), [(O,) C(0))
oe Be (0a)
0c+1 arginf ((0), and 0c+1 0 BECC by the definition of gradient descent, we have BEB,
(0)
1 1 (oo(()))1 < L(O.),
L()') S ((0++1) < ((0--1) S ((,,)
23
where (a) follows from (36). Since L(0(41) 2 L()0) and ((O.) > L()"), we have 7 2 Further, since L(0++1) < L(O)), we have "/ 1.
Now, with w/ E (0, 2), we have [(0 L()") [(0(+1) [(0++1) + [(0 L(0))
1 anwi
3 (2 wi)
(C((,) - L(0,+1)) + (1- (1-oowe (2 mi)) -C(0++1)L((0))
aowe
B
1 aowe (2 - -wi) ((0.HH)) (((17) -1- (2 wi) ((01)
+ ((01+1) -
anwi
(C(O,) - L(0")) - aywe (2 w:)(<(( L(0"))
3 (2 wi)
onwir
(2 wi) (C(O,) - C()")) - a,41 - (2-w)) - C()"))
3
Onwi
3 (1 (2 wi) (C((,) 2()))
That completes the proof.
17

---

Under review as a conference paper at ICLR 2023
A.4 OPTIMIZATION GUARANTEES FOR DEEPONETS: RELU ACTIVATIONS
Recall the DeepONet predictor (1)
K
Go (w')) (1())(())))
( (9)
E5 SA (D):u(()) 9h (0g: v/
(37)
k=11
In the analysis, it is useful to distinguish between the parameters up to the pre-final layer and the final layer, i.e. dim M, + M, + (m, + mg) K. where M, and Mg denote the number of parameters in the branch and trunk nets till the pre-final layer respectively and m K and my K are the number of weights in the last layer of the branch and trunk nets respectively. In essence we have dim(@) M m K and dim(O, = M, + my K. We note that it is sufficient to show positive definiteness of the above NTK at initialization. Once that has been established, standard approaches (Jacot et al., 2018; 8; Du et al., 2019; Arora al., 2019b;a, Allen-Zhu et al., 2019) allow one to show the geometric convergence of (S)GD. In the sequel it proves useful to rewrite the branch and trunk net outputs as:
5
mgg Σ
fk (05;
W Th (0): gk (ogiv)) (00
W K (0g:u;))
A k E [K]
(38)
where Wi,h: h [my]] are the weights of the linear last layer of the branch net and (o) " N' [mg]
(f)
the weights of the linear last layer of the trunk net. Similarly, 0 and 0g are the parameters leading up to the pre-final layer branch net with my outputs [ff,, h E [my]] and trunk net with mg outputs [ox. h' E [myll respectively. We will denote by f all the parameters corresponding to fk.
iee 0f == [ h E [mm]].0 0y} which includes all the parameters needed for fx for each k E [K]. Similarly we denote by gg all the parameters corresponding to gk. i.e. gg.k == (wkhh h E [m²]) 0z).
We can explicitly write the NTK for the DeepONet model, specifically entry corresponding to the
inputs (u()) .yj (0) and fulet, .y (2)) as:
(v) VoGo(u (w) (()))))
Σ
gh (0)i0{") (1)) ge (o):v(")) (Vo, fu (0r:u(0) .e, (Or:ulet)
k
Σ fir (or:u(o) fMr (Driultt) (Vegon (Doiv(") Vo,9w (0oi0)"))
k,==1
K
(0giv²")
gk (ogi (Vo fi (Driu(0) Vo, fo (0):u((1))
9h.
k=
T
K
Lk (D)su(()
So (Driulit))
(Vog (0)i Vo,gi (0gi (g:
T/
Σ gk (0giv))) (0)
gN (00i0)")) (ver fir (Br:u()) -oo (0):u(3))
(())
Σ Sk (Driu()) SKr (Driuctt) (Vo, (0nsv)") Vo,gi (0gju)"))
2 k.b
(39)
18

---

Under review as a conference paper at ICLR 2023
Theorem 5.1 (K((0) is positive definite at initialization). Given standard initialization for the branch and trunk nets, and granted that the individual branch and trunk net NTKs are positive definite (24)-(25), the NTK of DeepONet is positive definite initialization, i.c.
[((()) 20>0,
(26)
where denotes arbitrary block unit vector with blocks, and og corresponds to the j-th entry in the i-th block and e denotes a positive constant.
Proof. The proof follows as direct consequence of Proposition together with Propositions: 3 and
Proposition 2. With the branch and trunk net weights initialized using standard initialization techniques and the last layer of branch layer initialized according Assumption 3, have
E R.k
0gl =
BTT (4) k,k' 0g. 0gl =0. k,K' E [K]
(40)
where T. defined in (39)
Proof. As noted in Assumption 3, the last layer weights for the branch and trunk nets are initialized mean Gaussians, i.e., wk, the other layers. Now,
(f) (9) (0, mxx for l E [K], h [my], h' [mg], similar to
T gk (0giv;") (1) ge' (0givz²) (7)
k Ih (0gi (/("
(g)
Σ
(0)
(0gi (0g:o("))
(41)
w k. w (g)
(g)
(0gi v/; Ih. (0xs v/4))
w K.
which in turn implies
E B[T(3) (3) k, 10g
0g] = Σ
B Blwokgh (g)
(g)
nol (0giu)") Dā. (f)
k w
hr M
(42)
I k
(g)
E
(g)
1ow (0g: v//")) gh (0givf) (f)
ke kr.
A ====1
0
where (a) follows since Wk. and (g) completes the proof.
(g)
are independent. The analysis for T.k is similar. This
Proposition 3. Given that the terms (3) and k. be suitably bounded close to zero, have for any arbitrary block unit vector
K
aTK(0)a No.5
Ao,9
Σ a
Elao
(I,
(43)
where 1 denotes the q-dimensional vector of all entries equal to one.
19

---

Under review as a conference paper at ICLR 2023
Proof. Focusing on quadratic form of KC(O), and ignoring the T(3), block unit vector a, we have
2 -T.. terms, for any arbitrary
a -(()aa== (1M))))))
E
aigapd (eo))(
VoGo(u(())({f))
E
Σ 9k (gg: (0g:0)") gE (es:0 ( (Vo (or:u(t) -Vo (0::u(t))
(L
E
Σ Jk (0rsu(1) Sx (Ossw(t2) (V (0g:v)") Vo,yu (ggiu)) (8)) k=1
(3))(1)))
K
Σ
(0s:0²") gk (0gi (Voli (0):u(0) Vook (or:u(t))
(C)))
Σ
a (0r:ul() fx (or: ( (Vegu (Boin("). Vegk (evivs"))
Σ
Σ
(ais9k (0g:0] (av.rox (0giu)))) (opo))(vooo) (Vos% (0):u()) Vofk (0):u(())
(19)) (i.s)
K
Σ
(aisfh (0):u(1)) (av (0;: (Vegk (0g:v,") Vook (ogiu)"))
(()).(P-3))
K
| 9x(0g; 11 Amin (Kg,,t) a @ (1, s v)) |
A
2 (n))
K
K do.g
Σ
(0000
Σ
Σ
(1 (100 fu(0 ||²
a
Ao.
where I, denotes the g-dimensional identity matrix since 0ig9k (00: varies with j whereas the kernel Ky does not; 1, is the g-dimensional all ones vector since for the fn (l) terms, for a fixed i, ai differs with j but fx (0y; ulf) stays the same; (Kin((K,,. 2 d and Amin (K.g.) > Ao.g. This completes the proof.
Note that we require a block unit vector with oi, denoting the j-th position in the i-th for convenience in dealing with the quadratic form above. Given that T(3), ThE terms are close to 0 in expectation, shown in Proposition 2, now need to show that the NTK (C(O) is positive definite.
Proposition 4. Given that the terms The and k.k can be suitably bounded close to zero by Proposition 2, we have, for any arbitrary block unit vector a, and some k E [K].
a e (00i0 [1
or (10 fu
(1,
2220000 1
(44)
where 1 denotes the q-dimensional vector of all entries equal to one.
Proof. Now, for the first term, for some k E [K], making use of (38), we have
mg
Os Ir (0g: (00)=((())))
(Wk) on.9 (0g:v)))
09.j9h 1
Since the trunk net is ReLU network, following the argument in Lemma 7.1 in (Allen Zhu et al.,
2019), with probability at least Olge ((mm/aL)), for all (i.j)) we have |g. (0gi (1) || 2 1/2.
20

---

Under review as a conference paper at ICLR 2023
NOO,, tt), taking expectation over the randomness of wk,h have (a) Recalling that W.h
(a)
a 9h (0:: y (2) 1) =
Σ
wk. : 19. (niiN 1j (0rin("))
(1,)
TE(( u1 R.
(9)
)² og 0g: Ug
(1-)
Σ
g. (0:iV²)) > 112
K
(1.)
1
||01
2K
2K
Hence, we have
K
oo,
E
Σ k=1
a
3 9) (0)00
00) 001
zdn.s
1
2
and, with a similar argument
K
Ao.g
E
2 1 (1 u) Ao.9.
220.9-
1
k=1
As result, we have
a ['E()(() Ao.r + Ao 0.
(45)
2
The high probability version of the result can be obtained by applying Hoeffding (for cross terms) and Bernstein (for square terms) bounds on (w(() 0s..9. (0gi y1 (6)
) That completes the analysis.
A.5 EXPERIMENTAL DETAILS
For the optimizer we choose Adam (Kingma & Ba, 2014) with an adaptive learning rate schedule initialized at a learning rate To 10-3 In order generate training data for all three examples, we sample the input, denoted by u(x), from a zero mean Gaussian process (GP) on grid {2) )m E [0, 1] and generate outputs corresponding each sampled function by solving the ODE/PDE (see (Wang et al., 2021; Lu et al., 2021) for detailed discussion on data generation). For end-to-end training we use the deep learning framework JAX (Bradbury et al., 2018) and build our code on top of (Wang et al., 2021) for Inffission-Reactiin Burger's equation. We now briefly outline the problems below along with the specifics of the training process for each of them.
and Antiderivative operators and we develop our own for the
A.5.1 ANTIDERIVATIVE OPERATOR
The antiderivative (or simply the integral) operator corresponds to linear operator defined explicitly by a linear ODE (initial value problem) in the unknown function v(x) E V, given the input u(x) E U. and the constant v(0) for mathematical well-posedness, i.c.
do(x) dr
u(xx) x E [0.1] s.t. x(0) = 0.
(46)
We learn the operator mapping u(x) to its corresponding integral v(x) = (o( for all x E (0, 1]. For generating the training data, sample the input functions from univariate Gaussian process as outlined above and the output points randomly on the interval [0, 1] and choose n, = 10000 for our empirical results
21

---

Under review as a conference paper at ICLR 2023
A.5.2 DIFFUSION-RACCIOO PDE
In this example we learn the operator mapping the input forcing function u(x) to the output o(z; for the nonlinear Diffusion-Reaction PDE given by
(((0, x) 0 v(t,0) 0 o(t, 1) = 0
dv 01
D 80v Ox.2
ku u(x), (x, (0,1) (0,1) s.t.
(47)
where = 0.01 and k 0.01 are constants denoting the diffusivity and reaction rate respectively. Note that in this case the PDE is solved using a backward finite-difference solver on a grid (z, f) of size (150 x 120). For training, the number of input sensors fixed m 120. The number of input samples (n) is chosen to be 5000 and n 10000.
learning the operator v(a, t) Go(u)(a t). For each sampled input,
A.5.3 BURGER'S EQUATION
Finally, we look at the Burger's equation benchmark similar to the one investigated in (Li et al., 2021a) with the distinction that learn mapping from the initial condition v(a,, 0) =u(r) to the
solution o(x, 1) for (x,() E [0, 1] x (0,
du Ou
v 0%
+
= 0, (z,t) E (0.1) x (0.
Ōt
dx
0x
(48)
[v(2,0) 0) u(z),
x 2E (0,
(((,t) = v00tt) t) te (0
We generate the training data using a stiff PDE integrator chebfun (Driscoll et al., 2014) on a grid resolution of (,01,501) and p 200 training points sampled randomly the solution grid.
22


Reviews:
[{"review_id": "W3boKIYuGPU", "reviewer": "Reviewer_BsC5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper establishes theoretical guarantees about overparameterized DeepONets. In particular, they show that", "summary_of_the_review": "Overall, I lean reject. In particular, my concerns are primarily that 1) DeepONets are old and better, more novel architectures have been introduced, 2) the insights given by the paper seem rather weak/intuitive, 3) the proof structure feels like a reapplication of standard techniques from the field to a different architecture.", "strength_and_weaknesses": "Strengths\n-----------\n* The paper's results are presented reasonably cleanly. This is especially nice given the heavy analytic nature of the work and greatly enhances the communication of the results.\n* The paper's results all seem to be correct.\n* The experimental results show that increase network width does improve the results.\n\nWeaknesses\n--------------\n* The paper does not mention any of the more recent operator networks (e.g. fourier neural operators) which tend to show improved performance over DeepONets. This tends to make the analysis feel out of date.\n* The overall results feel rather weak. In particular, it's not that surprising nor insightful that overparameterization would help DeepONets, especially with the wealth of literature on overparameterization.\n* The proof techniques used feel like just reapplication of standard overparameterization tools. In particular, it treats the DeepONet as just a different architectural design (akin to MLPs or Resnets). However, the DeepONet primarily acts on functionals, which should necessitate something different. Is there any operator perspective to the analysis?\n* The experimental results are really not necessary, don't add much, and don't really feel like a worthwhile inclusion. This is true even in comparison with general theory papers (which tend to have a weaker experimental setup), and so I would move them to the appendix and expand upon the theoretical contributions.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VQphlUruIxo", "reviewer": "Reviewer_sgD3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper analyzes Deep operator Networks (DeepONets) regarding the optimization convergence guarantees. The author provides optimization analysis on two different networks: smooth activations and ReLU activations. Their analysis shows that the overparameterization on branch and trunk networks allows faster convergence to the global solution. ", "summary_of_the_review": "Overall, I believe it is a good paper to have the better understanding on DeepONet.", "strength_and_weaknesses": "Strength\n\n1. The paper is well-written and easy to read.\n2. The author's convergence analysis includes both smooth activations and ReLU activations.\n3. Theoretical results also match the empirical observations.\n\nWeakness\n1. Deng et al. (https://arxiv.org/abs/2102.10621) have studied the convergence analysis on DeepONets. The author should cite this work in the modified draft.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "P0QBbmXPFy7", "reviewer": "Reviewer_4Wz6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied the convergence of gradient descent algorithm in training the DeepONet in the overparameterized regime. Specifically,  the authors proved a spectral norm bound for the Hessian of DeepONets with smooth activation functions; this bound implies convergence of GD using the RSC property. For ReLU activated DeepONets the authors showed that the corresponding neural tangent kernel at initialization is positive definite which again implies convergence of GD. ", "summary_of_the_review": "I am inclined to reject the paper due to the lack of sufficient novelty and significance. ", "strength_and_weaknesses": "Strengths: The analytical results on the training of operator networks seem to be new and extend the prior analysis for learning finite dimensional functions. The paper is easy to read and follow.\n\nWeaknesses: I personally feel that this paper is merely a trivial extension of the early results of Du et al 2019 and Liu et al 2021 for finite dimensional functions to operator networks. The proof ideas and techniques based on the analysis of kernels and PL-condition are identical to those shown in the early literature. I do not see the essential difficulties in the extension except the technicalities arising from the new network architecture.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EKnXe3LmpTa", "reviewer": "Reviewer_QX4p", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides convergence analyses of gradient descent for learning deep, over-parameterized operator networks. The authors show the guarantees two types of networks: (1) with smooth activations and (2) with ReLU activation using two different analysis techniques.\n", "summary_of_the_review": "The analyses presented in this paper are non-trivial, but I think it needs some more work to be ready for acceptance (e.g., more explicit convergence results, discussions about limitations of each of the analysis techniques....)", "strength_and_weaknesses": "Regarding the strengths, this paper\n+ establishes first guarantees of GD for deep operator;\n+ shows the non-trivial convergence analyses, albeit using standard analysis techniques (restricted convexity and NTK).\n\nHowever, there are some drawbacks: \n- The authors do not explicitly state convergence results in dedicated theorems, so it is difficulty to understand the full picture (e.g., convergence rate, role of learning rate…) In particular, while I understand how the analysis goes, I cannot see how the width (or the over-parameterization) attributes to the overall convergence.\n- The paper presents two different analyses due to the smoothness without discussing limitations or why one can't apply one to the other.\n- The experiments are based on Adam, an adaptive SGD, instead of GD.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7LyIpQnj2M", "reviewer": "Reviewer_gCk5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the optimization convergence guarantee of GD for training DeepONets with wide layers. Technically, the authors prove the convergence for both smooth activations and ReLU activation and show that a linear convergence rate can be achieved. The authors further present empirical results to back up the theoretical findings.", "summary_of_the_review": "Overall, I am not satisfied with the initialization in Assumption 3, the authors may need to make the training of all layers be in the same level rather than using different initialization scaling for different layers. Besides, the contributions of this paper are not strong, especially given a large number of prior works studying the convergence of GD in the NTK regime.", "strength_and_weaknesses": "Strength:\n1. This paper proves the convergence for DeepONets with smooth activations based on the RSC condition.\n2. This paper proves the convergence for DeepONets with ReLU activation based on the NTK analysis.\n3. This paper conducts numerical experiments to justify the theory.\n\nweakness:\n1. The initialization of weights in Assumption 3 seems unreasonable to me. In fact, the authors consider using a small scaling for only the last layer, then the optimization on the last layer weights will dominate the training of the remaining weights. By (5), the neural network model will be very close to the linear model. \n2. It is also not clear to me why the authors need to use two sections to study the convergence of DeepONets with two different types of activations. My understanding is that they can be both performed via standard NTK analysis (in the lazy training regime, there is nearly no difference between NTK and RSC based analysis).\n3. In the ReLU case, it would be better to give a quantitative bound on $\\lambda_{0,g}$  or $c$ in Theorem 5.1.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject