Optimization for Neural Operators can Benefit from Width
Authors: Pedro Cisneros-Velarde
Bhavesh Shrimali
Arindam Banerjee
Venue: ICML 2025 poster (2025)
Keywords: neural operators
optimization
training
Primary area: deep_learning->theory
TL;DR: We provide optimization guarantees for two popular neural operators—Deep Operator Networks (DONs) and Fourier Neural Operators (FNOs)—under a common framework.

Abstract: Neural Operators that directly learn mappings between function spaces, such as Deep Operator Networks (DONs) and Fourier Neural Operators (FNOs), have received considerable attention. Despite the universal approximation guarantees for DONs and FNOs, there is currently no optimization convergence guarantee for learning such networks using gradient descent (GD). In this paper, we address this open problem by presenting a unified framework for optimization based on GD and applying it to establish convergence guarantees for both DONs and FNOs. In particular, we show that the losses associated with both of these neural operators satisfy two conditions—restricted strong convexity (RSC) and smoothness—that guarantee a decrease on their loss values due to GD. Remarkably, these two conditions are satisfied for each neural operator due to different reasons associated with the architectural differences of the respective models. One takeaway that emerges from the theory is that wider networks benefit optimization convergence guarantees for both DONs and FNOs. We present empirical results on canonical operator learning problems to support our theoretical results and find that larger widths benefit training.

Optimization for Neural Operators can Benefit from Width
Pedro Cisneros-Velarde 11 Bhavesh Shrimali 2
Arindam Banerjee?
Abstract
to scientific computing, we often need to learn mappings between input and output function spaces. Neural operators have emerged as the prominent class of deep learning mod- els used to learn such mappings (Lu et al., 2021). They have become a natural choice for learning solution operators of parametric PDEs and of inverse problems where multiple evaluations are needed under different parameters of the problem. Two of the arguably most widely adopted neural operators are Deep Operator Networks (DONs) (Lu et al., 2021; Wang et al., 202 102 and Fourier Neural Operators (FNOs) (Li et al., 2021a;b).
Neural Operators that directly learn mappings be- tween function spaces, such as Deep Operator Networks (DONs) and Fourier Neural Operators (FNOs), have received considerable attention, De- spite the universal approximation guarantees for DONs and FNOs, there is currently no optimiza- tion convergence guarantee for learning such net- works using gradient descent (GD). In this paper, we address this open problem by presenting a uni- fied framework for optimization based on GD and applying establish convergence guarantees for both DONs and FNOs. In partieular, show that the losses associated with both of these neu- ral operators satisfy two cndditions-resrricce strong convexity (RSC) and smoothness- that guarantee a decrease on their loss values due to GD. Remarkably, these two conditions are sat- isfied for each neural operator due to different reasons associated with the architectural differ- ences of the respective models. One takeaway that emerges from the theory is that wider networks benefit optimization convergence guarantees for both DONs and FNOs. We present empirical re- sults on canonical operator learning problems to support our theoretical results and find that larger widths benefit training.
The fundamental idea of a neural operator is to parameterize mappings between function spaces with deep neural net- works and proceed with its learning, i.e., optimization, as in standard supervised learning setup. However, contrary to classical supervised learning setting where we learn map- pings between two finie-diimeniionll vector spaces, here we learn mappings between infinite-ding fiiiiite-dimessinaal function spaces. While there exist results on the universal approxi- mation properties of DONs and FNOs (Deng et al., 2022: Kovachki et al., 2021), to the best of our knowledge, there are no formal optimization convergence results for the train- ing of these two popular neural operator models.
To address this open problem, in this paper, we establish formal optimization convergence guarantees for learn- ing DONs and FNOs with gradient descent (GD). To achieve this, we first propose a general framework that ensures the optimization of any loss using GD as long as two conditions are satisfied across iterations. The first condition is based on restricted strong convexity (RSC), recently introduced alternative (Banerjee et al., 2023a) to the widely used neural tangent kernel (NTK) analysis (Liu et al., 2021a; 2022b; Allen-Zhu et al., 2019). The second condi- tion is based on a smoothness property of the loss function. For feedforward neural networks, the RSC condition relies on the second-order Taylor expansion of the loss (Baner- jee et al., 2023a; Cisneros-Velaan Hessian of the neural network (i.e., second order structure), whereas the NTK approach relies on a kernel approximation of the training dynamics (Jacot et al., 2018), using the gradi- ent of the network (i.e., first order structure). For a specific model such as neural operators, the technical challenge in using our optimization framework establish suitable properties of the loss, its gradient, and its Hessian in order
1. Introduction
Replicating the success of deep learning in scientific com- puting such as developing neural partial differential equation (PDE) solvers, constructing surrogate models, and devel- oping hybrid numerical solvers, has recently captured the interest of the broader scientific community (Kutz& & Brun- ton, 2024; Kovachki et al., 2023). In relevant applications
et al., 2025), using the
'Equal contribution VVMware Research Uniiersity of Illinois trbana-Champaign. Correspondence to: Pedro Vellars- <pacisne gmail.com>, Bhavesh Shri- mali <bhavesh. shrimali @ gmail.com>. Arindam Banerjee <arindamb@ @ illinois.edu>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Optimization for Neural Operators can Benefit from Width
to show that the RSC and smoothness conditions are indeed satisfied. We point out that convexity of the loss is not one of the required conditions, so we are not attempting to show that the Hessian of the loss is positive semi-definite, as that will not be true for most neural models, including neural operators.
Paper Organization. Section 2 presents related literature. Section 3 outlines the architectures and learning problems for DON's and FNOs. Section 4 establishes our general opti- mization framework, and Section 5 and Section 6 establish convergence guarantees using this framework for DONs and FNOs respectively, highlighting the benefits of width. Section 7 compares our results and known ones for standard neural networks. Section 8 presents empirical evaluations the benefits of width. Section 9 is the conclusion.
Having defined general optimization framework based on RSC and smoothness conditions, the key novelty of our current work is showing that the losses for DONs and FNOs provably satisfy these two conditions when the neural operators are wide, despite the substantial differ- ences in their architectures and mathematical analyses. For both DONs and FNOs, we need to bound the Hessian of their respective empirical losses and of the neural operator models themselves in order to determine whether the RSC and smoothness properties are satisfied.
Notation || |l denotes the Z2-norm or the induced matrix Lg-norm when the argument is a vector or matrix, respec- tively. Given an operator/function f. raa(ff) and dom denote the range and domain of f. respectively.
2. Related Work
We only present a brief overview of the literature related to our work and provide a more extensive treatment in Ap- pendix A. In the case of DONs, approximation (Lu et al.. 2021) and generalization (Kontolati et al., 2022) proper- ties have been formally studied, as well as several appli- cations (Goswami et al., 2022; Wang & Perdikaris, 2021; Diab & Al Kobaisi, 2024; Centofanti et al., 2024; Sun et al., 2023). Nevertheless, optimization guarantees for DONs is an open problem. Approximation properties for FNOs have been formally studied (Kovachki et al., 2021), and diverse applications of FNOs and various Fourier-based operators have been formulated (Li et al., 2020a; Liu et al., 2022a; Wen et al., 2022; Pathak et al., 2022: Centofanti et al., 2024; Li et al., 2023; Yang et al., 2023; Harder et al., 2023). Nev- ertheless, optimization guarantees for DONs is also an open problem. Though formal optimization guarantees for neural operators are largely absent, there is more established liter- ature on such guarantees for neural networks. We highlight two particular approaches for optimization analysis: based on the NTK approach (Jacot et al., 2018; Liu et al., 2021a; aa; Banerjee et al., 2023b; Du et al., 2019; Allen-Zhu al., 2019) and on the RSC approach (Banerjee al., 2023a; Cisneros- Velarde et al., 2025) our work is related to the latter.
The challenge in the analysis of DONs stems from the fact that the output of this neural operator is the inner product of two neural networks. This greatly complicates the Hessian structure of the loss compared to standard neural networks. Indeed, the Hessian now contains roos-interrcction terms between two neural networks which have to be carefully analyzed and which require more complex definition of the restricted set over which the RSC property is defined compared to standard neural networks.
The challenge in the analysis of FNOs stems from the fact that it contains, inside their neural network structure, learnable weights that define transformations in the Fourier domain-ssmethiing absent in standard neural networks. This complicates the Hessian structure of the FNO since it contains parameters both in the data domain and trans- formed Fourier domain leading to cross-derivatives between parameters from both domains. Thus a more involved anal- ysis than of standard neural networks is required.
Remarkably, we find that the widths of both neural op- erator models benefit optimization guarantees in similar ways. First, the widths appear in the RSC condition such that larger widths make this condition less restrictive. Second, larger widths enlarge the neighborhood around the initialization point where our optimization guarantees hold. Similar benefits from larger widths were found for stan- dard neural networks by Banerjee et al. (2023a), despite the substantial differences between our analyses and theirs (as mentioned in the challenges above).
3. Learning Neural Operators
A neural operator (Li al., 2021a; 2020b; Lu et al., 2021) is parametric model based on neural networks that aims to best approximate mapping between two function spaces, which can be linear, such as the antiderivative or integral operator, or nonlinear such as the solution operator of a nonlinear PDE. Thus, letting Gf denote the ground-truth operator we are trying to approximate and Gg denote the neural operator parameterized by the parameter vector 0, the objective is learn O such that, given an input function u. we have Go (n)) R Gt (u)) Such learning is done by solving
Finally, to complement our theoretical results, we present empirical evaluations of DONs and FNOs and show the benefits of width on learning three popular operators in the literature (Li et al., 2021a; Lu et al., 2021): antiderivative, -ecattionn and Burgers' equation. Our experiments show that increasing the width leads to lower training losses and generally leads to faster convergence.
2

---

Optimization for Neural Operators can Benefit from Width
an optimization problem using data samples consisting of tuples of input and output function values of GT. This optimization problem is analogous to the notion of learning in finite dimensions, which is precisely the setup for which classical deep learning is used.
3.2. Learning Fourier Neural Operators (FNOs)
The FNO model (Li al., 2021a) is defined as follows: Ga(u)(z) == f(0; x) with
a (x) P(u: 0x)
We now introduce DONs and FNOs. More information about neural operators and the schematics of both DONs and FNOs are found in Appendix B.
a 1 (x) = (a(-))))) f(0: ((()
L [L +1] (4)
where the input function u has ran(u) C Rd. (a) R is the output of the FNO evaluated at output location x E Rd, (F(1 L+1 are nonlinear transformations with learnable parameters 0r- = [ pu+tll E IR" and which may contain operations the Fourier domain, P is an encoder that maps u and æ an ambient space of dimension d and has parameter vector 0, E R², and Q is decoder that maps the output from the block a (x) to scalar output with parameter vector 0g E Ry. The entire set of parameters for the FNO can be written as 0 = [ OF 0 With a slight abuse of notation, the FNO is simply written Go(u)( f(0; x:) in (4) when the input function u is known by the context.
3.1. Learning Deep Operator Networks (DONs)
The DON model (Lu et al., 2021) is defined as the inner product of two deep feedforward neural networks, each one with K output neurons. Given the branch net [ and the trunk net g {9x 1 the DON is
K Go(u)(y) = Σ ((00);)))(000)))))
where the input function u has ran(u) S Rd and E (o)(G)(w)) S Rd, is the output location on which the operator is evaluated. The training data is composed of n input functions (u(9) /= and qi output locations for each Gt (z))), i.e., (v, (0) 13= == with v/5² E IRd denoting the j-th output location for (z))). Each (() is repre- sented in R locations {-,},-- so that (x,) E IRd", r E [R). The entire set of learnable parameters is 0
The training data is composed of n input-output pairs {uu(9) G (z(())) } and a computational grid of eval- uations {x, J We let (() x0;,), denote the FNO model (4) with input function ( and evaluated at æ,. Then, the FNO learning problem is the minimization:
)R2
E argmin [(Go, G1)
(5)
107 0T E 1estts, where , IP'' and 0g E 18'6 are the parameter vectors of and g respectively.
(no)
P42
with empirical loss function
We only consider scalar input functions, i.e., du = 1. For each i E [n]. we stack (x,)), as an input vector to f. thus, f : TR& + K Note that : Rd + R Then, the DON learning problem is the minimization:
L(Go, Gt)
R
Glx))) (6)
Σ
n
R
θ (don)
E argmin & (Go, Gt) 0
and where Go Go(u (1)))
= f
4. Optimization Convergence Framework
where
We now establish two Conditinn--Conditin below-for the convergence of gradient descent (GD) when minimizing a loss function L. We show that as long as these two conditions are satisfied, the loss will decrease in value. In the following sections we show how the empirical losses used for training DONs (Section 5) and FNOs (Section 6), as in (3) and (6) respectively, satisfy these two conditions.
and 2
C (Go, Gt)
Goln (0) "Cd""))))) )
Qi
is the empirical loss function that measures the approxi- mation between Gg and GT, and where (o ΣK1 f& (0); (u(0) (x.)) gk (0g: (i)
We consider 0 L(0) be continuously differentiable. Let O IRP be suitable initialization point and (0:)22 be the sequence of iterates obtained by GD on loss L for some step-size iis > i.c.
Note that the ground truth operator G can either be explicit, e.g., integral of a function, or implicit, e.g.. the solution to a nonlinear PDE.
0.11 = 0.- - VoL(O))
(7)
3

---

Optimization for Neural Operators can Benefit from Width
We consider non-empty set B(O,) C IRP around and in- cluding 0g-
Theorem I's proof is found in Appendix C. We note that if the infimum loss inside B(Oa) is attained at time t, i.e., [(O.) @EB(O,)
inf (()), then there is nothing to prove-
Assumption 1 (Iterates inside B(O))). All iterates (0:h21 follow GD as in (7) and are inside the set (()))-
hence the conditional in the second sentence of Theorem 1. Remark 1 (The RSC to smoothness ratio). Theorem 1 re- quires / S 1, which needs to be proved for the particular function L being considered.
The first condition is based on the concept of Restricted Strong Convexity (RSC) being satisfied for L.
Definition (Restricted strong convexity (RSC)). A func- tion L is said to satisfy a-restricted strong convexity (a- RSC) w.r.t the tuple (S, 0) if for any O' E E S IPP and some fixed O E IR, we have
Remark (RSC and local strong convexity). If equation (8) were hold for any 0, 0' B(Oo), then C would be a locally strongly convex function in the set B((b) (Boyd & Vandenberghe, 2004). This is a stronger condition which makes in (8) independent from the choice of 0 (in the context of Theorem 1, a, would be independent from t). Moreover, it satisfies a/ < 1.
L
C (0')) 2 -(0)+ + ( O. VoL(()) + a 1 - -01-
2
(8)
with a > 0.
Our analysis is inspired by the recent works (Banerjee et al., 2023a) and (Cisneros- Velarde et al., 2025), where optimiza- tion guarantees were done for feedforward networks and normalization. We abstract out from those special cases, and demonstrate that our analysis works for any losses satisfying Conditions and 2-indeed, (Cisneros-Velarde et al., 2025) particularly satisfies Condition 1(..1) and (Banerjee et al., 2023a) satisfies Condition (bb.).. Thus, in the context of our paper, the largest effort in establishing optimization guarantees for DONs and FNOs is to show these two models satisfy Conditions and 2 with a S
Condition (RSC). Consider Assumption 1. At step I,
there exists non-empty set N₂ such that:
(a) No S B(0o):
(b) one these two conditions hold:
(b.l) Oct1 E N₁, with either O. & No or ((O.) + inføeN. (()).
(b.2) there exists some O' E N₁ such that L(0")
(():);
5. Optimization Analysis for DON
(c) L satisfies Rr-RSC w.r.f. (Ne, for some Or O.
We consider, analogous to (Liu et al., 2021b), the branch net as a fully connected feedforward neural network:
Note that L need not be convex for it to satisfy ag-RSC The second condition is based on the smoothness of L. Condition 2 (Smoothness). The function L is B-smooth in ( (.)). i.e. for any 0', 0 E (.O)). we have L(0') < (()) + (0' 0, VoC(O) ()) +
a u(x))
(0)
a (1)
1 w (1
1 (--11)
a
LE [6- (10)
/mll
2 1| 0| with 3 > 2
f a (L) 1
w (1-1)
aj aj
ymy
As long as Conditions 1 and 2 are satisfied at step t of the GD update in (7), the loss is guaranteed to decrease with a suitable size choice.
where with some abuse of notation u(x) [u(zi)..... u(æx)] is the vector of all scalar evalu- ations of u at each of the R locations, ó is a pointwise smooth activation function, of
Theorem 1 (Global loss reduction). Consider Assump- tion& I and Conditions 1 and 2 with at S G at step of the GD update (7) with step-size Tte 2e
(1) is the output at layer
E [L]. and the weight matrices are w-(") Rm and w/D Rm layer E (2...., L L- 13. The branch net has width my (all hidden layers have the same width). Similarly, the trunk net is a fully connected feedforward network:
3 for some We E (0, 2). If ((0,) # inf ((O), then we have OEB(0a)
inf. C(O) inf 2(0)
0 S 7t= C.0.) )))
BEN, ( ((0) < 1 and
(0)
[(0(+1) inf C(0)
BEB(O.)
a (1) ④b
W (oaaf-)) [E (11)
< (1 (1wi(1 - 76)
(2- wi) (C(O))
inf (())).
Mg
8
OEB(0.)
(9)
g = 0
(L)
walliaf--)) (1-11)
/mg
4

---

Optimization for Neural Operators can Benefit from Width
where y IRdy is the output location, and the weight ma- trices are (( E Rd, and w E IR","", at layer I E ( - 1]. The trunk net has width mg (all hid- den layers have the same width). Finally, we recall that we have K outputs on each network, i.e., / E IR x
needed for establishing the RSC condition for the analysis and does not change the computation of the optimization algorithm, which is simply GD run over all the branch and trunk network parameters.
Definition 2 (Q% sets for DONs). For an iterate O. (0 ftt 0 T and K E (0, 1). we define the set:
and Wh E R Given I E [L], we denote by (w (1)
y.k) and (wugh) the k-th row of the matrices w and w(()) respectively, and by w(t) and ( 9.tj their re- fai spective ij-entry. Using the notation in Section 3.1, the set of trainable parameters is A = 0 0 IT Res+Ps,
Q: = 0' = 10; T 0 1,'1" E cpttt::
coos(0' U., voGi)l 2 n,
with 0f = [ve)(WW TT ,...., ( and Og 9
(0, 0z1)²
1
Σ
n 1
E. Σ
Vo 1 "
Vonsk?? T) )
[vec ( ) ..... ( (W Let 0g be the parame- ter vector at initialization and 0 be it at time step t.
11
i=1
9i
x (0' - 0g.e) 2 0,
K
We make the following assumptions for our analysis: Assumption 2 (Activation functions). The activation func- tion & of the DON is 1-Lipschitz and Ba-smooth (i.e. o" S 30) for some B. > O.
(0) 0-,0² T
Vo, f
Yookks (
x (0', 0g.e)) 0, Vi [n], Vj E (n
(12)
Assumption 3 (Initialization of weights). All weights of
where VoG. = n - 1 E VoGo, (u(o (g))
"
क Cig = (ul,))))))))) G (1))))))) (1) ))2 and both ( Vo (0) and Vo,gk, are evaluated on 0--
the branch and trunk nets are initialized independently as
follows: (i) w fo.,, ~ ~N(0, 07,0) and w cg
(1)
~ NO, 22.0)
for 1 E [L- 1] where 0100- 2(14
ao
and 0.00
L
Vamy
We now prove the RSC and smoothness conditions (corre- sponding to Conditions and 2, respectively). Using the nomenclature of Section 4, the set B (0n) corresponds to B(O)), and B% := Q% n corresponds to N₁.
60
0 > 0; (ii) w fo and w (L)
(L)
E [K].
2(1+
Ving
go.k
Vamy
are random vectors with unit norms, ie., I/ Il l12 = and Il (1) are normalized as [|u(2))l22 = VR and || V Vay
BEue P.P. (0o) (Bo) (0,) with p2 > 0 P2
g0.k lla 1. Further, we assume the input to the branches
Theorem 2 (RSC for DONs). Consider Assumptions 2, 3, and 4, and Q' as in Definition with [ ll2 # O. Then, the set B := Q% n BB set that satisfies Condition 1(a) and (b) for suitable P2 and sufficiently small 1 + m, ). at step of GD, the DON loss least 1 2K 2KL( my
For given parameter vector 0- [0] 0 E RP/ we
p.P1 Bue (0.) n B (0t) is a non-empty f2
introduce the neighborhood set Euc (0) = 10 E RPstP= :
p.p:
[|w/)) - / 1l2 S p, ||w/"") g - ( 2| S p, E [L-
(0, 1). Moreover, with probability at
11. f.k
(L) v7 f.b ll2 S p1. [/ (L) - w "f ll I pr, E [K}}
(L)
A
9,4
L (3) satisfies equation (8) with
for radii p. > O. We say that an element of EEuc (0) is
strietly inside BFuc (0) when it satisfies every inequality in
Vmg (
the set's definition without equality. We also define Buc
04 = 2n= -
(13)
p
Miggg
as an Euclidean ball around 0 with radius > 0.
where 2 1 EE= VoGo, (ul)) (1) ()), and
The following is an assumption analogous to the general Assumption 1.
for some constant 6₁ > 0 which depends polynomially on the depth L,, and the radii p. pt, and p2 whenever og S pmmxal Jmay (Mmn)) Thus, the loss L satisfies RSC W.F. (B), 01), i.e. Condition I(c), whenever |V Vmill
Assumption& 4 (Iterates inside BEn (Oo)). All iterates (0:)22 follow GD as in (7) and are strictly inside the set p.p1 (0o) fixed p. P1 O.
26
We now focus on showing that the two conditions needed for optimization using GD as discussed in Section 4 are indeed satisfied by DONs. We start with the definition of a set Q% parameterized by K E (0, 1). which will help construct the set VV, in Condition for RSC. Due to the interaction of two where C2 >> 0 is a constant which depends polynomially on the depth L, and the radii p. pv. and p2 whenever oo S I
Theorem 3 (Smoothness for DONs). Under Assumptions 2 1 + 2 the and 3, with probability at least - 2 T my
DON loss L (3) is 3-smooth in BEue (0o) with 3 = cg K2,
neural networks (branch and trunk), the definition of Q' looks seemingly involved. However, note that Q; is only
- p maxx ymy Jmny }.
5

---

Optimization for Neural Operators can Benefit from Width
Remark 3 (Ensuring that 0g/3 < 1). As mentioned in
where is a pointwise smooth activation function, F is the discrete Fourier transform kemel (as matrix) with F being its conjugate transpose, the weight ma- trices are W() E R=xd W E Rmxm and R( e R=xm for layer E (2...., L. + 1) (all hidden lay- ers have the same width m). The ij-entries of w and R(()) are wij and rij (1(1) respectively, for an appropri- ate t. With some abuse of notation, we denote the en- tire set of trainable parameters by 0 = =ØT 0, with Ou = [ve)( y..... vee ( v vTjT and 0, [vec(R)(3)) ..... vec(H((+))) JTT. We denote the number of parameters by Pv + Pr. where O.r E IRP and 0, E R Let 0g be the parameter vector initialization and 0, be it at time step t.
Remark 2, in order use the optimization framework from Section 4, the statement of Theorem I requires an/B S
1. We prove that this condition is satisfied with a strict inequality for DONs in Proposition in Appendix D. D
Optimization Under Gradient Descent for DONs. We have that Theorem 2 satisfies Condition and Theorem 3 satisfies Condition 2. We also proved that 0z/9 / < 1. Thus, when &oGil = Q y Vimgg ), i.e.. 0l2 > O, a decrease on the loss function by GD is ensured with probability at least 1 2K
2 1 + myy f
1 towards its minimum value taken
within the set P.P1 (Bu) due to Theorem 1.
Remark 4 (The benefit of over parameterization for the RSC property). According to (13), |VooG,13 = 2 ymy + ymh is needed to ensure that 0₁ > 0, i.e., to ensure that the empirical loss & satisfies the RSC property at time t. Thus, as both widths my and my increase, L attains the RSC property at a lower value of I/VoG.1B
We remark that our model uses an m X m Discrete Fourier Transform kernel F, whose kjjentry is Fkj = e 2 (k-))--11,,, with representing the imaginary unit. Assumption 5 (Activation functions). The activation func- tion &b of the FNO is 1-Lipschilz and Sa-smooth (i.e. oj S for some 3. > 0.
D Remark 5 (Over-parane parererrramttiiiaiioon allows for a larger neigh- borhood around initialization). The condition du S 1 p m Smy' } (required for obtaining polynomial dependence on L for both RSC and smoothness parameters) implies p I min{ ( mmy, Myy since do must be positive. Thus, it is possible to increase the radius as we increase both my and mg. Thus, we can enlarge the neighborhood around the initialization point where our guarantees hold as
Assumption 6 (Initialization of weights). All weights of the FNO are initialized independently as follows: (i) (!) "0.1 ~ NN, )..) and o. ~ NN, 08,) for every appro-
priate I, where J0.00 2(1+ Vlemm
and Øo. 2(14
01,"
with 01,10 F1.r > 0: (ii) the decoder parameter v is a ran- dom vector with unit norm
|| = 1. Further, we assume the encoder output satisfies ( 1|2 = vd.
the widths increase.
For a given parameter vector 1 E RP= we introduce the neighborhood set BEnc
(0) {0 E R [[
6. Optimization Analysis for FNO
P...P..pp
(000/2 S Pwo E [L + 1], (| - R S Pr. I E {2, { L + 1}, 1|v -i--o vll2 S p1} for radii Pw: Pu.Pr.P1 > O.
As in the case of DONs, we also focus on scalar input functions u. To pass the imput function u, we discretize it by sampling it on R locations, forming vector of dimension R. Thus, the encoder P(u: 0p) (z:) in equation (4) takes a vector of dimension R+ d.. (Rt from the sampled and dz from the output location where we evaluate the operator on). For our purposes, we consider a fixed (not trainable) encoder with output dimension d: P(u: (z)) =/ (z))(a)= E IRd; and a linear decoder Q(xa (L+1); 0, (z)(
We say that an element of BEuc Pwoer.pp (0) is strictly inside Pt.
BEue (0) when satisfies every inequality in the set's
definition without equality.
The following assumption is analogous to Assumption 1. Assumption 7 (Iterates inside BEue
pow.pr.pp (0o)). (Bo))) All iterates
(0, (2.) follow GD as in (7) and are strictly inside the set BEne PP (0o) for fixed Pwr: Pe: P1 >> 0.
ym v ((L+)(()) E R with 0g = v E RT assuming a (z)) E RT.. Thus, following (Li et al., 2021a), the FNO model is:
We also introduce the following auxiliary set.
Definition 3 3Q2, sets for FNOs). For an iterate tt. let V,G, = Σ
(0) = (z)
= R Eÿ= VoGo, (z,)) For E (0, 1), define Q = (0 E RP- ||oo(( 0 Oz, voG.e) 2 s).
1
(1) = o
w a
m
a = $
W( a (1-1)
1
F* R RF"R0 Fal-t))
Note that unlike DONs, the O¹ sets for FNOs are relatively simpler due to a single network architecture
+
vm
1E (2...., L+ 11
Next, we prove the RSC and smoothness conditions (cor- responding to Conditions I and 2, respectively). Using the nomenclature of Section 4, the set Bue
f(0:z)= Vm
1
v T (L+1)
Pw Pr (0p) corre-
6

---

Optimization for Neural Operators can Benefit from Width
sponds to B(O)). and Bk Q% BBuc Peopoep (0 with P2 > O corresponds to Ni-
B (0c) P2
feedforward neural networks (FFNs), i.e., by being partic- ular instances of the general optimization framework from Section 4. Indeed, as indicated in Table I, there exist simi- larities and differences between our derivations for neural operators and the ones for FFNs.
Theorem (RSC for FNOs), Consider Assumptions 5, 6, and 7, and Q as in Definition 3 with || |VGrell + 0. Then, the set B == 2i n B
PusPr.p (0o) n B (0.) is non- empty set that satisfies Condition ((a) and (b) for suitable
p2- Moreover, with probability at least 2(L+2) at step m of GD, the FNO loss L (6) satisfies equation (8) with
The challenge in the analysis of DONs. The fact that the output of a DON is an inner product of two FFNs (1)-the branch and trunk networks makes the mathematical anal- ysis of the RSC and smoothness properties more involved than the analysis associated to a single FFN, Indeed, the appearance of cross- coos-inteeacctio terms between the two FFNs complicates the Hessian structure of the empirical loss C and requires more complex definition of the Q% set com- pared to the one used for FFNs or FNOs. On the other hand, since the branch and trunk networks are individually FFNs, their individual Hessian and gradient bounds are known.
04 = vx==2x21VVoAr
(14)
Vm
where VoG, - E for some constami 01 > O which depends polynomially on the depth L, and the radii pw, Pr Pt, and p2 whenever I,, + +01, S1 Pwtt-= Thus, the loss ((0) satisfies RSC /m
VoGo, (1z)) and
W.F. (B. 0.), Le., Condition I(c), whenever Vo 21
Theorem 5 (Smoothness for FNOs). Under Assumptions 5 2(L+2), the FNO loss C (6) is B-smooth in B Prop (0u) with 3 being positive constant which depends polynomially on the depth L,, and
The challenge in the analysis of FNOs. The fact that FNOs unOskekliik FFNs-include a series of learnable trans- formations in the Fourier domain makes the mathematical analyses of their Hessian and gradient bounds more involved than the ones for FFNS. Indeed, these Fourier transforma- tions introduce cross-derivatives between weights in data and Fourier domains in the Hessian that need to be carefully taken into account. On the other hand, since FNOs are com- posed of single network, their Q% set is similar to FFNs, as well as their RSC and smoothness analyses.
and 6, with probability at least 1
the radii Pw, Pr, and P1 whenever 11, +01, S 11- tu
vm
Remark 6 (Ensuring that az/3 < 1). Similar to our discus- sion in Remark 3, we prove that a < 1 in Proposition 3 from Appendix E, satisfying the condition required in the statement of Theorem 1.
Optimization Under Gradient Descent for FNOs. We have that Theorem 4 satisfies Condition and Theorem 5 satisfies Condition 2. We also proved that 0 < 1. Thus, when V = 2 ),), i.e., a₁ > 0, decrease the loss function by GD is ensured with probability at least 1 2(L+2) towards its minimum value taken within the set Pu.P. (0o) due to Theorem 1.
8. Experiments
We present experiments on the effect of over- parameterization on the training performance of DONs and FNOs, measured by the empirical risk over a mini-batch of the training dataset using the Adam optimizer. We consider three prototypical operator learning problems in the litcrature (Li et al., 202 la; Lu et al., 2021): (a) the an- tiderivative (or integral) operator, (b) the diffuuion-reacction operator, and (c) Burgers' equation. We do not consider vector-valued problems (e.g., Navier-Stokes) because they are not covered by our theoretical framework. For definiteness, we consider the branch and trunk nets to have the same width m (i.c., my = my = m) for the DON, and the same width m for the FNO. In all experiments, we increase the width from m = 10 to m 500. For all networks, we use the Scaled Exponential Linear Unit (SELU) (Klambauer et al., 2017) as their smooth activation function. We monitor the training process over 80, 000 training epochs and report the resulting average loss. Note that the objective of this section is to show the effect of over paameteriization on the neural operator training and not to present any kind of comparison between the two neural operators.
Blue
Remark 7 (The effects of over- parameerrizaaiionn for FNOs). Similar observations to Remarks 4 and hold for FNOs, i.e.,
that over-vveaaatee pee-paraamterrization ensures (i) a better condition for ensuring the RSC property, and (ii) a larger neighborhood around the initialization point over which our guarantees hold. Item (ii) follows from the relationship Pu + Pr I vm obtained when choosing 01, and oi, to ensure polynomial dependence as in Theorems 4 and 5.
7. Comparison between Neural Operators and Feedforward Neural Networks
Our presented analysis provides sufficient conditions that guarantee the optimization of DONs and FNOs under gradi- ent descent (GD). It is non-tre soivialiial that GD should converge for neural operators in a similar way to how converges for
7

---

Optimization for Neural Operators can Benefit from Width
Deep Operator Network
Fourier Neural Operator
Q% set
More Complex
Similar
Hessian and gradient bounds the neural operator model RSC and Smoothness characterization; computing the Hessian of C
Similar
More Complex Similar
More Complex
Table 1: We indicate whether specific neural operator (DON or FNO) has similar or more complex derivation of specific mathematical objects or properties compared to a feedforward neural network (as in (Banerjee et al., 2023a)). *The similarity is with respect to each individual network of the DON.
10
10
10
10'
10
177
10"
10-
107
(a) Antiderivative
(b) Inffusion-Reaction
(c) Burgers' Equation
Figure 1: Training progress of DONs measured by the empirical loss (2) over 80,000 epochs. The y-axis is plotted log-scale and the a-axis denotes the training epochs % 100 (i.e., the loss is stored every 100 epoch). Wider networks typically lead to lower loss for all three problems.
11- 201
200
07
10"11
10-11
10
107
do Epecrs
600
de
300
Epodi
(a) Antidcrivative
(b) Diffusion-Reannn Reaction
(c) Burgers' Equation
Figure 2: Training progress of FNOs as measured by the empirical loss (5) over 80,000 epochs. The setting of the plots is similar Figure 1. Wider networks typically lead to lower loss for all three problems.
The results for DONs (Figure 1) and FNOs (Figure 2) clearly show that both neural operators benefit from an increasing width m since it leads to overall lower training losses for all three learning problems and it generally leads to faster optimization convergence. The Antiderivative operator is linear operator and therefore is learned very accurately, especially for wider DONs and FNOs where the loss is around 10 and 10 respectively. The diffusion-reaction equation demonstrates lower loss with increasing width less markedly than the antiderivative operator for DONs and more markedly for FNOs. This could be attributed in part to the fact that the operator is inherently nonlinear. Finally, regarding Burgers' equation, lower training losses and faster convergence is more markedly for FNOs than for DONs as the width increases.
Additional information on the experimental settings and additional experiments are found in Appendix F,
9. Conclusion
We present novel optimization guarantees for gradient de- scent for neural operators with smooth activations: Deep Operator Networks and Fourier Neural Operators. Our guar- antees are based on the restricted strong convexity and smoothness of the loss, thus providing an encompassing framework to neural operator optimization. We argue that increasing the width of the neural operators benefits our the- oretical guarantees. We also present empirical evaluations on prototypical operator learning problems complement our theory.
8

---

Optimization for Neural Operators can Benefit from Width
Acknowledgements
Bhattacharya, K., Hosseini, B., Kovachki, N. B., and
Stuart, A. M. Model Reduction And Neural Net- works For Parametric PDEs. The SMAI journal of computational mathematics, 7:121-157, 2021a. ISSN 2426-8399. doi: 10.5802/smaiijem.77. URL
Part the work was done when Pedro Cisneros- Velarde was affiliated with the University of Illinois Urbana- Champaainn current
and was concluded during Pedro Cisneros-Velarte affiliation. The work supported in part by the National Science Foundation (NSF) through awards IIS 13,,, OAC 21-30835, DBI 20-21898, as well as a C3.ai research award.
http://mmaaaicccccmeeeeeeeeeeeee atticles/10.5800 2ttp://00..000/mmmm1aaaaaa ssss/jmmmiiii.maam 74/.
Bhattacharya, K., Hosseini, B., Kovachki, N. B., and Stu- art, A. M. Model Reduction and Neural Networks for Parametric PDEs. arXiv:2005.03//8 [cs, math, stat], June 2021b. URL h /arxiv.ivvvvv..////a org/ wtw/abs////ooooooooo//..oo....er 2005.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. Our work is theoretical analysis of already existing algorithms, therefore, we do not believe there are potential societal consequences that must be specif- ically highlighted.
03180. arXiv: 2005.03180.
Boyd, and Vandenberghe, L. Convex Optimization. Cam- bridge University Press, 2004.
Centofanti, E., Ghiotto, M., and Pavarino, L. F. Learning the hodgkin-huxley model with operator learning techniques. arXiv preprint 2it.::2000000272.. 2024.
References
Allen-Zhu, Z., Li, Y., and Song, Z. A Convergence Theory for Deep Learning via Pareetrratttiiittionn. Technical Report arX.v:1811.03962, arXiv, June 2019. URL http::/arxixt wwwpxiivooor....o wwwooorrbaa....oooooooeooooooeoe abb/1811. 03962. arXiv:1811.0396) [es, math, stat] type: arti- cle.
Chen, T. and Chen, H. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911-917, 1995.
Cisneros- P., Chen, Z., Koyejo, S., and Banerjee, A. Optimization and generalization guarantees for weight normalization. Transactions on Machine Learning Re- search, 2025. ISSN 2835-8856.
Arora, S., Du, S., Hu, W., Z., and Wang. R. Fine-graine graingr analysis of optimization and generalization for overpa- rameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322-332. PMLR, 2019a.
Cybenko, G. Approximation by superpositions of sig- moidal function. Mathematics of control, signals and systems, (4))303-311, 1989.
Arora, S., Du, S. S., Hu, W. Li, Z., Salakhutdinov, R. R., and Wang, R. On exact computation with an infinitely wide neural net. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019b. URL
Deng, B., Shin, Y., Lu, L., Zhang, Z., and Karniadakis, G. Approximation rates of deeponets for learning oper- ators arising from advection ddifuusioon equations. Neural Networks, 153:411 1-42,, 2022. ISSN 0893-6080. doi: https://doi.org/ wtt.r//////////// n.e.e
neurips oe wwtpp/ppe///19//ii..... dbc4d8 boooddeacccee2222aaaeeeeeeeeeeee pdf.
Diab, W. and Al Kobaisi, M. U-deeponet: U-net enhanced deep operator network for geologic carbon sequestration. Scientific Reports, 14(1):21298, 2024.
Paper..
Banerjee, A., Cisneros- P., Zhu, L., and Belkin, M. Restricted strong convexity of deep learning models with smooth activations. In The Fleventh International Conference on Learning Representations (ICLR), 2023a.
Du, S., Lee, J., Li, H., Wang, L.,, and Zhai, X. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pp. 1675- 1685. PMLR, 2019.
Banerjee, A., Cisneros-Velarde, P., Zhu, L., and Belkin, M. Neural tangent kernel at initialization: linear width suf- fices. In Evans, R. J. and Shpitser, I. (eds.), Proceedings of the Thithh Conference on Uncertainty in Artifi- cial Intelligence, volume 216 of Proceedings of Machine Learning Research, pp. 1 110-116 118. PMLR, 31 Jul-04 04 Aug 2023b.
Goswami, S., Yin, M., Yu, Y., and Karniadakis, G. A physics-infomee informed variational DeepONet for predicting the crack path in brittle materials. Computer Meth- ods in Applied Mechanics and Engineering, 391:1 14587, March 2022. ISSN 00457825. doi: 10.1016/j.cma cm2.222...222. 114587. URL http: /ttp::/arre /ttp///avvvvv///v///////e.e/..av v. org/ abs 2108. 06905, arXiv: 2108.06905.
9

---

Optimization for Neural Operators can Benefit from Width
Harder, P., Hernandez-Garria, A., Ramesh, V., Yang, Q.. Sattegeri, P., Szwareman, D., Watson, C., and Rolnick, D. Hard-constrained deep learning for climate downscaling. Journal of Machine Learning Research, 24(365):1-40 2023.
Graph Neural Operator for Parametric Partial Differen- tial Equations. arXiv:2006.09335 [cs, math, stat], Octo- ber 2020a. URL http: pett:://arxti /arxiv. wwwp/ogg///////oo//t/t..o/...s/t 2006. 09535. arXiv: 2006.09535.
Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhat- tacharya, K. Stuart, A., and Anandkumar, A. Neu- ral Operator: Graph Kernll Network for Partial Differ- ential Equations. arXiv:2003.03445 [cs, math, stat), March 2020b. URL http: / www.voroooooo.oooooooooooooooooe /abs/
Hornik, K. Approximation capabilities of multilayer feed- forward networks. Neural networks, 4(2):251-257, 1991.
Hornik, K. Stinchcombe, M., and White, H. Mul- tilayer feedforward networks are universal approxi- mators. Neural Networks, 2(5):359-366, January
2003.03483 03485. arXiv: 2003.03485.
1989. ISSN 08936080, doi: 10.1016/0893-6080(89) 90020-8. URL URL.htttss://1nkkiingubbbb sevier.. com/ com/retrieee http://teteeeeeep///033330009900
Li, Z., Kovachki, N., Arizzadenesheli, K., Liu, B., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Fourier Neural Operator for Parametric Partial Differential Equa- tions. arXiv:2010.08899 [cs, math), May 2021a. URL http://arxia www.awwwwwwww..aaaaooooooooooeee hts//bbs//2// 08895. arXiv: 2010.08895.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent ker- nel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. Markov Neural Operators for Learning Chaotic Sys- tems. arXiv:2106.06888 [cs, math), June 2021b. URL http:/ htt///xivvvvooo www.:/rgg//bb//////e/e. aaa/ab///11/0 06898. arXiv: 2106.06898.
Karimi, H., Nutini, J., and Schmidt, M. Linear conver- gence of gradient and proximal-gradient methods under the wojakkloojssewicz.ccooooo condition. In Joint European Con- ference on Machine Learning and Knowledge Discovery in Databases, pp. 795-811. Springer, 2016.
Li, Z.,, Huang, D. Z.,, Liu, B., and Anandkumar, A. Fourier neural operator with learned deformations for pdes on general geometries. Journal of Machine Learning Re- search, 24(388):1-26, 2023.
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. Self-normalizing neural networks. In Proceedings of the 31 International Conference on Neural Informa- tion Processing Systems, NIPS' 17, pp. 972-981, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860999
Liu, B., Kovachki, N., Li, Z., Azizzadenesheli, K., Anand- kumar, A., Stuart, A., and Bhattacharya, K. A learning- based multiscale method and its application to inelas- tic impact problems. Journal of the Mechanics and Physics of Solids, 158: 154:104,,, January 2022a. ISSN 00225096. doi: jmps016/.jmps..2211..0 http///xxivvv.////// www.xxi..og//bba... wtt.oo/ggaaba/..oaeaeeoo/ 2725. 2102.07256.
Kontolati, K., Goswami, S., Shields, M. D., and Karniadakis, G. E. On the influence of over wer-pprrameterization in mani- fold based surrogates and deep neural operators. arXiv preprint -xXi:22233.0077 2022.
URL
arXiv:
Kovachki, N., Lanthaler, S., and Mishra, S. On universal approximation and error bounds for fourier neural opera- tors. The Journal of Machine Learning Research, 22(1): 13237-13312, 2021.
Liu, C., Zhu, L., and Belkin, M. On the linearity of large non-linear models: when and why the tangent kernel is constant.
arXiv preprint arXiv:2010.01092, 2021a.
Liu, C., Zhu, L., and Belkin, M. Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. arXiv preprint arXiv:2003.00300 2021b.
Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Re- search, 24(89):1-97, 2023.
Liu, C., Zhu, L., and Belkin, M. Loss landscapes and opti- mization in prraaaateteri non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 2022b.
Kutz, J. N. and Brunton, S. L. Promising directions of machine learning for partial differential equations. Nature Computational Science, 4(7):483-494, 2024.
Lu, L., Jin, P., Pang, G., Zhang, Z., and Karni- adakis, G. E. Learning nonlinear operators via deep- onet based on the universal approximation theorem
Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhat- tacharya, K., Stuart, A., and Anandkumar, Multipole
10

---

Optimization for Neural Operators can Benefit from Width
of operators. Nature Machine Intelligence. 3(3): 218 229, Mar 2021. ISSN 2522-5839. doi: 10.1038/ $42256-021-00300-55 URL http://dde doi. 10. 1038
with physics-informed DeepOnets. arXiv:2103.10974 [cs, math, stat], March 2021b. URL htttt:///a/rtee org/ abs 20b//2103.11 10974. arXiv: 1103.10974.
Wen, G., Li, Z.., Arizzadenesheli, K., Anandkumar, A., and Benson, S. M. U-FNO-An enhanced Fourier neural operator-based deep-leaming model for multiphase flow. Advances in Water Resources, 163:104180, May 2022.
Lu, Z. Pu, H., Wang, F., Hu, Z. and Wang, The Expressive Power of Neural Networks: A View from the Width. In Guyon, I,, Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc., 2017. URL https: httppsss/ppccced.ngs.. neurips wwworip..ccpaaee. / 32051687800211777777777771333333
ISSN 0309-1708. doi: 10. 10.10166.aadwaarree2222.0.011110 URL https://ww..ccieneedireee acderce//rricdee//////8882202200
Yang, Q. Hernandez-Garcia, A., Harder, P., Ramesh, V., Sattegeri, P., Szwarcman, D., Watson, C. D., and Rolnick, D. Fourier neural operators for arbitrary resolution cli- mate data downscaling, arXiv preprint arXiv:2305. 14452, 2023.
pdf.
Pathak, J. Subramanian, S., Harrington, P., Raja, S., Chat- topadhyay, A., Mardani, M., Kurth, T., Hall, D., Li, Z Azizzadenesheli, K., Hassanzadeh, P., Kashinath, K., and Anandkumar, A. FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators. 0rXiv:2202.11214 [physics], Febru- ary 2022. URL http::///aaxiw...ga/b//////s//// 11214. arXiv: 2202.11214.
Qin, S., Lyu, F., Peng. W., Geng, D., Wang. J., Gao, N., Liu, X., and Wang, L. L. Toward better understanding of fourier neural operators: Analysis and improvement from spectral perspective. arXiv preprint arXiv:2404.07200, 2024.
Qiu, Y., Bridges, N., and Chen, Derivative-enhanced deep operator network. arXiv preprint arXi::2402.19242, 2024.
Sun, Y. Moya, C., Lin, G., and Yue, M. Deepgraphonet: A deep graph operator network to learn and zero-shot trans- fer the dynamic response of networked systems. IEEE Systems Journal, 2023.
Tran, A., Mathews, A., Xie, L., and Ong. C. S. Factorized fourier neural operators. arXiv preprint arXiv:2111.13800 2021.
Wang, S. and Perdikaris, Long-time integration of para- metric evolution equations with thysics-informed Deep- ONets. arXiv:2106.0533 [physics], June 2021. URL http:///rxe 2106.05384.
05384. arXiv:
Wang, S., Wang, H., and Perdikaris, P. Learning the solution operator of parametric partial differential equations with physies-infommed deeponets, 2021a. URL https:// arxiv. abs /200s//133..m 109744
Wang, S., Wang, H., and Perdikaris, P. Learning the solu-
tion operator of parametric partial differential equations

---

Optimization for Neural Operators can Benefit from Width
A. Related Work
Learning Operators. Constructing operator networks for ordinary differential equations using learning-basee approaches was first studied in (Chen & Chen, 1995), where a neural network with a single hidden layer was shown to approximate a nonlinear continuous functional. This was, in essence, akin the Universal Approximation Theorem for classical neural networks (Cybenko, 1989; Hornik et al., 1989; Hornik, 1991; Lu et al., 2017). While this theorem only guaranteed the existence of a neural architecture, it was not practically realized until (Lu et al., 2021) provided an extension of the theorem to DONs. Since then, several works have pursued applications of DONs to different problems, c.g., (Goswami et al., 2022; Wang & Perdikaris, 2021; Diab & Al Kobaisi, 2024; Centofanti et al.. 2024; Sun et al., 2023), as well as improved the DON model itself, e.g. (Wang et al., 202 Qiu et al., 2024). From a standpoint of generalization. Kontolati et al. (2022) studied the effects of over- peeeeerramttrinntiooon on the generalization properties of DONS in the context of dynamical systems. Nonetheless, an optimization analysis of DONs an open problem.
The operator learning paradigm has also been explored in parallel by other works seeking to directly parameterize the integral kernel in the Fourier domain using a deep network (Bhattacharya et al., 2021b;a; Li et al., 2021a; 2020b; 2021b). Several subsequent extensions explored different architectures for Fourier-based operators tailored to specific problems (Li et al., 2020a; Liu et al., 2022a; Wen et al., 2022: Pathak et al., 2022; Centofanti et al., 2024). Other notable techniques include the use of a factorized spectral representation (Tran et al., 2021) and employing FNOs in latent space in an encoder-decoder framework (Li et al., 2023). We also mention that Qin et al. (2024) noticed that FNOs have good performance for learning low-frequency information from PDE data, but are less effective on high-frequency information. Thus, they proposed using an FNO ensemble that induces the neural operator to focus on learning the latter. Recently, FNOs were used to accelcrate simulations in climate science (Yang et al., 2023; Harder et al., 2023). Nevertheless, while significant progress has been made for FNOs from an applied perspective, their formal optimization analysis is an open problem.
Optimization Analysis of Neural Networks. Optimization of wee-parameterized deep neural networks has been studied extensively, e.g., (Du et al., 2019; Arora et al., 2019b;a; Allen-Zhu et al., 2019; Liu et al., 2021a). In particular, Jacot et al. (2018) showed that the NTK of a deep network converges to an explicit kernel in the limit of infinite network width and stays constant during training. Liu et al. (2021a) showed that this constancy arises due to the scaling properties of the Hessian of the predictor as a function of network width. Banerjee et al. (2023b) showed that a deep network whose width is effectively linear on the sample size can ensure convergence under appropriate initialization. Du et al. (2019) and Allen-Zio et al. (2019) showed that GD converges to zero training error in polynomial time for deep over wer-parameterized models such as ResNets and CNNs. Karimi et al. (2016) showed that the Polyak- Lojasi-Lojaieeiizz (PL) condition, a weaker condition than strong convexity, can be used to explain the linear convergence of gradient- methods. Banerjee et al. (2023a) showed convergence of GD for feedforward networks using RSC, which leads to a variant of the PI. condition. Cisneros- Velarde et al. (2025) used RSC to prove the optimization of networks with weight normalization using GD.
B. Additional Information on Neural Operators
B.1. Learning Operators
We briefly outline the notion of learning for neural operators (Li et al., 2021a; 2020b; Lu et al., 2021). Consider two separable Banach spaces, the input space L/ and the output space V, and a possibly nonlinear operator G l - V.
The standard operator learning problem seeks to approximate G by a parametric operator Go : + V that depends on the parameter vector 0 E ④ defined over some parameter space θ This done by proposing an optimization framework where we learn vector 0t E θ that "best" approximates G in some sense. Given observations (u(;)) E l and (u())) E V where u (3) ~ μl μ. = 1,..., n, is an i.i.d sequence from the probability measure μ supported on U, we take 0f as the solution of the minimization problem
@t argmingee Eusy [C (Ge(t)
(15)
where C is a suitable cost functional that measures the discrepancy on the approximation between the operators (u) and Gi (u) for a given u E U. This optimization problem is analogous to the notion of learning in finite dimensions, which is precisely the setup for which classical deep learning used.
12

---

Optimization for Neural Operators can Benefit from Width
B.2. DON Architecture
The schematic for the Deep Operator Network's architecture is presented in Figure 3.
Branch Net: f(0,:u)
mT, width of the branch net
Input Functions
m width of the trunk net
m
MWW
"inner product"
g
K
Co(u)(v)
fx ();x
E
f
0 ER²
Output Locations
0g
0g ER
9K
Trunk Net: 3(O,:3))
Figure 3: A schematic of the DON architecture by Lu et al. (2021) used in our study. We refer to the notation used in our paper. Note that the input functions need not be sampled on a structured grid of points.
B.3. FNO Architecture
schematic for the Fourier Neural Operator's architecture is presented in Figure 4.
Input Functions
Output
Specs cumy (gel))
Spue, cubV (wis)
Spuc. DEDY (s90)
Bypase (1i9))
Hlypas conv (1rtt)
Hypone COnS (1))))
Block
Block
Block
Sampled on locations Tor (n
Encoder (o))
Fourier Blocks
Decoder
(0,)
Spec. conv (Spectral Convolution)
Bypass convolution w a (1-1)
(* (--1)
Figure 4: schematic of the FNO architecture by et al. (2021a) used in our study, We refer to the notation used in our paper. "Spectral convolution" and "bypass convolution" are terms used the FNO literature to denote the effect of the linear mappings in the spectral and spatial domain, respectively.
13

---

Optimization for Neural Operators can Benefit from Width
C. Optimization Convergence Analysis for Section 4
establish relevant results for Section 4. Our analysis follows very closely the recent work by Banerjee et al. (2023a) and generalizes it. We now provide all the relevant proofs.
We start with the following lemma which shows that Condition implies form of restricted PL condition.
Lemma (Restricted PL). Assume Condition I is satisfied. Then, L satisfies a restricted form of the Dolyak-Lojasiewic (PL) condition w.r.f. (Ne, 0z):
L(O.) inf L(0) S VoL(0.)
(16)
BEN.
20g
Proof. Define
(0) = [(O,) + (0- 0. VeC(O.))
di 110 0.112
By the Rs-RSC property of Condition 1(c), AQ' E NN, have
[(0') 2 Lo.
(17)
Further, note that Lo, (.) is minimized at o.41 = 0. VoL(Oe)/oz and the minimum value is:
inf La, (0) = Lo. (0.(0) = L((,) 200 OER
[Vo((,))
Then, have that
inf (0) inf čo, (0) C(O,)
1
Voo((.))1
(18)
BEN:
BER*
204
This means that that Lo, (-) lower bounded by the expression on the right-hand side of (18) and we can take the infimum over Nr on both sides of (17) and obtain
inf (()) inf (0).
(19)
eEA
eeN.
Finally, we obtain the expression in (16) by using both inequalities in (18) and (19) and rearranging terms.
Next, we show that the restricted PL condition on N₁ in Lemma 1 along with smoothness (Condition 2) can be used to show a loss reduction on No.
Lemma 2 (Local loss reduction). Assume Conditions and 2 with a₂ I B at step t of the GD update as in (7) with step-size 1/s 3 for some we E (0,2). 2). Then, we have
[(0++1) inf L(0) < (1- area (2 -wi) (C(O.) C(0))
inf nEN;
(20)
OEN,
Proof. Since L is 3-smooth by Condition 2, we have
3 1 - 0,112
C(0,+1) < ((0.) + (0 0s VoC(O.)
= C(O.) - ml VoL(O,))1 +
(eL(O,)
(21)
2
= L(.)) - 'Nt (1 (1- 37
|vo[(0))
Since ar > by assumption, from Lemma we obtain
Vo -2a(((()))
inf L(0))
0EN:
14

---

Optimization for Neural Operators can Benefit from Width
Hence
[(0++1)
inf. OEN
((0) - ((0))
inf (
7)
1
(Vo((0))
BEN,
2
(4)
8TT
I [(0,)
inf [(0)
1
20,(C(0,,) inf (())) OEN:
BEN:
2
1 20
81
1
(C(O,)
inf L(0))
2
BEN;
where (a) follows for any th 5 because this implies
Bri 2
2 0. Choosing The # wi E (0,2),
(()) - (1
Orgiah
(2 wi) (C(0.) inf C(0))
((0++1) inf
BEN,
BEN:
This completes the proof.
Finally, we show that the local loss reduction result in Nr from Lemma 2 can be extended to show loss reduction in B(0a),
which is the main optimization result.
Theorem 1 (Global loss reduction). Consider Assumption 1 and Conditions and with S B at step t of the GD update (7) with step-size πι = we
3 for some We E (0,2). If ((),) * inf L(O), then we have S 7t = OEB(0,)
Inf ((0) inf (())
BCN+ ecB(
((0) < 1 1and
((0,)- @cBres)
((0++1) inf L(
OEB(O,)
< 1 anwe( 1 71)
B (2 ww) (C(0.)
(9)
inf (())) (c
infagN, [(0) inf C())
Proof. We start by showing Y [(0.)
((B(0))
satisfies 0 S 7t- < 1. First of all, we note that this quantity is
inf (())
well-defined because we assuming that L(O,) * inf [(0), i.c., that the current iterate does not attain the minimum
(E
loss. The fact that % 2 0 follows immediately from inf (( < L(0,) and inf C(0) - inf (()) since M₁ S ((())) OEB(O,)
eeN.
by Condition I(a). Now, there are two ways to prove that % < 1 depending on whether we consider Condition 1(b.1) or Condition 1(b.2).
We start by considering Condition (bb. and prove by contradiction that % < 1. Assume that 7 1, i.e. inf L()) > [(O,). Then, we note that
eeN:
(4) inf C(0) < C(O..1) < ((O.) (1 oEN,
(6)
BT% 2
(c)
2 inf L()) 7) (1 BTg 2
I VoL(O))
(oo(
(22)
where (a) follows from Oct1 E Nr. (b) from (21), and (c) from 7t > 1. Then, comparing the leftmost and rightmost
inequalities in (22), we must have |/))(()))l = 0 since 1/:
fn. 2
> 0 because of # with wt E (0, 2). Now, L(0); thus, we analyze both cases.
when considering Condition I(b.1), we either assume that & Nr or that C(O.) + inf BEN,
Assuming 0 & N₁: Note that |)oC(0,)) = implies (oC(O,) = 0p (i.e., the gradient evaluated at O, is the zero vector), which then, due to GD (7), implies 0, 0.41. Since we had 0,+1 E NN this then means that O, E M contradiction our assumption.
iii) Assuming ((0.) * inf L(0): Note that (())()) = 0 implies that all the inequalities in (22) are also equalities. BeN. This then implies that C(..) inf L(0)-a contradiction to our assumption. OEN,
15

---

Optimization for Neural Operators can Benefit from Width
In either case (i) (ii), our proof by contradiction finally shows that 7 1.
We now consider Condition 1(b.2) with the element O' E Ne as described in the condition's statement. We immediately have that 0' satisfies inf (()) - (()') < ((),), which then implies DENi
< 1.
Having shown that " according Condition 1(b), we now proceed to prove equation (9). We consider two cases: (A) Tt 0 and (B) = 0.
We start by considering Case (A), which holds if and only if inf L())
inf (()). We now define 6, == @EB()))
DEN.
((0,)- ((0) and note that 6, E (0, 1) since o = Now, with wa E (0, 2), we have ((0,) lof 203
[(0++1) inf (()) = [(0++1) inf [(0) + inf L()) inf C()) OEB(0.)
DEN,
OEN, @EB((o)
(a)
Ontwn (2 - wi) (f((.)
inf OEN:
((0) + ( inf [(0) inf
(()))
BEN,
BEB(0o)
o (2 - wi) (,(((())
inf (EB(O.)
(())) + (1 - (C(,))
inf ())
B
OEB(Oo)
(
att (2 wi)(1 - -1) (C(0,)
inl ( (E
which is (9), and where (a) follows from Lemma 2 and (b) follows from
(I ord't
3 (2 (2-)) + (1 56) B
aywr
(2 we)de = agwe (2 wi)(1 71)-
We now consider Case (B), i.e., 7/- = 0, which holds if and only if inf L(0)
inf [(0). Then, we have BEN. OEB(O,)
[(0(41)
inf (()) = [(0++2) - inf L())
0EB(0,)
BEN.
(a)
1 anwi (2 w/) (E(0,) inf [())) eEN,
ande (2 wi) (C(0.) 0EB(O))
inf (())).
which is (9) when y = 0, and where (a) follows from Lemma 2. This completes the proof.
D. Analysis for Deep Operator Networks
D.I. Bounds on the Hessian, Gradients and the Predictor
The convergence analysis makes use of the gradients and Hessians of the empirical loss with respect to the parameters O, namely,
H Hgf
H Hgg
VoL(0) = [T Vo, and v2c(0)= = H (0) =
(23)
where Vo, aL(0)/00 E RR and Vo, (()) = aC(0)/00, RR., Note that we make use of the notation Vo, to denote the derivative with respect to the parameters O, and this is not a functional gradient. Similarly, the individual blocks in the 2 x 2 block Hessian H are given by
8C
HGs = HJ = 8C
8CC Hgg = V2, C=
H, = c=
Hyy=Vo,c-
H = 00
(24)
00, 2'
d0 00,00,'
a0, 2
where Hss E IRIIP!, Hgg E RPSP,, Hig E RPiPP, Hgg E RPEXPS and the argument 0 is ignored for clarity of exposition. Using (3) and rewriting the derivatives in (23) and (24), recalling that b,, = (Go (G(()))) (t())) (3)
16

---

Optimization for Neural Operators can Benefit from Width
we get
ac
91
K
3
Ve:
(
and
00,
0c
g
71 ==1
1
48
K
k
V
0
9k.3
(25)
1=1
j=1
1
9
K
2
K
00, 2
-
9
S Vo,
100 Vor
110T
1=1
j=1
k=1
j=1
00, 2
11
91
V
9k.
1
9
Σ
Sk Sk
(3) Vo, ko
Vo, 9K's (0)
(26)
9
71
4i
k=1
71
j=1
k,W==1
11
98
K
K
Vo
k
(8)
T
Σ
Vay 1k Vo Va, gh (0)
00 000
71
4:
for the individual blocks of the Hessian (23) where we make use of the notation f( = fx :((0):ur) (1) and gh. = 9k (0 v
In the rest of the paper, with some abuse of notation, we also make use of the implicit notation f (0f) (0,; u ()) and
( g& (0.g y ).
In order to prove the RSC and smoothness properties of the empirical loss L, we need to upper bound the spectral norm of its Hessian. As can be seen above, the gradient and Hessian of the predictors (i.e., the branch l( and trunk Øk.j networks, E [K]. j E [g:]. E [n]) appear in the Hessian of L, and thus, we will eventually need the upper bound of their norms. For this, we will make use of the next lemma.
Lemma 3 (Bounds on the predictors). Under Assumptions 2 and and for O BBuc (0o), with probability at least 1 2KL
1 n we have for every & E [K], i E [m], j E (ai).
my
cl))
5 c0
Vo, ((0
S /mn
and
18.0k
(1)
mmg
Vo,
(1)
112 S e
and
Vo,
(6)
S $g4 (10)
(27)
A1, S and
1g S X2
where clf), cla), gi gia), X2 and 22 are suitable constants that depend on ou, the depth L and the radii p. P1. The depen- dence of the constants reduces to the depth and the radii and becomes polynomial whenever do S p pmmxx Vnyz yingl
Proof. The proof follows from a direct adaptation of Theorem 4.1 and of both the statement and proof of Lemma 4.1 in (Banerjee et al., 2023a) to our setting. Indeed, these results show that c(f), g(f), and a₁ depend on Jo. the depth L and the radii P. P1; and that such dependence reduces to the depth and the radii and becomes polynomial whenever Tu I 1 P
.m A similar dependence is obtained for the constants cl g(و), and A2 whenever Tp 5 - The last statement in m
Lemma 27 follows immediately. Finally, since the bound for single branch network output holds with probability least 1 2L and for single trunk network output holds with probability least 1 2L then in order for these bounds to hold my
m, for the K outputs of all predictors, we obtain the overall probability using De Morgan's 's law and union bound.
D.2. RSC and Smoothness Results
Using the results from the previous section, we derive the RSC and smoothness results.
Theorem 2 (RSC for DONs). Consider Assumptions 2. 3, and 4, and Q¹ as in Definition with [|VoG1l2 + O. Then. the set B = Q% O Buc (0n) O BEuc (0.) a non-empty set that satisfies Condition ((a) and (b) for suitable p2 and p.m
02
17

---

Optimization for Neural Operators can Benefit from Width
sufficiently small (0, 1). Moreover, with probubility at least -2KL 1 satisfies equation (8) with
1), my
step I of GD, the DON loss L (3)
02 22=2n2 K2
(13)
m
mg
where VoGi 1. VoGo, (n (1,) ), and for some constant which depends polynomially on the depth L, and the radii P1. and P2 whenever 0y S paxx 1, i.e., Condition 1(c), whenever | = 22( ymj + ymg
mmyy vingh }. Thus, the loss L satisfies RSC w.T. (B,, 0²).
Proof. We start by proving the first part of the theorem's 's statement. Since B C BEuc (0o). we satisfy Condition 1(a). We now need to satisfy Condition 1(b). We particularly focus on Condition 1(b.2). For this, we first show the existence of an element 0' E By For such 0', must be true that 0' E Qk From Definition 2, 0' needs to satisfy three conditions:
cos(O' 02, VoG.)l 2 Ki (cosine similarity condition),
K
00 - 0pa))
Σ
1.3
Vo, Sk
Uk.
(0' -0,r) 2 0
(average condition),
48
i=1
j=l
k=1
K
10 1
(2) Jk Vo,
Vo
(1)
(6' -,,)) S Oi vi [n].
[ad] (output condition).
Let Ve, Ge be the first py components of V VoGo, and let Vo, G. be the last Py components of VeGe.-
We start by assuming that | Vo, Gell2 # 0. Then, let us consider = 00 0 JT where 6¹ E TR will be specified later and 0 ggtt. Then, belonging to the 2% set conveniently reduces to the feasibility of the cosine similarity condition as follows:
|cos(0' - or V&G.)l k k.
(28)
Now, let or be defined as | = o, Vo, Gell2- We immediately notice that e > Moreover,
(
0
02 Oz.
Vo,Gr Gt. |V
1
0, 0z
Vo,
(os (0' Or, VoG.)|
(29)
100 f
Ct
110
f
|Vo
Gill2/
and thus, we immediately have that (28) can be expressed as
coss 0' - OyH, Vo, Gi)l 2 ax.
(30)
We now choose x > sufficiently small so that c < 1.
With all of this in mind, we proceed to show the existence of an element 0' E B of the form 0' = [00 T condition (30) and the following two conditions:
0g,t JJ satisfying
Condition (A): 100;, felll2 = e for some c < 2 C(0,)lla Illlay// 1 (cux) and
Condition (B): the angle v' between (0', - O..)) and -V ((.)) acute, so that cos(v') > 0.
To show the existence of such element 0' B we propose two possible constructions:
Choice (A): the points f,,411, Vo, G, 0y.,, and f are not collinear, then they define hyperplane P that contains the vectors Vo, G2 and Ve, ((O,) (recall that 0f,-+1 0,,; Vo, [(0.) by gradient descent). We choose 6y such that the vector 0-, 87-60; y, lies in P with cos fs,,, Vo, G.) C,R (i.e., it satisfies condition (30) with equality) while simultaneously satisfying Condition (B). the points A f,441, Vo, G. + fyt. and ,,, are
18

---

Optimization for Neural Operators can Benefit from Width
collinear, we choose fy such that it is not collinear with these points, thus defining hyperplane P with these other three points, and such that 8 is also taken so that cos O0; f Vo, Gt) = CcR while simultaneously satisfying Condition (B).
Thus far we have only defined angle (or direction) conditions on the vector 0' 0 j.t and so there could be an infinite number of values for / satisfying such angle conditions without O' belonging to the set B (0v) nor Of satisfying Condition (A). To determine the feasible values for O,, we observe that Or is strictly inside the set Eue (0a) by Assumption 4, and so 80 can be taken arbitrarily close to oz that O' Baps (0o) and P.p1
Condition (A) is satisfied.
We remark that, regardless of the collinearity of the points f,0+1, Vo, G. + O and s, hyperplane P contains the vectors 0'2 - f, Vo, Ge, and ((,), all sharing its origin at ff E P.
Choice (B): We choose 0' as in Choice (A) but with Ve, G2 replaced by - -V,,
We immediately notice that 6' defined by either Choice (A) or Choice (B) satisfies O' E Q% n Bipp (0o). To make O' belong to the set B we need to find a radius P2 such that 0' E B B (0t), or, equivalently, such that 0 E B (0),s) due to our construction of O'. Such oa is found by taking P2 > with e as in Condition (A). Finally, it is straightforward to verify that such 0' E B defined by either Choice (A) or Choice (B) will always exist, by considering the following cases for the angle between Vo, G2 and -o,,(()):
If E [0,=/2] or v E (3m/2, then Choice (A) will be true, since ((o, in the positive half space! of Ve, G and
(i) if v E / π. or V E [T, (2]] then Choice (B) will be true, since ((,, is in the positive half space of Gr
Now, let us assume we are in the case of item (i) above, so that O' is constructed according to Choice (A) (the rest of the proof can be adapted to the case of Choice (B) and to item (ii) by using a symmetrical argument and it is omitted). Let o be the angle between 6/ o,,t and Va, Gr, that cos(v)) C,K according Choice (A). Then, we have that
cos()') = cos(v v1)1 2 (TOS v1) = (in = vi -1-cone cos?(2i = V (cxx)²
Further, by the construction in Condition (B), cos(v/) > 0, which implies cos(v') 2
> O. Now, by the
smoothness property of the empirical loss C we have
C(0') ((0,) (6' VoL(())) -
B
110'
0.12
8
(().) 110; - 0 Vo ((),) cos(e') +
0100 0,,01
B
= L(),)
cos( v2)
2
8
s <(()) - (cxx) +
_
?
2
Bc 2 V[ 122 (cxxi)²
-(())
-c)
2
B
< L()))
where the last inequality follows by the construction of in Condition (A). Thus, we have C(0') < ((,)). which implics that the constructed 6' is as described in Condition 1(b.2). This finishes the proof for the first part of the theorem's 's statement.
So far, have assumed that VV, G l2 * 0 in order prove Condition 1(b.2). If, instead, we assume that |V, Gell2 * 0. then the whole proof for Condition 1(b.2) is done symmetrically: we start by taking 0' = 00,,² 6 TT and then characterize 0', E RRPa.
'We say vector a is in the positive half-space vector if a b 2 0.
19

---

Optimization for Neural Operators can Benefit from Width
We now proceed prove the second part of the proof. For any 0' E B by the second order Taylor expansion of the DON loss with respect to iterate 0,, have
1 =(0)== = + (0'- Vo + +2 20(0 Tf)T0F
ag² (0' 0²)
where θ - ç0' + (1 E) for some & E [0, 1]. To establish 0₂ RSC of the loss with as as in (13), it suffices to focus on the quadratic form of the Hessian and show
(() (0' 0,)
002 (0 0,) 2 ail - 0e112.
(31)
Note that the Hessian, by chain rule, is given by
1 n
H(0)
- Σ Σ
VoG (3, (ocgl"")))) E. ((f
002
n
where l,j (Gg (1 10) (2) G (u (u, )3 Given the block structure of the Hessian as in (23), denoting
50 0' 0, for compactness, the quadratic form on the Hessian is given by
80 H(0)80 ; H H + 2 H (0)00 80 Hgg 00)00
(32)
T3
Tb
Focusing on Tj and using the exact form of (θ) as in (26), we have
T q:
K
n
K
50
Ih. 0, Vo, S 1 (0,)
(0,)
9k. (0,)
(
00 Vo, Vo (0))880
n
k=1
(a)
(2KA1A2 + A)
50,, Vo, (0
1100
Vmf
where (a) follows from e/ = 2 and the different bounds in Lemma since 0 E p.Pl (0o). so that 1g 22Ar + with Z = maxee]])je(e] |G (w( (1) (y)
(3) )/ Similarly, for T3 we get
80g. Vo, (o) (2K\1d2 + (11
T3 2
1:00,12
Mgg
Then,
(50g)) Vo, (1)))) + (80,, Vo, (o((
T + T3
2
qi
1=1
j=l
(Aic(s)
Mmly + A2c(())
- \2 +0)
/myg/ 110011
2
Vo, (00 g: Vo, (o, (o Coo
4
&
Σ
60g g. Vo, Go (8) (80,. Go(u
n
96
cls)
+ \ge(()
12 A2 +2)
|10012.
mg
vmng
where (a) follows from |00||12, 00,122 S ||0||2
20

---

Optimization for Neural Operators can Benefit from Width
Focusing on 722 and using the exact form as in (26), have
1 n
K
T2 - =
V
0
k
vo
80g
48
1=1
j=1
k=1
+ +200]
1
Σ
K
Ik.
Jor (0))
(
Σ
K
Ve,gg
Ve,
(00T) )) 80,
j=1
k=1
(a) ]
n
41
0, 2 (,)
T
80g
n 1=1
qi
==1
k=1
11
1
1
50g. Vo, Go, (u (,) (1))) (888, Vo, (o,
11
96
where (a) follows from ( 2.
For I our goal is to first transfer the dependence of the gradient terms on i to 0., so that we can use properties of the restricted set 0% which is based on O, simplify the analysis. Towards that end, note that
K
11
-80
1
V
e k
(0 (O.,)) (0..))))))))) 80,
2
71
4
1=1
j=1
k=
+ +80
1
9
K
Vo, (0) (0,) - (0.s)) (0, 80g
1=1
1=
12
9
K
+ +80]
Σ
Vo, fk (01.)) (Vo,th)(0)) - Vorgk (..)))
50g
1j
i=1
j=1
k=
(a)
(2K\1)2 + 2)
1
4
Vo, f (0,) Vo, f(0 (0,)) |/ 2
Vo, ( (0,) || ||0 , 1e1|1l1l 11
n
91
j=1
(2 + 2)
2
Vo JK
( ( ) /1, Vo,00 (0,)
Vorgk ( (O..)) 2 ||00||2|||85 ||2||600|1
1=1
j=1
km
(
(2KX1)2 +2)
Σ
n
1
0:
k (0,) 1|_ 110, 01.1112 0
|
||21|50g11
n
q:
2
2
j=1
(2 + 2)
Σ 4i j=1
2
Vo, 1( (0.r) (,,))11 /11
V2 (3)
(0,)
110 0 0 12 1l2 180,112
n
g19)
(
-2 +0)
180, 50 - (2 A2 +2)
||80|12||500118
/mg
N6 -22KA1 +2)
(1 gl
cls) g())
|150|1 3
Vmg
Vmg/
(l olf)
cls)
g(5)
2 (2KA
|:0 1
Vin
Vmf
where (a) follows from the definition of Q set (Definition 12) since 0' E B C Q2; (b) follows from the generalized mean
value theorem with 0,- = e 0 + (1 - O.., for some & E [0, 1] and 0,, = E20, + (1 -2) for some &a E [o. 1]: (c) follows from the results in Lemma since 10] JT E BB (Oo), and the fact that 10, Ozell2 S ||60, l| and
10, 0g.1122 S ||50g|12: l2: and (d) follows from ||00 ||50, | < ||50|| 2-
21

---

Optimization for Neural Operators can Benefit from Width
Replacing 1 back in T3 and then combining the bounds on T1 + T3 and T2, we have
80 T H(00000 z
800g
Vo, (o)
+ (80,, Vo, (,(
11
==1
Aiile)
A2c(()
-
(2 + +2)
|100| 2
mmg
Vm
(6 gl
(60)
( 6l
-
|100
Vmn
2 ΣΣ
2 n 4
Σ
50, (oCg(u())((
)
2
7
(1cls)
Age(()
-2K +8)
|1001|
Vmg
Vm
( g(
(00)
clo)
cl0)
oin)
- 2
11001
Vm
2
4.
(60, (u
((00, ((o)(
- (50, VoGo. (1))
n
(1icl))
)ac(()
- A2K +2)
|1
Vmgg
vm
(69)
g(f)
c6
- 2 E)p2
+2pp
+
|000
mm
ymf
(33)
Then,
1
Σ
80, (uG
+ Σ %
2
Σ
60, (e) - (oGo(u"))))))))
1
9: 1=1
1=1
+
4
1
(50 VoGo (u))))))))
50, VoGg(u
[u
96
(
2 11
1
4 Σ
60, VoGo. (u
Σ
n
2 R
50 [o)
VoGo. (2l0) (e)
12
46
q:
+
1
50, (n
((6))))))))8)))))))) (v 0.))
96
1=1
(34)
2
1
80, (u (z))))
N Σ
50. VoGo. (u )1 (100) (a))
(0) 01))
+ 71
q6
(
2
50, V VoGo. (n (1 2
96
4Eg ((2 + X1 (0))
||00
1 l2
(50) T V3GG (v))
qi
where (a) follows from the generalized mean value theorem and has θ E Eg + (1 -, for some Ea E [0, 1]: and (b)
22

---

Optimization for Neural Operators can Benefit from Width
follows from the fact that 0 0₂- E0' 0') and
|12 <
Vo, Go. (u)))) (1) 0/12 +
Vo, Go. (10)),)
(0 Vo, (e))
Ik
Ve
(0.) (e))
2
K
K
(0: Vo, 1 1
(0)1l2 +
1 (6 Vo) ( 2 (0)12
9k.j
K)20 + KK1g'n)
where the last inequality follows from Lemma 3. Now, we have that
K
V (3CG(((())() (1))) (60) ()))
Σ (50,) [(0 0 Vo. 10 (6)))))) > (50g) [T50))((1%)) (1 Vo, (,)
K
+ 2 _ (50,) T (Vo, ( (,, (Vo (0,)) (80,)
k=l
(0)
(2e(())
+ \iclo)
K
|100|3 +2 Σ (80g) (Vo, (1)
Vo, (V, ( (0))) (60,)
3
K
/m
/mg
k=1
(35)
where (a) follows from Lemma 3 since E BEuc (0)).
P.P1
23

---

Optimization for Neural Operators can Benefit from Width
Now, for 13.
K
1/ - 50}
Vo, Jo [ (o, 80g
2
A=1
K
+ +
Vo, (0)) (0,) No,5
100 (0t,) Ve, (0) (0,) T 50g
K
So ( (Vo,g (e,) Voggk (0..0) (1.0)))(10)) )) 60g
+ +
Σ
K
Vo, /k (0.,) V Vo, Ro. (2 0rh1) 80g
K
Vo, 1o (0)) Vo, (
(O.,,) Vo,0 19 (3)
|60,112|60011
K
+
Vo,sk (0..))
(o. Vo,gi
00 (0.))) 160 |0011221600,1
K
807
Σ
Vo,
(())
(0.,) Vo 1
(2) (100) 80g
c(5) g1o)
cl0)
o(1)
+ E
|150, 1|50,121800 1000100122211 + EBEK
|50 100,|l21100 1l2155,,11
mm
mmg
(b)
K
807
Vo, (1)
(O..,
Vo, (2) (0..))
50g
cl9) 0(5)
c(.l
+ K
|500
mmg
/mg
$ _
Σ
Vo,
Vo,
(o..,))o,
(0.o)
(1) (t..))
50g
k=1
cl0) ((1)
cl5) gها)
+ KKp
|15012.
mg
m
where (a) follows from the generalized mean value theorem, from 110 100--0.0 - =
,r, ll2 = E3El
0e,fllz = f3E1|50y112. and from the results in Lemma 3 since ē E BEuc (0v); and (b) follows from |0,112, 10y12 - |0 l/ and Eag < 1.
Replacing the bound on 13 back to (35), we obtain
TV (o))) s 507
Vo, (
(o.))oo,,k 1 (0.o)
50,
cloo(A) + 20
dl + 2010))
+ K(1 + pz)
|
(36)
mgg
mmy
c(s)(A1 + 20
dl 20(s)
< K(1+ P2)
|100 ?
My
mm
where the last inequality follows from the fact that 0' E Ø2 using the properties of the restricted set Q: Definition 12.
24

---

Optimization for Neural Operators can Benefit from Width
Replacing (36) back la in (34), obtain
2
1
60, VoGo, (w)))
12
n
qi
(co0(A)- 2011)
(clo(A) -
c0 + 2glo1)
453 K (Azel1)
+ X1 ( ||50|12 K(1 + p2)
|5001
mm
(37)
2
80, (u
n
qi
(cl)(() + 20!11)
c( + (
46x K1 p2) *)eel)))
|1501|
m
/m
Replacing this lower bound (37) back to the Hessian expression in (38),
2
50, (u 2
50 (0)00 2
1=1
(cl)) (A) + +2001)
cl1 + )0l0))
- -4ggK2 (1 + P2) P2) + (1
|lgell
mmg
my
Alc(o)
AzolS
-2KA1 +2)
||0011
mm
Vm
(cl)) e11)
clo)
p(1)
(38)
2((2KA1 0)p2
|500
Vm
vm
[ 200, 2 10- K2
1
+
|180|]
( 2 2n²² VoGo. 12110011 ak²
(yms
+
vmg
|800
arl 0013.012.
where (a) follows from Jensen's inequality with Go = = Σ 1 E (o) where (b) follows from the ai fact that 0' Q' and using the properties of the restricted set Qk in Definition 12; and where a, 2x ll ei K2 my - Note that adding all the constants from the second to the fourth line in (38) define the constant C₁ and so e1 depends on d1, the depth L, and the radii p. Pl, and D2 due Lemma As in the statement of Lemma 3, this dependence reduces to the depth and the radii and becomes polynomial whenever do p max ymy ymg J. This
completes the proof.
Theorem 3 (Smoothness for DONs). Under Assumptions and 3, with probability least 1 - 2K my + -,))) the DON
loss C (3) is B-smooth in BEuc (0o) with c2K2,, where 02 > i constant which depends polynomially on the depth L, and the radii p. Pl, and P2 whenever do I p maxl 1 1_
Vmny" M
Proof. By the second order Taylor expansion of L(0') about the point 0 with 0', 0 E
BBEu (fo), we have C(0") PiPl
25

---

Optimization for Neural Operators can Benefit from Width
[())+ - - 0, VoL(O) + (0' 0)
o (()) 00
(f' Q). where A = =00' + (1 - E)o for some & E [0, 1). Then,
(0' 0) 0 (
e' 0) (0' 0)
1
V No
002
71
9:
(0
11
96
0'
0, VoGo(u
1.
j=1
10' 0) T VVGG(u()) -),
where b (c)()())( - (x())))))))
Now, note that
0 - 0, (o
1
71 1=1
2
VoGo(u()) (0) 1100 -012
- 4K 3>2e" + \1 ) - 012
where (a) follows by the Cauchy-Schwand inequality and (b) from Lemma follows
K
VoGGga( 11) S
199 (o,) 60) + (,) s k()00l)) + -)egl))).
k=1
since 0 BRue (Oo).
Now, for Iz.
Kylle' - 0) "' 1 (0' - 0)1
Is <
(0) + ell) mgg
+ dl + )
)) 100
(2KA1d2 2) (glf)gl9 + +K(1 + p2)
K(1+
0
mi
with [mx, (ull)) (. and where (a) follows from modifying the result in equation (36) from Theorem according to our setting.
Putting the upper bounds on 1ı and 1g back, we have
- (0' 0) < [4K2 + 01e10)/2
(0' T) 002
cl + ((9)) (cl)(2,, + o(5) Mmg
12 +6)
(o + +K(1 + p2)
+
mm
x 1 - 013
Note that all the constants on the right-hand side of the inequality above form an expression that depends on K and on 01 the depth T. and the radii P. Pl, and P2 due Lemma 3. As in the statement of Lemma 3, the dependence of such expression
26

---

Optimization for Neural Operators can Benefit from Width
reduces the depth and the radii and becomes polynomial whenever og 1 p max proof.
This completes the
ymn
ymg
Proposition 1 (RSC to smoothness ratio). Under the same conditions as in Theorems 2 and 3, we have that az/ < 1
with probability at least 2LK
+ mgg
Proof. From the proof' of both Theorems and 3, at < Vo, - 2x (2201)))0l + +1gol)) & 4K 2K2000000) + X1 g < B, and so de < 1.
E. Analysis for Fourier Neural Operators
We recall the FNO model
(0) (au)(a:)
1
W (0))
m
a 1
1
W(() (
(1-1)
R ( le (2,...,,++))
(39)
Vm
vm
f(0:x) (L+2)
v (T(++11)
vm
where W R() mm for (2...., L + 1]. W() E Rmxd
E.1. Bounds on the Hessian, Gradients and the Predictor
Lemma (Bounds on the Predictor). Under Assumptions 5 and and for Buue (0o) we have with probability at least 1 2(L+2) that for any input function and evaluation point in Section m
| < Vm"
(40)
Volll2 S ! 1 A
(41)
(42)
where e, O, are suitable constants that depend on 01,w 01,r, the depth L, and the radii Pwo Pv, and Pt. The dependence of the constants reduces depth and the radii and becomes polynomial whenever 01,00 + +irr S tu
V
In this section we will prove all the bounds in Lemma
Lemma (Initialization of the Parameters). Under Assumption 6,with probability least we have |w!'l2 112 onw and || 22 S o..
m.
(43)
Proof. The proof follows directly from Lemma A.l in (Banerjee et al., 2023a).
Proposition (Layer-wise matrices). Under Assumption for E BEuc pupp..p (6o), with probability at least 1
2 we have
(1...u+ + Pw: ) Vm, lE and | < 01, + mm Vm. lE E f2..... L L+11 (2....,L+11.. vm
|000 <
(44)
Proof. By the triangle inequality and Lemma 5,
( ||2 < [| 112 + ||(()) w:(O) 112 S Fl.w / Pws
[ ||2 S | || + ( R ||| - 01rr Pr
27

---

Optimization for Neural Operators can Benefit from Width
We now bound the norm of the output (1 the layer I [[+ 1].
Lemma 6 (Norm of the 1-th layer output), For I E [L 1). under Assumptions 5 and 6 for 0 E B BBuc Pusstipi (0o). with probability at least 2 m
we have
i-1
[0(0)] (0) [o(0) Σ )t1 Vm,
|a=1|12 s vm
P vm
01 +
(45)
vm
where,
7 = 01 P
01 Fl, + 1ir
p == Pw + Pr.
and
v m
Proof. We prove the result using induction (e.g., see Lemma A.2 in (Banerjee al., 2023a)). First, note that for the first hidden layer, using the fact that is L-Lipschitz,
Φ
(ta
1 V
W
a (0)
)))l (0)()) s ( -(0)
1 vd
w
(46)
2
where 0 denotes the zero vector of appropriate size. This in turn gives, using ||a
1|2 vd.
|o0|
(0)
015
- 1
w 00)
+ +))2
Vd
Vd
1
|r1)
(0)
+ =0(0))mmm
2
Pw
01,
+
vm + (0(0)/m
mm
01, + 01.r
Pw + Pr
Vm+ @00)lmm
m
Now, consider also the output layer 2, namely,
||a (2) 12
0 (
w a
F F RO (
vm
which gives,
1 (33)
1 F. ((() Fa!!))
- -11o(0)]12
vm
F 2(2) Fall) ((
-100
1 | (1
1 F Fa
S
+ vm
m
mm
vm
2
and, in turn.
1
aa
1
W
2(2) (
F*
+ +|o(0))l2
V
mm
(1)
Fm
Fo (1)
+ +lx(0)/m
2
m
|| (10
1
|
1221 122 + V x0)
Vm
01, + Pw
+ +
+
Pr
||a00112 + vml6(0)
Vm
vm
< v 01
P
2
+
σ1 + p ) vm|o(0)).
vm
28

---

Optimization for Neural Operators can Benefit from Width
where (a) follows from the fact that the operator F is unitary matrix. Now, for the inductive step, consider that the output layer - satisfies
1-1
P 1-1 vm
| S </m
01 +
vm
P
m
01
(0()))
Finally, at layer I, we have
a
(1)
W + F Rloal
||00
||a/1-1)12 112 + vmlo(0))
(47)
2
m
1,w + +1,, + Pw tpr)
vm
1100-1)122
vm))(())
(48)
S vm
01 +
P V
+ vm Σ
01 \
1o(0))
(49)
P mm
Introducing 7 01
write
|a (1 12 S vm
f0(++10000
Σ 7--11
(50)
This completes the proof.
From now on, we will use the notation p. di, and y as defined in Lemma 6.
Lemma 7. For E (2....,L + 1}. under Assumptions and for E B
Pu (0o). with probability at least
2 we
m'
have
ia!!!) (a(l-1)) 5.
(51)
Proof.
We first note that
!!!!!)
1
ø (a (11
( [F* +R=+ 1is)
1011111
vm
ij
Now, from the definition ||A]|2 = siii|i||2=1
|Aull2 we have,
Oa!(!)
0d' [d[(m(0) + F p() R(OFF) F) 00 =1(,)
(w(O)
(111-1)
sup |ml|a==
mm
(Iw(We/12 + | R()) Fullz)
sup ||0]
sup
[| || +|R(FFlll))
(52)
||=|==1 m
[| + |R || (2||||112)
sup
||0|/==
m
1
+|
Vm
where (a) follows from the fact that ò is 1- L-Lipchitz and by using the triangle inequality. and (b) and (c) follow from the
fact that Ft and F are isometries with respect the Lg-norm, i.e. F ||2 || and |F #1l2 ||mi||a for R This
29

---

Optimization for Neural Operators can Benefit from Width
finally gives
da!! 0a"1-1) 2
(1100122
+1) < 11 + Pmu
vm ) +
(o.r +
Pr
mm
+
vm
7
where we used Proposition 2. This completes the proof.
We make use of the Einstein summation convention, i.e. repeated indices imply summation, unless explicitly stated. We also use the notation vec(-) to denote the vectorization of the matrix argument according to some fixed manner (e.g., row-wise vectorization).
Lemma 8. Under Assumptions and 6 and for E BLue
P..p..p (0o), with probability at least 21
m
(0
Oa!" !1)
L-1 S (
1000)
)
Owili
(r)
i=1
where, w(") vee(W()) for + 1), and r(0) vec(R!)) forle Ef,
f2..... Be....,Le L
1}.
Proof. We can index the vectors w and (() according to their matrix form W jj' and R respectively, with the indices j E [m], and j' E [d] when I 1 or j' [m] when I E (2...., L+ + 1}. Therefore,
to (2)
1111) 5i -
i=j
Ow(()
1.55"
mm
otherwise
Now, for 1 E f2,..., L + 1}, we can write the Lz-norm of the matrices as follows
(1) 2
Owill
= sup
0'
(1)
88 Viy
||||== =1
1=1
3.j
VVa (1-1) 12
sup [|]|=== m
a1 1 |
sup
[|]]===
|| (1-1)
sup
IIV [F=1
1) b
1-1
]
2
Σ Σー 2
1
m
---1+ + (o(0)
(0(
where (a) follows from the fact that ||V|| Σ || F and (b) from Lemma 6. The case follows similar fashion:
(1
2
< a 12
(w
30

---

Optimization for Neural Operators can Benefit from Width
Similarly, for 1 E (2,.... L L+ 13.
1a (1) ?
m
Fij Pippa-a--y
= sup
0r(1)
1=1
(F"VF)a (1-2))
1 sup
[| m
- sup
1FVFF
[|
sup
[|V]===
[ (11)
sup
V|I==1
1 (a
771
(x(0) -
2
m
[o(0)
Σ
Σ
m
where (a) follows again by ||V||2 < ||||| and the fact that F* and are unitary matrices, and (b) from Lemma 6. This completes the proof.
Hessians. We now focus on bounding the Hessian of the predictor f in equation (39). Note that the FNO model can be considered as having L layers, with Layer being a feedforward single layer encoder top of the encoder P, the L layers from Layer 2 Layer L + 1 being FNO hidden layers, and Layer L + being the output of the lincar decoder. Likewise, we decompose the Hessian matrix H of the FNO in three different blocks corresponding to the aforementioned encoder, FNO hidden layers, and decoder, respectively.
Firstly, the Hessian blocks associated the hidden FNO layers are:
the L x L sub-blocks corresponding to H (11.22))
0² for 11,12 E (2....,. L 1}. Bowl)
the L L sub-blocks corresponding to H.!1.12)
8
for Nor4,44 11 E {2...., 1}, and
the cross blocks have terms of the form (ldd))
8 Sw d2 for 4 E (2,.................. L + 11.
Secondly, the Hessian blocks corresponding the single layer encoder, i.e., with respect weight W(L);
diagonal block (...)
82
off diagonall blocks H(
8 Ow "a and (l.1) for I (2...., 1), and
off-diagonal blocks (1,,a))
(1) ans (l...)) for E (2...., L 1}.
Finally, the Hessian blocks corresponding the decoder, i.e., with respect to weight v:
diagonal block H, := f which is the zero matrix 0mxm.
off-diagonal block H!(1)) = ow Wav
81 and (b) for l (1,.... I 1}, and
off-diagonal block H!!a) = aev and (())) (1)) for la (2...., , I + 1}.
0²1
31

---

Optimization for Neural Operators can Benefit from Width
First, we note that due to the symmetry of the Hessian matrix of the FNO model H:
L+1
L+1
L+1 L+1
L+1
L+1
H
+
Σ
|
12 +2
Σ
Σ
H we
(
112
+
l2
+ 2
Σ
H
12
(53)
12.
11222 2
1 1 2
1=
1==
We define
0f i
2x) == max
LE|L+1]
{ (t
w.r) (f) = max
[E|++1]
(r(!)
12
(12)
11 (1)
0a (12)
2,2,1 (2) ;= max 1sh
(w
(r((()))(())
Ow(h)
al
1) (12)
2-125--1
2,2,1
2,2,1
35la5
da (4)
(2))
da!!
0x
8 (1a)
Or(1.)
da (12
(12)
Ow(li)
Or(())
(00(1s-1))
2,2,,1
2,2,1
an)
(1)
(aa(
(2)
(w))
max s
(ow(l1))22
Ow(h)
0a
1) (w((2))
2C12ST-1 3515L-11
2,2,1
2,2,1
(1)
(k
(2) (b))
(w((
(2)
(0cels-11)3
2
2
2,2,1
{
0 (1) (2r(h)))2
aa
(12)
2 02),1())= = max
25h,--1
@r(h1)
-a (12)
3CLSL-1
2,2,1
2,2,1
4S1SS--1
aa(())
da (11)
da (2)
pa (1s)
(1))
(r((s)
(1a
2,2
(54)
where, for an order-3 tensor T E Rddxxdd xd2xdsx we define the operator
||-1|22,1 as follows,
da
da
|T ==
sup ||l||=|||l|| 1 k=
T
Tijkabb
a EER BRR..RR.
(55)
i=1
j=1
Note that it seems from (54) that we need the depth L. of the FNO to be 1 3. However, the bounds presented in Lemma 4 also hold for FNOS with depth L < 3: indeed, the upper bounds we derive in this section for an FNO with depth will trivially hold for FNOS with depths
I,...,I.
Lemma 9. Under Assumptions 5 and 6 for f E BEue pasppeps (0o), the following inequalities hold with probability at least 2(L+2) forr 1 [L+ [ 1).
(a(
< <86(1++798(1110000333
(56)
(Ow(l))z2
2,2,1
and 12 E f2..... L+ 1}.
(b)
< Boll (86
(57)
2,2,1
(a((2a)
< 223072
(58)
(0a((1-1))3
2,2,1
32

---

Optimization for Neural Operators can Benefit from Width
+1,
(59)
da (12 (0001)00(l))))
2,2,1
(all)a)
S (1
and
(60)
da (/2 (12
2,2,1
S 1 (1+0++)0(
(61)
(ar(la))2
22,2,1
Proof. We first begin by proving (57). Note that from (39) have
8
1 - (allhi)
(12-1)
Dey Dig Fw a (1--1)
aj
dw
kk
where we make use of the Einstein notation. Now,
Pa, (1))
Ow @w (12) 0r(12)
2,2,1
112 "dg Pik F (12 11-1)11 Veew
(12)
sup
(a:
|V
m
Φ" (a:
(12))
(Vi.aaaf1-1) (ra Venue ow
sup
[|
i=1
m
(62)
B
(1=2))))
sup
[|]l|=
1=1
36
[|Via(lo-111 + ,F 12 (1)1) (2)
sup
[V |||
[V/l
2m
(6) Bo
(1a/tt-112 + +|a000--111
S 30 ( 10(0)
Σ
7 1-1
2m
1=1
where (a) follows from the quadratic expression; where (b) follows from || |Ville, 1/2/l22 s [|2]|e, 12"2aa4>>>"11 [|V2Fa((>>))1> s || being unitary operator; and where the last inequality follows from (50). Finally, we can upper bound the last quantity above as in (57) and complete the proof.
< Vi [| |1 <
||
|| 12 due F
For proving (58), again note from (39) that
(22)
= (all))) (W!!la) + FT Pe
R2lk)) Fas) (wilee) + F (2(2)) Fuk
(1)2
m
(63)
Φ"
W (12) W (1a)
w(lla) + w- (l))
F Relal
Fot
F R(la))
Faj (aa
Fip Rila)
F F R(la) Fuk
Ti
T2
Tss
T4
Then, we can write
82 pg. (1a)
m ea(
sup
01, U2 U2
Oe 11)
2,2,1
||e]la=11
laa
0alll -1)-
33

---

Optimization for Neural Operators can Benefit from Width
Let us consider the notation Too
1,w
Pau vm
and
7r
-,r + vm
Pr.
Now, handle each of the terms separately:
Σ
m
d" Tu
|
m
(wr(eless) : (Wilte) o2.
sup ||
U1, U2a
sup
13
l|a
m
m || la 12l|vall2 m1
1=1
sup
(1 12 21 112 + || |1
(64)
2m ||e (edll111] 11,==111/2/=-0 2=1
= Bo 1100 +
Pwu
= Tw
/m
sup |/ ||a 11 o na lla
172
8"
T U1 02
1Ø"1
sup
m
Flijer))
((F
R((i)
i=1
m
m
||mil22 12 lla
86
(IF F 01 11
sup
2m
| |=.1:==1||0a/1===
(65)
B
| (2(lla) (211)F112
_
S m
| ||2 S B
m
(W 01,) (F (((l)) P)in
sup a 1==
T2 2. 1, U2
sup
m
m | 11:=1,||
1=1
||
|1 11113 + || R (1aa) (22 F1
(66)
sup
2m
I| |1| la=1
BB [[l + +112(>0)13)
Be (-2 + 1
-
S 2
2m
Similarly, for the term corresponding T3 obtain
m
d"
83 5
sup | |||==||||
Σ
Tassa ,
(: + +
(67)
Uv
m
2
Putting together (64), (65), (66) and (67), we get
8a
< 230 + S 280(12 + + + 2%wr)) = 2,307
(68)
dall 2a 1)2
222,1
This completes the proof for (58).
We now look at the proof for (60). First note that
8
1
(Wrlla)
+ Fr (((ba) Fat)
FijEEy 11--1) +
1
o'(a!
(2)
)
da
(1- 1)
Dr
(12)
m
mm
31'
6"
m
F ij (w(lb))
F..g Fy 12-11)
+
m
Rile)
ip pg
Fyth Fig Fy. a
12-1)
+
vm
pl(a!
(12)
)
Fyk
Ti
Ts
34

---

Optimization for Neural Operators can Benefit from Width
Again, we analyze each of the terms separately
Ø" Fi
[i (1))
=
sup
||l1
1,11
Vsll =1
1=1
m
36
(11 +
|| V2F (1=1111111
(69)
2m
12-1 1 ())
2/ +
Σ x
2
1=1
172
Φ"
T2
2,2,1 122.1
sup
Frp
plla)
Fg U1 F.j V2 Fs (1-11))
|/ I=1.1//lll e 1= i=1
m
[ p((2))
Fu + (1-11)
sup
2m
As
(70)
(IF* F (1
11 12- 112
sup
| 1,11/1]]
2m
(a) 3o
(0(0)) 12-1 2/ (
[|R
+ (2-11
11
S
2 +
2m
2
where (a) follows, again, by exploiting the isometry of F and F with respect the L2 norm, and using VVelle VValle. Finally,
To T
sup
F 12 255 Fg
2,2,1
p=1
m
sup
(F* (F'V
m7 ||o |la=
i=1
(71)
sup 19 [
1 FV2 l12
sup |1 1 1.[l
|2/|22| 1n 1|2 =1.
Combining (69), (70) and (71), we get
Paa
8 (>2 + +B
(/2- 1 +d(0))
1-1
Σ
i-1
+1
da (1s 10) (12)
2,2,1
(72)
1-1
Bo
72 [o(0)
Σ
+1,
and finally we can upper bound the last quantity above as in (60) and complete the proof.
For proving (59) consider the following
(12)
(a(a!(a) (12)
(((a!12)
(12)
Willa)
a
(1--1)
1)
big
F.e pli2)
(12 F
01)(2-1))00(0
m
m
ip pq
mm
T
T2
Ta
35

---

Optimization for Neural Operators can Benefit from Width
Then analyzing each term separately, we get
12.21
" w
Uix V2.,, V.
a (2-11)
2,2211
=
sup
|| [ol1==l p=
==1
As
(11w (2 12112111 112 + | 62-
|
sup
2m
(73)
30
(11
||ax01211113
30 5
2e (11-1 [c(())
1-1
Σ x )
2m
m
6"
T2
12 2,2,1
sup
F p(aa)
FaLuIr V2,,, ag (12-1)
Ie |||
V/
i=1
m
(IF R(b2)
F 113 + 1122 (1-0)=11)
sup
Vvll2=1.1[V5
p=1
2m
B$
[I R(22)
F1 01 + 11221111a (2
sup
(74)
l0 |=lla=1,1|V/1 1=1 2m
Bi
(1 (1)))) 111 + ||a 11
2m
pst + (xt (6(0))
(
2
==1
and, finally,
Φ 12 v
Ta
sup
Vall ||
V m
im
m
vm |/ ( 12 1/2
sup
||o.|l=11.|VV/|=== =1 1=1
(75)
< sup ||a||===
Σ 1=1
1/ 11
= 1.
Hence, we have
82 (12)
B
(1 + + + Bo (/ 7/11+ [1-1+10(0) [o(0)]
Σ
7-1
<
+1
da 012 (w0(()) (12)
2
2,2,1
(76)
122 + (1 (g(0))
4-1
$ B$
Σ x-1
+1,
and finally we can upper bound the last quantity above as in (59). We now focus on proving (56). Note that from (39) we have
(1)
(Owll))yz
_ 8" (a!li)
-aj
(1-11) (1-11) (1111)8,50aa
(1
ax.
81z0an.
36

---

Optimization for Neural Operators can Benefit from Width
Now,
Pa
(0w(h)))3
2,2,1
m
sup
(ā; (11)
V..,. aj'
V2
(1:-1)
122w ak (1-
||
m
36
(Vialli-1) (2200 )1
sup
(77)
[12 |=
m
BΦ
(|via
122a011 (12)
sup
[14 |= Vo 2m
2
Σ 5tt
a (1 1)
S S$
(
[0(0)
Finally, we can upper bound the last quantity above as in (56) and complete the proof.
Now, the last result to prove (61). Note that from (39) we have
aa
1 d" (alli))
Fi (12 1) FrF
(ar()))
Akk
m
where we make of the Einstein notation. Now,
(a (b)
(2
2,2,1
M
sup
- aa
(1)
Pyraz
(1- (1-11) Fit V2r Fw (
|V
m
3,
((FVFF)) (FFVFF))
sup
(78)
|Val===
i=1
(VV (22 13-00 113
sup
[ |V/l==
2m
1-1 2
8g
(2-11)
2 S 30
(d(0)
Σ
1=1
Finally, we can upper bound the last quantity above as in (61) and complete the proof.
Now we upper bound the terms in equation (54). Thus, we obtain that, with probability at least 1 2(L+2)
0ʃ
(4)
1
Qx
max [E[L+1]
max [E[L+1]
7 1-11-((1 p1)
(1 + (1+1*)(1 p)),
(79)
(x((
mmn
vm
(8)
H-1 (41-1 + +1((0)
Q )(w (
max
Σ
< (1 (1+ (100)
(80)
[E[L+1]
1=1
and
(0) & 28 + +1 + 21 + (1+ [[)())))² +1,
(m. (f), 2m) (1), 22.
mm)
(81)
where (a) follows from direct adaptation of the results from Section A.6 in (Banerjee et al., 2023a), (b) follows from
Lemma 8, and (c) follows from Lemma 9.
We now proceed to analyze the Hessian. We also recall that (2..
Bo (1)
< 7 from Lemma 7.
dall Ij
37

---

Optimization for Neural Operators can Benefit from Width
We introduce some notation. Given an order-3 tensor T E Rd xdxxx, have that its first dimension has d1 entries, the second has d2 entries, and the third has d3 entries. Consider the matrices X E Rkixd, xd1, Y E R and Z Ra We use the notation (X))))(Z) E g& x to denote X multiplying A long its first dimension, Y along its second dimension, and Z along its third dimension. We use the notation (Z) E I& x x to denote X multiplying A long its first dimension and Z along its third dimension.
Ofr-Diagonal Blocks. For the off-diagonal blocks, we focus on bounding [11W):/")) ||2 for (Case I.A) 4 S l2, (Case I.B)
l2 S 4. Further, we bound (Case 2.A) [| la and (Case 2.B) || ( ||2.
Case 1.:: 2 s 4 12 L 1. By building on the form of the gradient, we have
(all))
Of
(aa
12
dall")
pall))
0f dallla)
1 (1416))
@w(h)r((h) dall)
[1=bb] + stal
@wlli]
Dall-13
1=+++1
I+1
dclll--T) )
dall)
(Dalla)
dall"
)
pa
0f
Σ
+ [|1
(w)
Or((2)
(0ax1111)2 20e
1=1+1
1'+++1
M=+++1
Then,
(1)
0f
H 12)
1 -
[//=
@w(1))r((1
(c((l)
2,2,1
(1
da!l))
aa(l) (b2
0f [c.(
+ [|1
Owlli
Oar
dall
1=
2,2,1
I++1
(h)
1-1
Oa!!")
1))
(1
(xx()x))
dall))
+ [1 <la)
Σ
@w(())
II
(0)
@r((s)
II
Ocel''-1)
112+1
-=+++1
2 11
0 a
0f
(dall-1))
ix(()
2,2,1
(a)
0f
Ow(li)r(()
(a(())
2,2,1
(h)
0 (12)
05
111<1177 //
(w))
da (12 Or
(12)
2,2,1
100
da
02 02(11) (1)
8ʃ
+ 1/ <lal
21
1-
Ow((1)
0r
(da!!-1))
ia
1m/++1
2,2,1
Then, based on the definitions in (54), we have
|1 22 S (L + 1)(1
2,21 (70 (f)
(a) 1U++)(1+pi) (1+ (1 (1 + 1).
vm
where for (a) we used equations (79) and (81).
Case 1.B: 2 < la < 4 S
L + 1. By building on the form of the gradient, we have
H(1d)) Pall)) of (r(h) da(l))
(
(wal
dall)
) Palli) Of
(ff
II 1=1++1
@rila)
doxll-1))
dalli-1)ow() dall.))
L+1
(2)
1-1
Bal'--)) )
dall')
(olll)
1-1 dall!)
(a((-11) ) Pal!)
( Øf
+ [(1
Σ
(r((s)
II p=b++1
Ow(h))
II U=h++
(dall-1)32 dda!!)
1=/++
38

---

Optimization for Neural Operators can Benefit from Width
Then,
(1)
0f 0arl())
1 11.
1|2 <
=/a=
(w((z))r((a))
1,221
(t)
4--1
dall")
aall) (1)
+ [bcb]]]]]
Or(1a)
da((--1)
8all
(oall--1)80 (ww(())
(c.(1.)
11=1++1
2
2,2,1
L+1
(a!ls))
1-1
oa!"")
1)
(1)
(a
da
Σ
II
II
+ [(2ct])
@r(bb)
Oa("--)
(w((1)
da(l'-1)
141+1
2 11=22
0a
0f
(0))
(1(()
2,2,1
(a.
0f
@r(())
)2((2)
[11=2]]
1,2,1
(1)
a2
0f dall))
(12 12-1111
(r(())
0a - 11 @ww(
2,2,1
20
(1)
2a
aʃ
21 21-4 12- 2-2
[bch]]
Örllz]
(w))
(111111)
0c(()
1h++1
2,2,1
Then, the upper bound similar to the case Case 1.A,
||
(2) 1i
12 I (I. + 1)(1 + +2 ( (12
(a) (I+1)(1+ p1)
(1 ( (1 +3 7) +11
(l)()))) 1).
Vm
Case 2.A: < < L + 1. For Hessian terms involving (w, v),, since Øf 1 a (1+1),
we have
vm
da(L+1)
(a
L+1
H()) w.
=
vm Ow(li)
mm
Ow!l
111
l'=1++1
Then,
(1) (w)
L+1
(')
(e) 1 "0 Z(o0))
(+) m0
- /m
(w)r) Qun
(f)
Vm vm
10(
vm
1
where (a) follows from equation (80).
Case 2.B: 2 < le < L + 1. For Hessian terms involving v), since
af (b+1),
we have
80
vm
(1+1))
(t
L++
da!l")
Vm
Or(())
Vm
0r((s)
Iall"-1)
1=b++1
Then,
da U1a)
(2)
L+1
d'!l")
(0)
(1)) 12 <
II
7 4(x0)),
(1
Vm
Tr!lz)
Vm
v=1++1
2
where (a) follows from equation (80).
39

---

Optimization for Neural Operators can Benefit from Width
Diagonal Blocks. For the diagonal blocks, we focus only on bounding (Case 3.A) || |[FF((1d))) for 4 S 1g, since the case l/ I L is just symmetrical and will have the same bounds.
|l2 and (Case 3.B) || |1F,,
122
Case 3.A: S 1 1/ S L + 1. By building on the form of the gradient, we have
(a 0f
da!l!)
1-1
dall)
pa
0f @a(l))
H(
(dwll1))3 dall!)
1 11111] +
Bistal
Ow((i)
II
dall--11 (10((1))
1-1++1
L+1
Dall)
(dalla)
salll))
Palll))
0f Oa.11
+ <1c
Owll
dcx(I-13)
awlla)
(dax!1-1)3
1=
p=11+
1'=1++1
Then,
aa
0f
|
(11(1
112 1
(8w(1)))2
dce (1)
2,2,1
100
(1)
12-1
dall")
02 aa (1a)
0ʃ (a(
+ 1 <1a]
II
Ow((1)
Oo (11)
Dallz (1112-1)@ww11) (22
1 1'=l++1
2,2,1
8
L+1
@a
dael')
(1) )all
1-1
+ 1 11 <bg
II
II
@w((+)
)all"'1)
Ow(lk)
0a.(''13
1
1=
pa (1)
0f
x
11)2
2,2,1
(a)
0f
-<
(8w(l1))²
(1
[11=1]]
2,2,1
(1)
(a (22)
0f
+ 1//26177//-111
11111
(w)
00ll2-110w0l> (1))
(c)
2,2,1
L+1
(1((1)
(1s)
a2 (1)
0f
+ 11, <b2]
Σ
21 1/ 12
Ow((1)
(2) (2)
(all-1133
(1
da
1m
2,2,1
Then, based on the definitions in (54), we have
(1 la S (L 1)(11)((1 24 22) (w) (10,00)))
(a) (L+ 1)(1 p1)
(1 + 21 2 + +)) (1 ((+(0))) 1),
vm
where for (a) we used equations (79) and (81).
Case 3.B: 2 < 4, </2 S L + 1. By building on the form of the gradient, we have
pall)) 0s H((ed2) d2)
(alli)
4-1
dall
oall) (b)
as dallo
II
+ [nci
(ar(11)²2 (1)
Tr(lit
Da
1=
L+1
1-1
dall))
(ollls)
da((')
pa(l))
0s
1 sis
Σ
Orll1)
darl'-1)
Or((z)
01ll--1)
(1))
(ta(()
1=1++1
1=11++
M=
Then, we can obtain prove the following upper bound in similar way to Case 3.A based on the definitions in (54),
| (1.122 (r) 12 5 (L+ 1)(1 + +2)))22220110022
(a) (T. 1)(1 p1) (1 (1++2))008
(1 (())))) + 1),
vm
40

---

Optimization for Neural Operators can Benefit from Width
where for (a) we used equations (79) and (81).
Putting all the shown results back in (53), prove equation (40). We also note that all the constants in the Hessian bound depend on Lans Ir,, the depth L, and the radii Purs Pro Pt. and P2- This dependence of this bound reduces to the depth and the radii and becomes polynomial whenever y S 1, which is equivalent to Fi, + o1r S 1
Putte
Now, we focus on proving the rest of equations in Lemma 4, namely, equations (41) and (42).
Gradient and predictor bounds. We observe that for / E [4]. Swis
af
Bo (t) Swil
(II
Bo C 11)
Dallyiy and so
ar
dex!"")
os
do (1)
of (c(1+11
-
1+1
(w
Ow
2
Oo
1+1
(1 + +P1)
Ow(() (1)
mm
2
1
$ (1 )2) 1|o00)))
(1 + p1),
mm
where the last inequality follows from Lemma 8.
We also have that
0f
da((+l) Of
(w(++1)
=
Ow(L+1) da(l+1)
2
dall+1)
0f 2 (c((++)))
@w(L+1)
< (1 (1++00000)))
(1 + pi).
Vm
Similarly, we can obtain for l2 E (2...., L}.
05
(1 11+00010" 1
(1 pı),
0r(())
vm
and
0f
(100))
(1 + p1)
0r((+1)
12
mm
Using all these derivations,
L+1
0f
L+1
2
+
Ow((!)
(r(()
2
<
( + 1)(1 +)(+ 4000))
m
which finishes the proof for equation (41).
Now,
Ifl
vm v T ((+1)
$ m || l12
< (1 + P1) + 71 (((0))
which finishes the proof for equation (42). Again, we notice that all these bounds have polynomial dependence on the depth L,, and the radii Pws Pr. Pı. and P2 whenever y 1, i.e., whenever 1,, 1rr S 1 Pet
ym
Thus, we finish the proof for Lemma 4.

---

Optimization for Neural Operators can Benefit from Width
E.2. RSC and Smoothness Results
Using the results from the previous section, we immediately obtain the RSC and smoothness results.
Theorem 4 (RSC for FNOs). Consider Assumptions 5, 6, and 7, and Qi as in Definition 3 with || + 0. Then, the set B == 0% n B pu (0a) n B Ap2 (0t) is non-empty set that satisfies Condition I(a) and (b) for suitable p2. Moreover,
with probability at least 2(L+2), at step t of GD, the FNO loss L (6) satisfies equation (8) with
0t 2x IooG.
(14)
m
where VoG.= = Ei Ei=1 depth L, and the radii Pws Pr. Pt, and p2 whenever 11,00 + 01 S 1
and for some constant
> 0 which depends polynomially on the Thus, the loss L(0) satisfies RSC w.r.t
Dutlr, vm
(Ba 0t), i.e., Condition I(c), whenever o (((m))))
Proof. We start by proving the first part of the theorem's 's statement. We immediately see that, since B B Pr P. (0o).
satisfy Condition (a)) We need satisfy Condition 1(b). We particularly focus Condition 1b..2.. For this, proceed to show the existence of an element O' E B% that is an element of the set 2% as in Definition 3, i.e., satisfies
coos(0' Os, VoG.) R;
(82)
and that also satisfies the following two conditions:
Condition (A): O 0.l12 = for some E <
2I and
3
Condition (B): the angle s' between (0' 0.) and - Vo.(O,) acute, so that cos(v') > O.
To show the existence of such element O' E B we propose two possible constructions:
Choice (A): If the points Oc+1, VoG, O,, and are not collinear, then they define a hyperplane P that contains the vectors VoG, and V (oC(O,) (recall that 0++1 0. Vo by gradient descent). We choose O' such (i.e., it satisfies condition (82) with equality)
that the vector 0' O, lies in P with cos(0' O,, VoG,) while simultaneously satisfying Condition (B). If the points l1+1. VeG, + 0,, and 6₂ are collinear, we choose 0' such that it is not collinear with these points, thus defining a hyperplane P with these other three points, and such that 0' is also taken so that cos O'' - 0,, VoG.) = while simultaneously satisfying Condition (B).
Thus far we have only defined angle (or direction) conditions on the vector 0' - O,, and so there could be an infinite number of values for O' satisfying such angle conditions without 0' belonging to the set B Buu
-P- (0n)
nor 0' satisfying Condition (A). To determine the feasible values for 0', we observe that 0, is strictly inside the set Euue (0v) by Assumption 7, and so O' can be taken arbitrarily close 0 so that 0' BEuc and Condition (A) is satisfied.
Pu PP (On)
We remark that, regardless of the collinearity of the points 0 VoG; 0r, and O,, hyperplane P contains the vectors 0' 0,, VoGr, and (O.( all sharing its origin 0, E P.
Choice (B): We choose 0' as in Choice (A) but with VeG; replaced by - G.G.
We immediately notice that 0' defined by either Choice (A) Choice (B) satisfies 0' E Q n BEuc Be (0o). To make O' belong to the set Bi, we need to find a radius p2 such that 0' E BBuc Pr.pp. (0o), which is done by taking P2 > with e as in
Condition (A). Finally, straightforward to verify that such 0' E B defined by either Choice (A) or Choice (B) will always exist, by considering the following cases for the angle between VoG. and - VoL(())
(i) If v E x0) or v E 33//2,2m], then Choice (A) will be true, since - -Vo, C(o,)( is in the positive half space? of V VoG.; and
We say is in positive half-space if (a, b) 2 0.
42

---

Optimization for Neural Operators can Benefit from Width
(ii if E 1m/2, m| or [m,3,/2], then Choice (B) will be true, since -Vo, is in the positive half space of VoG.
Now, let us assume we are in the case of item (i) above, so that O' is constructed according to Choice (A) (the rest of the proof can be adapted to the case of Choice (B) and item (ii) by using a symmetrical argument and it is omitted). Let Di be the angle between 0' O2 and VaGr, so that cos(v1) according to Choice (A). Then, we have that
cos((/)] = cos(vv v1)] 2 cOS((T/2 11)] sin(11)] = v (1))(11) v -I-K².
Further, by the construction in Condition (B), cos((/) > 0, which implies cos(e') 2 vI x² > O. Now, by the smoothness property of the empirical loss & we have
3 (((') S ((O.) (0' O,, VoC(()))
=10'
-0--0113 0
B
((0.) |001 0:112| 0.1la cos(1))
1100
-
2
3
C(0.) V cos(1/)
2
B
< C(O.) |/ ((0,)(00)1 x²2
B6
V - x2
[(0.)
8
< L(..)
where the last inequality follows by the construction of e in Condition (A). Thus, we have C(0') L(O,). which implies that the constructed 0' is described in Condition 1bb.2). This finishes the proof for the first part of the theorem's 's statement.
The second part of the proof, i.e., the RSC condition over the non-empty set B follows from direct adaptation of Theorem 5.1 in (Banerjee et al., 2023a) using Lemma 4. Since we are using Lemma 4, the condition for polynomial dependence on the bounds carries on.
Theorem 5 (Smoothness for FNOs). Under Assumptions 5 and 6, with probability least 1 21L+2), the FNO loss L (6) P..Pe (0,) with / being positive constant which depends polynomially the depth L, and the radii pw Ps
is 8-smooth in Buc
Tn
Pr, and P1 whenever 1, + T1r S
vm
Proof. The proof follows from direct adaptation of the proof of Theorem 5.2 in (Banerjee al., 2023a) using Lemma 4, where it can be shown that B = 2²² + Tm for some positive constant é which inherits the dependence on the constants
Ii Ir,r, the depth L and the radii pws Pr. and pl from Lemma 4.
Proposition 3 (RSC to smoothness ratio). Under the same conditions as Theorems 4 and 5, we have that ot B < 1
with probability at least 1 2((+2)
Proof. From the direct adaptation of the proof of Theorem 5.2 in (Banerjee et al., 2023a) using Lemma 4, we can obtain (o)
(B) 3, where (a) follows from (14) and (b) from Theorem 5.
|VoG, 1 S _². Then, at < 2x Do.l This result shows that de
< 2x e² < 20²
< 1.
f
F. Supplementary Information for the Experiments
In this section we expand on the mathematical description of each operator learning problem studied in Section 8. We also present further results on how the accuracy of each neural operator model improves as the width m increases. Finally, we provide details about the hyperparameters and datasets used in the training of the corresponding models.
The associated code for the experiments in Section 8 and the ones presented in this appendix are found in https: //gi thub. com wtt./coo///ceeeeeeeeeeeeee wwtp:/asavsssshheseees htt.i//eeelpppppe woporrlloppppttee www.pppiopiipeeeeieeeiiiiiiippie mmiizaiiionn on.
43

---

Optimization for Neural Operators can Benefit from Width
m
10
71
50
171
500
0.8
0.8
0.8
pred
pred
pred
s(z) = Jn u
0.6
data
s(x) = Jo u(g)dg
0.6
data
s(x) Jo u(E)d
0.6
data
0.4
0.4
0.4
0.2
0.2
0.0
0.0
0.0
-0.2
0.2
-1
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
I
Figure 5: Sample solutions obtained for the Antiderivative operator for DONs for m E {10, 50, 500} at the end of the training process (80,000 epochs) for randomly chosen input function. The "data" refers to the ground truth (obtained by standard numerical solver) and "pred" corresponds to the learned operator.
We remark that all experiments with widths m E {10,50 50} were run on a personal computer with one NVIDIA Quadro GPU, while the rest of widths were on Google Colab with single NVIDIA L4 and A100 GPUs.
F.I Antiderivative Operator
We consider a simple one-dimensional Antiderivative or Integral operator given by
s(x) (z)) I u({) dg.
E [0.
(83)
Note that (u) is a linear operator and therefore learnable up to high accuracy. This is evident from the training loss in Figure well from the sample solutions presented Figure for DONs and in Figure 6 for FNOs. We observe that overall increase in the width m leads to higher training accuracy and lower training loss.
The sample size of the training data is 2000, with every input function (1), i [ni]. being one-dimensional Gaussian Random Fields (GRF). For DON training, we choose R 100 input locations and we choose 100 output locations for each input function, i.c., = 100, E [n] (according to the notation in Section 3) For the FNO, the input function is also sampled across 100 locations (i.e., R = 100 using the notation in Section 6); however, since we are interested in the model to provide an output of 100 output locations, we modify the FNO architecture to provide this vector-valued output. For all the experiments we fix the learning rate for the Adam optimizer 10 and with full- training, i.c., the batch size of 2000 for both DONs and FNOs. For testing the trained neural operators, we generate another one-dimensional GRF,
F.2. Centtssoon-Reactionn Operator
We are interested in learning an operator G u(x) + s(x,t) t) for the solution operator of the one-dimensional Diffusion- Reaction equation implicitly given by
Is at
03 Or. + kis² + u(x), (x. E (0, x (0,
D
(84)
'During training; however, for each E [r]], instead of averaging the loss over all the qi points, we simply randomly choose one of the points and evaluate the loss on it. This is strictly done in the interest of computational efficiency, since it is known to not reduce the accuracy of the results for the Antiderivative operator, e.g., (Lu et al., 2021).
"This means that we have output vector of size 100. This is done an alternative an FNO with scalar output which is averaged across the 100 locations (for which we would have R = 100), which is what described in Section 3. We remark that we considered this modification on the output of the FNO just for the sake of computational efficiency, and this only empirically works for the of the Antiderivative operator. Again, as in the case DONs, we randomly sample one of output locations to compute the loss during training.
(according to the notation in Section 3.2) with the understanding that for each input function, we
44

---

Optimization for Neural Operators can Benefit from Width
m-
10
m
50
173 500
0.0
0.0
0.11
0.1
8(x) 1 "[(]de
-0.1
0.2
s( 56
0.2
00.2
0.3
0.3
0..
pred
pred
pred
0.4
data
-0.4
data
00.4
data
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
x
Figure 6: Sample solutions obtained for the Antiderivative operator for FNOs for m E {10,500 50, 5003. The setting the same in Figure 5.
with D >> O and zero initial and boundary conditions, namely,
s(0,t) =(1,t) - and s(2; 0) = =0
along with forcing function u(x) defined by GRF. This is the same setup in (Wang et al., 202 Lu et al., 2021). The corresponding solutions for DONs and FNOs are presented in Figures and 8 respectively. Again, larger width m leads to more accurate solution.
The neural operator aims to learn a mapping from the forcing function to the solution at different times in the interval (0, 1]. in other words, the forcing function would be the input function as defined Section 3. We make use of a slightly modified solver provided https: //thtuub.bb//.///g/// www///reeeeeeeeeee.ooee Physics- in
to generate the training data for the equation. We generate solutions for 5000 input functions which are sampled on 100 points in the space dimension (i.e., the interval (0, 1] for æ in (84) is divided in 100 points) so that R = 100 for DON and R = 100 for FNO (according to the notations in Section 3.1 and Section 6 respectively). For computing the solutions, we are interested in computing them at 100 different times t within the time interval (0, 1] in (84) (in order to be able to plot the two-dimensional map on x and in Figures and 8). This division of both spatial and time dimensions results in a grid 10, 000 points that can be chosen as output locations. For the training of DONs, for each E [n]. we only select 100 scattered points from the grid of output locations (out of their 10, 000 points), so that (8) = 100, which will become the input to the trunk net. However, for the training of FNOs, do choose the full grid output locations and thus modify the FNO to provide 10, 000 outputs instead of the scalar output provided in our theoretical analysis.5 5 We fix the diffusivity as D = 0.01. For all the experiments we use constant learning rate of 3 x 10 and Adam optimizer with a batch size of 4000. For testing the trained neural operators, we generate another one-dimensional GRF.
F.3. Burgers' Equation
The Burgers' equation operator learns an operator G :: u(x) + s(x, 1). where
Øs
Øs
83s 0, (z, E (0,2ml x (0.1]
+ dr
Øt
(85)
s(x, 0) u(x), (0, 2T]]
with v > 0 and periodic boundary conditions
s(0, t) = s(2T, t)
Ös
Øs
dx (0,1)
Zz (2T,, t)
'A scalar output needed if were interested in evaluating the operator only one specific spatial location æ and one specific value of time f; however, as can seen Figures and 8, we are interested in plotting solutions at multiple locations times.
45

---

Optimization for Neural Operators can Benefit from Width
Data
Pred
Error
1.00
0.30
1.00
1.00
0.45
0.75
0.18
0.27
0.75
0.16
0.24
0.32
0.75
0.18
0.27
0.36
0.00
0.09
0.50
0.50
0.00
0.50
0.00
0.09
0.088
-0.09
0,18
0.25
0.27
0.25
0.16
0.25
0.18
0127
0.36
0.32
0.36
0.00
00.45
0.00
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Data
Pred
Error
1.00
0.30
1.00
0.36
1.00
0.021
0.27
0.27
0.018
0.75
0.18
0.75
0.18
0.75
0.012
0.09
0.09
0.006
0.00
0.50
0.50
0.50
0.000
-0.09
0.09
0.006
-0.18
0.25
0.27
0.25
-0.27
0.25
0.018
0.36
0.36
0.00
0.45
0.00
00455
0.00
0.024
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
r
Figure Sample solutions s(z, f) obtained for the Celfusion-Reaccionn operator for DONs for m E {10, 500) given an input uxx). The top row corresponds to 10 and the bottom difference of the ground truth or "Data" (first column) minus the obtained results from the learned DON or "Pred" (second column).
500. The third column represents the pointwise
The corresponding solutions for DONs and FNOs are presented in Figures 9 and 10 respectively. Again, a larger width m leads to a more accurate solution.
The neural operator aims to learn a mapping from the initial condition to the solution at time u(x:) the final solution s(r, 1). This is the operator learning problem originally studied in (Li al., 2021a). We note that the initial condition would then be the input function defined in Section 3. We make use of the datasets publicly available at https: /tte:://giitii tituubb.... com/ http://euuloopeett/o/euuaaaaa available at https: /ttf://drrii /driee. google com/ /tt/ddiiee///ee////e con/drrive/folders wwwnnhnhhnheheeeceeee 6ktEHbl.... ZaXrKU2hp 7p 7pjt-, which comprises of 2048 input functions and corresponding final solution (i.e., ( with associated solution (() 1), i e [2048]). All solutions are calculated for single viscosity / = 0.01. For all the experiments we use a constant learning rate of 10 and Adam optimizer with batch size of 800. We the trained neural operators on simple GRF sampled from the training dataset.
i.e. the mapping from
specifically the Burgers R10 mat dataset
46

---

Optimization for Neural Operators can Benefit from Width
Data
Pred
Error
1.00
1.00
1,800
1.00
1.4
1,575
0.36
0.75
0.75
1 1,350
0.75
0.24
1.0
1.125
0.12
0.8
0.000
0.00
0.50
0.6
0.50
0.675
0.50
0.12
0.4
0.450
0.21
0.25
0.25
0.225
0.25
0.36
0.000
0.0
0.48
0.225
0.00
-0.2
0.00
0.00
0.60
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Data
Pred
Error
1.00
1.00
1.6
1.00
0.0352
1.4
0.75
1.2
0.75
1.2
0.75
0.0264
0.0176
1.0
0.00S8
0.8
0.50
0.50
0.50
0.0000
0.6
0.6
-0,088
0.4
0.25
0.2
0.25
0.2
0.25
0.0176
0.0264
0.0352
0.00
0.2
0.00
0.2
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
x
Figure 8: Sample solutions obtained for the Diffusion Reacttio operator s(x, f) for FNOs for m {10, 500). The setting of the plots is the same in Figure 7 where the top row corresponds to 10 and the bottom row 500.
m = 10
50
500
pred
pred
pred
1.0
1.0
1.0-
s 1) (z)
data
s(r, 1) ([w)(a)
data
s(x, 1) ((v)
data
0.5
0.5
0.5
0.0
0.0
0.0
-0.5
-0.5
-0.5
-1.0
-1.0
-10
0
2
4
6
0
2
4
6
0
4
T
Ø
Figure Sample solutions obtained for Burgers' equation for DONs m E (10, 50, 500). The setting is similar the one in Figure where we plot the obtained solution from the learned operator (denoted by "pred") along with the ground truth (denoted by "data") for different widths.
47

---

Optimization for Neural Operators can Benefit from Width
m 10
m 50
m 500
3 1) ((u)(
0.5
afz, 1) G(u)(z)
0.5
G(u)
0.5
0.0
0.0
0.0
-0.5
pred
-0.5
pred
0.5
pred
data
data
data
0
2
4
0
4
0
4
Figure 10: Sample solutions obtained for the Burgers equation for FNOs for m {10, 50, 500). The setting is the same as in Figure 9.
48


Reviews:
[{"review_id": "8nu3Mu5hOt", "reviewer": "Reviewer_bY3J", "summary": "The paper provides convergence guarantees for neural operator learning, which are valid under assumptions of restricted strong convexity and smoothness of the loss function. The authors demonstrate that two learning operators (DON and FNO) satisfy these conditions. Both theoretical and experimental findings show that networks with greater width achieve superior optimization performance.\n\n## update after rebuttal \n\nThe authors have partially addressed my concerns, so I'm leaning toward acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for taking the time to review our paper. We now respond to the reviewer's concerns.\n\n---\n\nWe respectfully argue that our work **is not incremental** to (Banerjee et al., 2023a) [using the reference as cited in our work] for three reasons:\n\n1. Our work **generalizes** the framework by (Banerjee et al., 2023a), as shown in Section 4. Indeed, we show that the results for feedforward neural networks (FFNs) studied by (Banerjee et al., 2023a) and for the neural operators we study (DONs and FNOs) are **particular** instances of our framework presented in Section 4—despite the architectural differences among FFNs, DONs, and FNOs.\n\n2. Just the fact that our work and (Banerjee et al., 2023a) use a *similar* approach to obtain optimization guarantees (namely, the use of restricted strong convexity (RSC)) **does not imply** that our work is an \"extension\" of the other. If that were the case, this would be similar to saying that *every* work that uses the Neural Tangent Kernel (NTK) approach to obtain optimization guarantees is an extension of the seminal paper (Jacot et al., 2018). We remark that Section 2 lists several works that are based on the NTK approach, yet they are not considered a mere extension of each other: they study networks with different architectures or activation functions. Therefore, we feel that this is an unfair criticism of our work, and ignores the extensive set of new results and analysis we have presented for FNOs and DONs (see details in Appendices D and E).\n\n3. Finally, as carefully explained in Section 1 (lines 063 to 081 of the first column of page 2) and Section 7, the **structural differences** between the neural operators (DONs and FNOs) and FFNs lead to a series of **challenges** in their analysis compared to FFNs. These challenges are reflected in our **involved and non-trivial** analysis: in the case of DONs, they stem from the empirical loss Hessian structure being substantially more complex due to the interactions between two neural networks; in the case of FNOs, they stem from the operator's Hessian structure being substantially more complex due to the Fourier transformations inside the activations. Proving that the RSC method still applies to these neural operators despite the architectural differences **do not** automatically follow from any prior work in the literature, including (Banerjee et al., 2023a). Indeed, before our paper, there was no evident reason to anticipate that the RSC method would be general enough to provide optimization guarantees to neural operators. \n\nFor all the presented reasons, we respectfully reiterate that our paper is not incremental to (Banerjee et al., 2023a)—please see the details in our Appendices D (for DONs) and E (for FNOs). We remark that, to the best of our knowledge, our work is the first one showing optimization guarantees for operator learning. We are grateful to the reviewer to consider the justification we just provided. We hope our arguments will lead to a more positive assessment of our paper.\n\n---\n\nFinally, the code is currently sitting in a private repository and we will provide the code used in our experiments through a public repository if our paper is accepted. We will also address the typo raised by the reviewer.", "claims_and_evidence": "the claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "--", "theoretical_claims": "Theoretical claims are convincing.", "experimental_designs_or_analyses": "good", "relation_to_broader_scientific_literature": "This paper appears incremental compared to https://arxiv.org/pdf/2209.15106 Banerjee et al. 2022. The main difference is that they extend the results to neural operator learning instead of feedforward models with smooth activations. Many of the proofs follow a similar structure to the work of Banerjee et al 2022. The paper's primary contribution seems to be verifying that DONs and FNOs satisfy the RSC and smoothness conditions (in Theorems 2-5), which then allows them to directly apply the convergence framework established by Banerjee et al. 2022.", "essential_references_not_discussed": "--", "other_strengths_and_weaknesses": "**strenghts**\n- clear, well-written\n- interesting topic\n\n**weaknesses**\n- incremental compared to Banerjee et al 2022\n- the code is not provided", "other_comments_or_suggestions": "\"Nevertheless, optimization guarantees for DONs is also an open problem\". line 076 typo (not DON)", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RxmA7iMoPQ", "reviewer": "Reviewer_sqV3", "summary": "The main results of this paper are to derive optimization convergence results for both Deep Operator Networks (DONs) and Fourier Neural Operators (FNOs), under gradient descent (GD). The main techniques in this paper are to show that the empirical loss functions for such two kinds of networks satisfy Restricted strong convexity (RSC) and Smoothness conditions under certain assumptions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for taking the time to review our paper. We now address the points raised by the reviewer.\n\nWe are grateful for the question of guaranteeing convergence for the whole training procedure. If we understood correctly (and we kindly ask the reviewer to correct us if we have not), the reviewer is concerned that because our results (i) hold with high probability and (ii) require larger widths, they may not hold for the whole training procedure. Although this is a valid concern, we respectfully mention that conditions (i) and (ii) can be found in the existing literature of optimization guarantees for deep models—for example, among the works cited in Section 2 for both neural tangent kernel (NTK) and restricted strong convexity (RSC) approaches. \n\nWe also point out that the reason why we end up with condition (i) is because we have to bound the norms of the gradients of the neural operators, which leads to the appearance of weight norms that can only be upper bounded by some constant *with* high probability. In contrast to our work, the work (Cisneros-Velarde et al., 2025) does not need such upper bounds since it studies feedforward neural networks with weight normalization, and so all of its results are deterministic (though condition (ii) is still needed). Finally, we point out that condition (ii) is expressed differently in each optimization approach: in terms of a gradient norm for the RSC approach and in terms of sample size for the NTK approach.\n\nHaving shown that the nature of the conditions that ensure our optimization guarantees is not foreign to the existing literature and is intrinsic to our mathematical approach, we hope the reviewer finds our conditions well justified. Having said that, we believe that deriving deterministic convergence results for the whole training procedure is an important problem—a problem which may require further assumptions and even a different mathematical approach. \n\nWe hope that all of our arguments presented in this rebuttal lead to a more positive assessment of our paper.", "claims_and_evidence": "The results in this paper are more on the theoretical side. Yes, the claims made in the submission are supported by clear and convincing evidence, such as theoretical derivation.", "methods_and_evaluation_criteria": "The results in this paper are more on the theoretical side. Yes, the proposed methods make sense.", "theoretical_claims": "Yes, I checked and thus believe the proofs for theoretical claims in this paper should be correct.", "experimental_designs_or_analyses": "The results in this paper are more on the theoretical side. Yes, the experimental designs and analyses are sound.", "relation_to_broader_scientific_literature": "This paper first derived the optimization convergence result of loss functions using GD for Deep Operator Networks (DONs) and Fourier Neural Operators (FNOs), which extends the previous work in neural operators.", "essential_references_not_discussed": "No. The references are enough to understand the key contributions of the paper.", "other_strengths_and_weaknesses": "Strengths\n\nThis paper derives optimization convergence results of loss functions using GD for Deep Operator Networks (DONs) and Fourier Neural Operators (FNOs), which are also verified by empirical results.\n\nWeaknesses\n\nThe main results in this paper, Theorems 2,3,4,5, are about the step t of GD, and are not deterministic results. Thus, although in each step, the loss decreases with a high probability (which requires the width to be very large), it doesn't guarantee that the loss will decrease after a large number of iterations of GD. Therefore, it will be interesting if the authors can derive some convergence results for the whole training procedure.", "other_comments_or_suggestions": "No.", "questions_for_authors": "It will be interesting if the authors can derive some convergence results for the whole training procedure.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No."}}, {"review_id": "4mgYiw9GCF", "reviewer": "Reviewer_qXNc", "summary": "This paper addresses the  problem of optimization convergence guarantees for neural operators, specifically Deep Operator Networks (DONs) and Fourier Neural Operators (FNOs), when trained using gradient descent (GD). The authors propose a unified optimization framework based on two key conditions: restricted strong convexity (RSC) and smoothness of the loss function. They demonstrate that these conditions are satisfied for both DONs and FNOs, particularly when the networks are wide. The paper provides theoretical guarantees for the convergence of GD in training these neural operators and supports the theory with empirical results on canonical operator learning problems.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for taking the time to review our paper and for the list of questions which we now address.\n\n- **Question 1**: **First subquestion:** Our theoretical work establishes sufficient conditions for optimization and shows that the convergence rate may benefit from width, i.e., *less* optimization steps may be needed to achieve a lower loss value as the network increases in width. Nevertheless, increasing width also means that the computational cost increases *per* optimization step. We believe that the tradeoff between width and computational cost should be determined empirically depending on both (i) the user's tolerance for error and (ii) the available computational resources for training. Assume a user can tolerate a certain $\\epsilon$ value of error on the empirical loss. If the user has limited computational resources and uses a wide network, it could happen that the *time* it takes to do a single optimization step is long, even though there are only a few steps needed to reach the tolerance level $\\epsilon$. In such case, the user should opt for using networks with less width, because, even though there may be more optimization steps to reach the tolerance $\\epsilon$, the *total* amount of time spent during training could be less. If, on the other hand, there is abundant and better computational resources, the user can now afford to increase the width more and even decrease the tolerance level. \nNow, it could also happen that increasing the width will not help too much, depending on the application. Thus, our recommendation would be to not start training with very large widths. For example, if we look at the simulations in Section 8, it is clear that how much help is attained by incrementing width is largely application driven—jumping from a width of 10 to 50 helps the Diffusion-Reaction problem much more than it does to the Burger's Equation. **Second subquestion:** We believe that the question of choosing between increasing depth and increasing width is delicate. Though this goes beyond the scope of our work, we can mention a few things in response to the question by the reviewer: \n  - (i) justifying the benefits of depth versus width in optimization requires a different theoretical framework than ours; \n  - (ii) adding more layers (depth) also adds more weights and thus more matrix-matrix multiplications in the gradient computations, so the computational benefit (if any) of increasing depth versus width has to be carefully considered; and \n  - (iii) deeper networks may need extra architectural changes to avoid *vanishing* gradient effects (such as adding residual connections, normalization layers, etc.), which needs to be taken into account and whose effect on the training of neural operators needs to be studied.\n\n- **Question 2**: Our results are based on the calculation of the RSC condition and smoothness of the empirical loss, which requires calculating the Hessian of the empirical loss—and for this, we ultimately make use of the differentiability of the smooth activation functions. Thus, due to their lack of (global) differentiability, using non-smooth activation functions such as ReLUs would require a different analysis approach—for example, we may need to formulate an alternative notion to the RSC condition for non-smooth functions, as well as an alternative to the smoothness requirement (perhaps using *semi-smoothness* as in the work (Allen-Zhu et al., 2019) which uses ReLUs). We believe this is an interesting future direction. Finally, we would like to point out that, even though our paper only covers smooth activation functions, it encompasses commonly used activations such as sigmoid, hyperbolic tangent, and Gaussian Error Linear Unit (GELU).\n\n- **Question 3**: This is something we hope could be the topic of future theoretical work—to the best of our knowledge, no work has explored SGD with restricted strong convexity. In order to study such problem, we would need to start by adapting our general optimization framework in Section 2 to the stochastic gradient descent setting.\n\n- **Question 4**: This is beyond the scope of our paper, however, we hypothesize that generalization can also benefit from width for both DONs and FNOs. It is relevant to mention that (Kontolati et al. (2022)) has empirically shown that over-parameterization benefits generalization for DONs, as mentioned in Section 2. \n\n---\n\nFinally, we would like to mention that, although the problems we study in our experiments may be regarded as \"toy problems\", they were chosen because they are **representative** of problems typically found in the operator learning literature. Indeed, the seminal papers on both DONs (Lu et al. (2021)) and FNOs (Li et al. (2021)) report results on Burger's equation, and the paper on DONs also reports results on the Antiderivative operator and the Diffusion-Reaction equation. \n\n---\n\nWe hope that this rebuttal leads to a more positive assessment of our paper.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The contributions of the paper are related to scientific machine learning and optimization.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengthes:\n1. The paper provides the first formal optimization convergence guarantees for DONs and FNOs, addressing a significant gap in the literature.\n\n2. The authors complement their theoretical results with empirical evaluations, demonstrating that wider networks indeed lead to lower training losses and faster convergence for both DONs and FNOs.\n\nWeaknesses:\n1. While the empirical results are promising, the experiments are limited to three canonical problems. It would be beneficial to see how the theory holds up on more complex or real-world operator learning tasks\n\n2. The paper does not discuss the practical implications of the theoretical results in detail. For instance, how does the width of the network affect the generalization performance, and what are the trade-offs between width and computational cost?", "other_comments_or_suggestions": "No", "questions_for_authors": "1. The paper suggests that wider networks lead to better optimization convergence. However, wider networks also increase computational cost. How do the authors suggest balancing width and computational efficiency in practice? Why not deeper networks?\n\n2. The paper assumes smooth activation functions. How sensitive are the results to the choice of activation function? Would non-smooth activations (e.g., ReLU) affect the optimization guarantees?\n\n3. Have the authors considered comparing the performance of gradient descent with other optimization methods, such as SGD?\n\n4. The paper focuses on optimization convergence, but how does the width of the network affect the generalization performance of DONs and FNOs? Are there any theoretical or empirical insights on this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yjyn2PqQbf", "reviewer": "Reviewer_bZNF", "summary": "This paper proposes a unified optimization framework using Restricted Strong Convexity (RSC) and smoothness to establish gradient descent convergence guarantees for Deep Operator Networks (DONs) and Fourier Neural Operators (FNOs). The key contributions are as follows:\n1. A theoretical proof that the empirical losses for both operators satisfy RSC and smoothness under over-parameterization (Theorems 2-5).\n2. A demonstration that wider networks improve optimization convergence, supported by both theoretical analysis and experiments.\n3. Empirical validation on three operator learning tasks: antiderivative, diffusion-reaction, and Burgers’ equation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for taking the time to review our paper. \n\n---\n\nWe start by responding to the reviewer's concerns.\n\n- We respectfully state that our work is not a \"limited novel extension\" of (Banerjee et al., 2023a): for a detailed justification we refer to our response to Reviewer bY3J, where we justify our claim on three facts: \n  1. we present a more general optimization framework that is used by both (Banerjee et al., 2023a) and our work; \n  2. several works in the literature share the same underlying mathematical approach and are not considered \"limited novel extensions\" of each other due to their different models; and\n  3. our models have differences that make their analysis complex and non-trivial (please see the details in Appendices D (for DONs) and E (for FNOs)), without automatically following from (Banerjee et al., 2023a).\n\n- Regarding our \"experimental diversity\", we stated in our paper that the objective of Section 8 \"is to show the effect of over-parameterization on the neural operator training and not to present any kind of comparison between the two neural operators.\" We consider that our experiments already achieve the goal of illustrating and complementing the theory of our paper, and thus, there is no need to include additional performance metrics such as the ones mentioned by the reviewer. Indeed, since our analysis is about gradient descent over the **empirical loss**, which is an **L2 loss**, it is sufficient to study the behavior of this L2 loss to prove the benefits of width during training. Finally, since our theoretical results are the centerpiece of our work, having straightforward yet concrete empirical findings that support the theory helps to ensure that the focus remains firmly on the paper's core theoretical contributions.\n\n- We are grateful to the reviewer for suggesting more discussion on concurrent works including (Qin et al., 2024); we will include them in Appendix A. \n\n---\n\nWe now proceed to respond to the questions about our paper.\n\n**Question 1**:\n\nWe are grateful to the reviewer for pointing out this issue. Indeed, as pointed out by the reviewer, the formula in equation (28) of Appendix D.2 (proof of Theorem 2) should be $|\\cos(\\theta' -  \\theta_{t}, \\nabla_{\\theta}\\\\bar{G}\\_{\\\\theta_{t}}) |\\\\geq \\\\kappa$ with the understanding that \n$\\theta'=[{\\theta'\\_f}^{\\top}\\\\;{\\theta\\_{g,t}}^{\\top}]^{\\top}$. Then, the rest of the proof is **still correct** since it becomes virtually equivalent to the proof of non-emptyness of the $Q^t_\\kappa$ set for FNOs (Appendix E.2). This follows from the fact that the $Q^t_\\kappa$ set for FNOs only depends on the *cosine similarity condition*, which is similar to the reduction  (28) we obtained for FNOs by taking $\\theta'$ as above. \nIn conclusion, our proof **still holds** after correcting equation (28) and appropriately changing the notation, following the same proof as the one used for FNOs. Again, thanks to the reviewer for spotting issue.\n\n**Question 2**:\n\nWe are grateful for the suggested ways to strengthen the practical validation of our assumptions. We start by respectfully clarifying three things:\n  1. Our work's aim is not to \"extend NTK theory\" since we use a different mathematical approach (namely, RSC theory). Indeed, Section 2 proposes an alternative optimization framework to the NTK one. \n  2. We must emphasize that Assumption 2 and part of Assumption 4 **are actually found** in the NTK-based paper (Liu et al., 2021a).\n  3. We respectfully believe that Assumption 2 is not \"overly strong\" for *two* reasons. First, it is satisfied for *commonly used* smooth activation functions such as sigmoid, hyperbolic tangent, and Gaussian Error Linear Unit (GELU). Second, besides being used by the NTK-based paper mentioned above, it is used by all RSC-based papers mentioned in Section 2. \n  \nWe now discuss the empirical validation of Assumption 4 (Assumption 2 is not to be verified empirically since it is satisfied by the design choice of the activation functions). Assumption 4 requires *all* iterations to be within a neighborhood of the initialization point. This could certainly be validated in the way suggested by the reviewer—the task would be to find the $\\rho$ and $\\rho_1$ radii, which we suspect will depend on both the initialization point and the training data (training is a non-convex problem and its optimization landscape depends on the training data). Nonetheless, Assumption 4 is *only* required if we want our results to hold for *all* iterations of gradient descent—indeed, we could state a weaker version of Assumption 4 where it only holds for some *finite* set of iterations, and still our guarantees would hold for such iterations. Thus, even if we find that Assumption 4 is not satisfied, it is still possible that our theoretical conditions were met for some iterations along the training procedure. \n\n---\n\nWe hope that this rebuttal leads to a more positive assessment of our paper.", "claims_and_evidence": "The authors demonstrate that increasing network width benefits optimization, supported by Hessian/gradient bounds (Theorems 2-3 for DONs and 4-5 for FNOs) and loss reduction trends (Figures 1-2). However, the local loss descent established in Theorems 2 and 4 does not necessarily imply reduced optimization difficulty with increasing network width. Additionally, there are technical concerns in the RSC proof, particularly in the handling of interaction terms.", "methods_and_evaluation_criteria": "This study proposes a unified theoretical and experimental framework to explore the impact of network width on optimization. While empirical results validate the benefits of width expansion, the theoretical analysis relies on overly restrictive assumptions that limit its generalizability. Furthermore, the practical feasibility of these assumptions is insufficiently validated.", "theoretical_claims": "Upon reviewing the proofs related to the DON section, we identified issues in the construction of the RSC proof. Specifically, certain steps in the derivation lack sufficient justification, raising concerns about the validity of the argument. Additional clarification or further details are necessary to ensure the correctness of the proof.", "experimental_designs_or_analyses": "The experimental results clearly show that increasing width improves optimization performance. However, experiments should also be used to validate the key assumptions (Assumptions 4 and 7) to ensure their practical relevance and applicability.", "relation_to_broader_scientific_literature": "The paper connects to NTK theory (Jacot et al., 2018) and RSC analysis (Banerjee et al., 2023a).", "essential_references_not_discussed": "This work lacks a discussion of concurrent work on operator-specific optimization (e.g., Qin et al., 2024 on Fourier spectral improvements).", "other_strengths_and_weaknesses": "Strengths\n\nOriginality: While the paper provides an RSC-based convergence proof for neural operators, much of the theoretical framework closely follows prior work, particularly Banerjee et al. (2023a), with limited novel extensions specific to neural operators.\n\nSignificance: Offers practical insights into how network width impacts optimization, which could guide applications in scientific computing.\n\nWeaknesses\n\nClarity: Proof sketches in Appendix C.2 lack intuitive explanations of cross-network interactions, making the derivations difficult to follow.\n\nExperiment diversity: All tasks rely on L2 loss, and the absence of adversarial or uncertainty-aware metrics limits the scope of the evaluation.\n\nTheoretical novelty: The theoretical contributions are incremental, as the RSC proof heavily builds on Banerjee et al. (2023a) with minimal adaptation to the specific challenges of neural operators.", "other_comments_or_suggestions": "No other comments. Just see \"Strengths And Weaknesses\" and \"Questions\".", "questions_for_authors": "1. There are some issues in the proof of Theorem 2, where the authors aim to demonstrate that $Q^t_\\kappa$ is non-empty, characterized by the following conditions:\n$| \\cos(\\theta' - \\theta_t, \\nabla_\\theta \\tilde{G}_{\\theta_t}) | \\geq \\kappa$ (cosine similarity condition),\n\n$(\\theta_f' - \\theta_{f,t})^\\top\n\\left(\\frac{1}{n} \\sum_{i=1}^n \\frac{1}{q_i} \\sum_{j=1}^{q_i} \\ell_{i,j}'\n\\sum_{k=1}^K \\nabla_{\\theta_f} f_k^{(i)} \\nabla_{\\theta_g} g_{k,j}^{(i)\\top}\\right)\n(\\theta_g' - \\theta_{g,t}) \\geq 0.$\n\n$(\\theta_f' - \\theta_{f,t})^\\top   \n\\left(  \n\\sum_{k=1}^K \\nabla_{\\theta_f} f_k^{(i)} \\nabla_{\\theta_g} g_{k,j}^{(i)\\top}  \n\\right)  \n(\\theta_g' - \\theta_{g,t}) \\leq 0, \\quad   \n\\forall i \\in [n], \\forall j \\in [q_i]. \n$\n\nTo simplify the analysis, the authors set $\\theta_g' = \\theta_{g,t}$ and claim that belonging to the $Q^t_\\kappa$  set conveniently reduces to the feasibility of the cosine similarity condition as follows:\n\n$\\left| \\cos(\\theta_f' - \\theta_{f,t}, \\bar{g}_f) \\right| \\geq \\kappa.  $\n\nHowever, in this case\n\n$\\left| \\cos(\\theta' - \\theta_t, \\nabla_\\theta \\tilde{G}_{\\theta_t}) \\right| =  \n\\frac{\\langle \\theta_f' - \\theta_{f,t}, \\bar{g}_f \\rangle}{\\| \\theta_f' - \\theta_{f,t} \\| \\| \\nabla_\\theta \\tilde{G}_{\\theta_t} \\|}$\n\nwhich is not equivalent to $\\left| \\cos(\\theta_f' - \\theta_{f,t}, \\bar{g}_f) \\right|$ without additional control\nover $\\bar{g}_f$.\n\n2. Assumptions 2 and 4 appear to be overly strong. In the convergence analysis of NTK, such assumptions are not required. As this\nwork aims to extend NTK theory, it would be valuable to theoretically or empirically validate these assumptions. One feasible approach could be\nto examine the norm of the training trajectory during the neural network training process. This would provide insights into whether these assumptions hold in practice and help justify their necessity in the proposed framework.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No ethical concerns identified. This work focuses on the theoretical analysis of existing methods."}}]

Author rebuttal:

Meta-review: The paper provides valuable theoretical insights into the optimization of Deep Operator Networks (DONs) and Fourier Neural Operators (FNOs), two critical architectures in operator learning. Its main strength lies in developing a unified framework based on Restricted Strong Convexity (RSC) and smoothness conditions, contributing to a deeper understanding of how network width impacts gradient descent convergence. The empirical validation across standard operator learning tasks demonstrates practical improvements with increased network width, aligning theory with observable performance gains.

Whle there are some concerns regarding the rigor and clarity of certain theoretical proofs—particularly related to RSC derivations and the cosine similarity condition, the core theoretical approach remains meaningful and novel enough to merit further consideration in ICML. Given the practical significance of understanding width-dependent optimization behaviors in neural operators, and the potential for meaningful refinements in theoretical presentation and empirical validation, in my opinion, the strengths outweigh the noted weaknesses. Therefore, I recommend acceptance, contingent upon authors adequately addressing the technical and presentation concerns raised.
Decision: Accept (poster)