Title: Generalization in Neural Operator: Irregular Domains, Orthogonal Basis, and Super-Resolution
Venue: NeurIPS 2023
Accepted: no
Keywords: Deep Learning

AI for Science

Neural Operator

Partial Differential Equation
Abstract: Neural operators (NOs) have become popular for learning partial differential equation (PDE) operators. As a mapping between infinite-dimensional function spaces, each layer of NO contains a kernel operator and a linear transform, followed by nonlinear activation. NO can accurately simulate the operator and conduct super-resolution, i.e., train and test on grids with different resolutions. Despite its success, NO's design of kernel operator, choice of grids, the capability of generalization and super-resolution, and applicability to general problems on irregular domains are poorly understood.
To this end, we systematically analyze NOs from a unified perspective, considering the orthogonal bases in their kernel operators. This analysis facilitates a better understanding and enhancement of NOs in the following:
(1) Generalization bounds of NOs,
(2) Construction of NOs on arbitrary domains,
(3) Enhancement of NOs' performance by designing proper orthogonal bases that align with the operator and domain,
(4) Improvement of NOs' through the allocation of suitable grids, and
(5) Investigation of super-resolution error.
Our theory has multiple implications in practice: choosing the orthogonal basis and grid points to accelerate training, improving the generalization and super-resolution capabilities, and adapting NO to irregular domains.
Corresponding experiments are conducted to verify our theory. Our paper provides a new perspective for studying NOs.

Abstract

Neural operators (NOs) have become popular for learning partial differential equation (PDE) operators. As a mapping between infinite-dimensional function spaces, each layer of NO contains a kernel operator and a linear transform, followed by nonlinear activation. NO can accurately simulate the operator and conduct superresolution, i.e., train and test on grids with different resolutions. Despite its success, NO's design of kernel operator, choice of grids, the capability of generalization and super-resolution, and applicability to general problems on irregular domains are poorly understood. To this end, we systematically analyze NOs from a unified perspective, considering the orthogonal bases in their kernel operators. This analysis facilitates a better understanding and enhancement of NOs in the following: (1) Generalization bounds of NOs, (2) Construction of NOs on arbitrary domains, (3) Enhancement of NOs' performance by designing proper orthogonal bases that align with the operator and domain, (4) Improvement of NOs' through the allocation of suitable grids, and (5) Investigation of super-resolution error. Our theory has multiple implications in practice: choosing the orthogonal basis and grid points to accelerate training, improving the generalization and super-resolution capabilities, and adapting NO to irregular domains. Corresponding experiments are conducted to verify our theory. Our paper provides a new perspective for studying NOs.

1 Introduction

Partial differential equation (PDE) operators are widespread in science and engineering. However, 20

traditional numerical methods are known to be slow and ill-suited for high-dimensional problems. 21

As a result, there has been a surge in the popularity of utilizing deep learning techniques for 22

operator learning. Neural operators (NOs) [20, 19, 8, 21] are among the most important models. 23

As a mapping between infinite-dimensional function spaces, each layer of NO contains a kernel 24

operator and a linear transform to convert the input function, followed by nonlinear activation, 25

conducted numerically based on the discretization of the input function on a grid. With appropriate 26

kernels, e.g., shift-invariant kernels in Fourier NO (FNO) [20], facilitates the construction of complex 27

operators. Stacking multiple NO layers further enhances the operator's complexity and demonstrates 28

its universal approximation capabilities [15]. Moreover, empirical evidence reveals that NO exhibits 29

fast convergence and excellent generalization, making it a practical choice. In addition to its operator 30

fitting and generalization abilities, NO can also perform super-resolution tasks. This involves training 31

the model on a low-resolution grid and accurately predicting outcomes on a high-resolution grid. This 32

capability expands the utility of NO beyond precise operator fitting and generalization, showcasing 33

its versatility and accuracy in super-resolution applications. 34

Insufficient understanding surrounds the design of kernel operators, choice of grid points, and 35

NOs' capabilities, despite their notable features in generalization and super-resolution tasks. For 36

instance, while the Fourier basis-based FNO has been established, alternative approaches employing 37 polynomial basis [21] and wavelet basis [8, 28] have been proposed to enhance NO's performance in 38 handling operators related to non-periodic functions and multiscale functions, respectively. However, 39 the underlying reasons for their effectiveness through basis changes remain elusive. Besides, the 40 effectiveness of the super-resolution effect in NO is currently based on empirical observations, and 41 the factors influencing its efficacy remain unknown. Furthermore, NO is commonly trained on a 42 predefined uniform grid with a predetermined sparsity level. The impact of utilizing grids with 43 varying sparsity levels or randomly sampled grid points on NO's performance remains unexplored. 44

45

46

47

48

49

On the other hand, the applicability of NOs to general problems on irregular domains poses a significant challenge, mainly because the kernel operators utilized in popular models [20, 21, 28] are typically defined on regular bounded domains. Extending NO, based on orthogonal bases, to encompass general irregular domains remains a formidable task. The difficulty lies in effectively incorporating the physical information of the domain into the design of NO's basis and grid.

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

In this paper, we provide a novel perspective for studying NOs by examining the role of orthogonal bases within their kernel operators. The kernel operators in NOs are constructed such that their eigenfunctions are predefined orthogonal bases, and eigenvalues are trainable parameters. This unified view enables the analysis of NOs in various aspects. Firstly, we establish generalization bounds for NOs, considering them mappings between infinite-dimensional function spaces. Moreover, by carefully designing orthogonal bases for the input domain and functions, NOs can be constructed on irregular domains, improving generalization. The impact of grid points on NO convergence and generalization is also investigated. Additionally, we analyze factors influencing the super-resolution error in NOs. Our theory carries practical implications, such as selecting appropriate orthogonal bases and grid points to accelerate convergence, enhance generalization and super-resolution abilities, and adapt NOs to irregular domains. Extensive experiments are conducted to validate our theory, which sheds new light on understanding NOs and improving their properties in practical applications.

2 Preliminary

2.1 Notation & Problem Definition

2.2 Understanding Neural Operator

Given an input function f : Ω ⊂ R d → R h where h is the hidden/output dim, and the complete orthogonal basis set { ϕ i } ∞ i =0 on L 2 (Ω; R ) arranged with increasing frequencies/degrees/orders, each

layer of NOs contains three operations: (1) the kernel transform, (2) the linear transform, and (3) the 87 nonlinear activation. The overall model structure can be summarized as follows. 88

Overall Model . Denote the input function as ˆ u 0 ( x ) , then the recursive formulation of NO is 89

where v ( x ) is the output, l is the layer index, and L is the total number of layers. ˆ u and u are the post-activation and pre-activation input functions, respectively.

Kernel Transform κ . We set up trainable kernels such that the basis set with N modes lowest frequencies form the eigenfunctions of the kernels, i.e., K ( B ; x , y ) = ∑ N modes i =0 B i ϕ i ( x ) ϕ i ( y ) where B i ∈ R h × h and B ∈ R N modes × h × h are trainable, and h is the hidden dim. Thanks to the lowrank property of the kernel, i.e., the dimension of its kernel spectra are only N modes, κ ( f )( x ) = ∫ K ( B ; x , y ) f ( y ) d y = ∑ N modes i =0 B i c i ϕ i ( x ) ∈ R h , where c i = ⟨ f, ϕ i ⟩ ∈ R h is the dimension-wise inner product between the input f and the basis ϕ i . This form is exactly the same as the kernel in FNO if the basis ϕ i is Fourier series. For implementation, we usually (1) project the input function onto the basis with N modes lowest frequencies to get { c i } N modes i =1 ; (2) linear transform the coefficients c i with the matrices B i ; (3) finally project from the coefficient space back to the function space by multiplying ϕ i ( x ) to each B i c i .

Linear Transform ω . The linear transform ω is a straightforward operation ω [ f ]( x ) = W f ( x ) , W ∈ R h × h , f ( x ) ∈ R h . Unlike the kernel operator, the linear transform can be conducted in the original function space without projection. The final output is obtained by combining the results from the kernel transform and the linear transform: T [ f ]( x ) = κ ( f )( x ) + ω [ f ]( x ) . Finally, the transformer function is passed through a nonlinear activation, i.e., σ ( T [ f ]( x )) .

Examples of Bases . In FNO [20], the orthogonal basis used is the Fourier basis, specifically over a bounded regular domain. However, alternative orthogonal bases are employed in other variations of FNO, such as those discussed in [8, 21], including polynomial and wavelet bases. Regarding the implementation of the most popular FNO [20]: (1) is the fast Fourier transform (FFT); (2) is its coefficient transform; (3) is the inverse FFT.

Numerical Integration . In practice, input functions are discretized on a grid. Transformations and activations can be applied pointwise, but kernel transforms relying on the inner product with bases require numerical integration on the grid. However, the linear transform does not need numerical integration and can be implemented more easily in the original function space.

Efficiency of Numerical Integration . In FNO, the Fast Fourier Transform (FFT) is used to compute integrals, with a time complexity of O ( N grid log N grid ) . However, in practice, only the first N modes integrations between the basis and input need to be calculated. This reduces the complexity to O ( N grid ) since N modes ≪ N grid. This approach, adopted in Geo-FNO [19], ensures that the integration step does not become a bottleneck in the NO model's time complexity. As a result, NO models are generally faster than Transformers.

3 Theory

We have examined the traditional function perspective of NOs. However, machine learning often overlooks the complexity of mappings between infinite-dimensional functions. To address this, we propose studying NOs in the coefficient space. By representing NOs as mappings between infinite sequences of real numbers, derived from the expansion of input functions on an orthogonal basis, we can leverage the extensive literature on mappings between finite-dimensional vectors and extend it to our context. This enables a comprehensive analysis of NOs from a new perspective.

Given a complete orthogonal basis { ϕ i } ∞ i =0 , the input function f and the output function G ( f ) can be expanded as f = ∑ ∞ i =0 c i ϕ i , G ( f ) = ∑ ∞ i =0 d i ϕ i where c i = ⟨ f, ϕ i ⟩ and d i = ⟨G ( f ) , ϕ i ⟩ . For the infinite sums to converge, the infinite sequences { c i } , { d i } ∈ l 2 . So, the operator learning problem on G can be abstracted to a mapping between infinite sequences between numbers in the l 2 space, i.e.,

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

NOs aim to learn the mapping { c i } ∞ i =0 ↦→{ d i } ∞ i =0 . Thus, we can define the input space from the viewpoint of coefficients B := { ( ⟨ f, ϕ i ⟩ ) ∞ i =1 , f ∈ A} . where A is the space of input functions.

We reinterpreted the operations in NOs from the sequences mapping perspective in l 2 .

Kernel Transform . It maps f = ∑ ∞ i =0 c i ϕ i to κ ( f ) = ∫ K ( B ; x, y ) f ( y ) dy = ∑ N modes i =0 B i c i ϕ i , which can be abstracted to: c ≤ N modes ↦→ B ≤ N modes c ≤ N modes , c >N modes ↦→ 0 , due to the kernel's local rankness and truncation at the N modes-lowest frequency.

Linear Transform . For an input function f ( x ) , the linear transform maps f ( x ) to W f ( x ) . Also, considering the channel-wise operation, from the sequence space point of view, it is c i ↦→ W c i . Compared to the previous kernel transform, the linear coefficient is the same for all elements in the sequence, but those are different and truncated in the previous kernel transform.

̸

Nonlinear Activation . The nonlinear activation σ is an abstract and fixed mapping between l 2 to itself, denoted Σ : l 2 → l 2 , c ↦→ Σ c . In the original function space, the mapping is from the input function f ( x ) to σ ( f )( x ) where σ : C (Ω) → C (Ω) . However, in the sequence space, the original c i is c i = ⟨ f, ϕ i ⟩ , and (Σ c ) i = ⟨ σ ( f ) , ϕ i ⟩ . The i th entry of Σ c may depend on all c i for every i since the activation is performed on the whole input function. For instance, for the input function f ( x ) = cos( kx ) and the cosine basis, only c k = 0 while other entries equal zero. But σ ( f )( x ) = σ (cos( kx )) is a very complicated function even for simple σ like ReLU, Sigmoid, and Tanh activations, with the post-transformed coefficients (Σ c ) i = 0 for all i . Although it is abstract, the Lipschitz continuity is kept:

Proposition 3.1. If σ is L -Lipschitz, i.e., | σ ( x ) -σ ( y ) | ≤ L | x -y | , then the mapping Σ is also L -Lipschitz in the l 2 space.

Model Summary . Denote the input function as ˆ u 0 ( x ) , and its expansion over the orthogonal basis to be { ˆ c 0 ,i } ∞ i =1 , i.e., ˆ u 0 = ∑ ˆ c 0 ,i ϕ i , then the recursive formulation of NO in the coefficient space is

where v i is the coefficient for the output function, i.e., the output function v ( x ) = ∑ ∞ i =1 v i ϕ i ( x ) , l is the layer index, and L is the total number of layers. For all the indices c l,i , the first l is for the layer, while the second i is for the index of the orthogonal basis. ˆ c and c are the post-activation and pre-activation input coefficients, respectively.

3.1 Generalization of NOs

From the sequence perspective, we can derive the generalization bound of NOs via the robustness bound [29, 13]. The generalization gap of the model given in equation (2) can be bounded as follows. Theorem 3.1. (Generalization bound of NOs) For any δ ∈ (0 , 1) , with probability at least 1 -δ over the choice of random samples S = { f j } N train j =1 ∼ µ , let the model parameters after optimization to be θ S = {{ B l,i } N modes i =1 , W l } L l =0 , the following bound holds:

for all γ > 0 , where K = N ( γ/ 2 , B , ∥ · ∥ l 2 ) is the γ/ 2 -covering number of the input space B under the norm ∥ · ∥ l 2 . M is the upper bound of the loss function.

All proofs are presented in the Appendix. The generalization bounds, similar to vanilla neural nets, rely on the products of multilayer parameter norms [2, 29]. Theorem 3.1 offers a more detailed characterization of the generalization bounds compared to the findings in [14], and it guides selecting orthogonal bases in NO. We will delve into this topic further in Section 4.

Extension to Discretized NOs . In Theorem 3.1, we primarily focus on continuous NOs. However, the presented theory can be extended to discrete NOs by substituting the infinite-dimensional l 2 space with a finite-dimensional vector space. In this context, inner products and orthogonal bases can still be defined. The modification lies in the term involving the covering number in the bound N ( γ/ 2 , B , ∥ · ∥ l 2 ) . Here, we replace the l 2 space and norm with the finite-dimensional Euclidean space corresponding to the discrete NO and its vector 2-norm.

3.2 Super-resolution Error 178

179

180

181

182

183

184

Super-resolution involves training a model on a low-resolution grid and evaluating it on a highresolution grid, with the expectation of comparable performance. While FNO [20] demonstrates excellent super-resolution capabilities, the underlying reasons remain poorly understood. This understanding is crucial for two reasons: (1) enabling training on sparse grids, leading to reduced training time, and (2) ensuring NOs can effectively handle inputs of the same function on different grids and produce satisfactory results.

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

Before delving into the analysis, it is important to address the numerical integration errors that arise when training NOs on low-resolution grids compared to high-resolution grids during superresolution. Intuitively, if the integration error is significant, there will be notable discrepancies in the integral values obtained from the sparse training grid and the high-resolution testing grid, resulting in inconsistent model performance between training and testing.

As u l , ˆ u l , and v represent variables in continuous NOs, we use U l , ˆ U l , and V to represent variables in the discrete NOs using numerical integration rule ˆ ∫ g ≈ ∑ N grid i =1 w i g ( x i ) for any integrand g :

where f is the input function, w i is the weight for numerical integral at the grid point x i . The error in numerical integration generally depends on two factors: the grid size (defined as e grid ( N grid ) ) and the smoothness of the integrand function (defined as e func ( f ) ).

For instance, on a uniform grid over the interval [ a, b ] , the integral is approximated using the Darboux method ∫ b a f ≈ 1 /N grid ∑ N grid i =1 f ( x i ) where x i = a +( i -1)( b -a ) /N grid, which yields an integration error of O ( f ′′ ( ξ ) /N grid ) where ξ ∈ ( a, b ) , i.e., e grid ( N grid ) = 1 /N 2 grid , e func ( f ) = f ′′ ( ξ ) . However, using the trapezoidal rule instead, the error can be reduced to O ( f ′′ ( ξ ) /N 2 grid ) without requiring additional computations. FNO [20] assumes that the input function is periodic, so the error of the uniform grid decreases to O ( f ′′ ( ξ ) /N 2 grid ) . For the Gaussian quadrature on the interval [ a, b ] , the error can be reduced to O ( ( N grid !) 4 f 2( N grid ) ( ξ ) / [(2 N grid )!] 3 ) for ξ ∈ ( a, b ) [10]. With the background, we can write the discretization error of NOs:

Theorem 3.2. (Discretization error of NOs) Suppose the numerical integration's error is e grid ( N grid ) e func ( f ) where N grid is the grid size, and f is the integrand, then the discretization error of discrete NOs compared with continuous ones due to numerical integral is upper bounded by

The discretization error in discrete NOs relies on the norm of model parameters and the accuracy of the integration method employed for integrating intermediate output functions. Building upon this, we can derive the super-resolution error of discrete NOs. Firstly, we bound the prediction error of continuous NOs across all points in the domain Ω (i.e., the super-resolution error of continuous NOs). During the training phase, NOs are trained on a finite training grid, leading to this error. Subsequently, we bound the discrepancy between continuous and discrete NOs, corresponding to the discretization error stated in Theorem 3.2. These two terms are reflected in the following theorem.

Theorem 3.3. (Super-resolution error of NOs) Assuming a uniform grid on a bounded regular domain with N grid points in the FNO [20] setting, under the same notation as Theorem 3.1. Then, the super-resolution error of the discrete NO model for the input function f with intermediate output ˆ U l (as detailed in equation (3.2)) can be bounded as follows:

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

The first term pertains to the integral error in Theorem 3.2, while the second term relates to the interpolation and generalization ability of NOs across the entire domain. These factors significantly influence the super-resolution of NOs. Understanding and addressing these factors is crucial for enhancing super-resolution accuracy in NOs, which will be thoroughly discussed in Section 4.

4 Implication and Application of the Theory

In this subsection, we introduce the implications and applications of the proposed theory and correspond them to the following numerical experiments.

Tighter Bound in Theorem 3.1 . Our bound's proof and form are much more general and tighter than previous work [14]. In particular, in terms of parameter matrix norm contributed by each layer, the bound in [14] depends on ∥ W l ∥ F + ∥ B l ∥ F N d/ 2 modes where ∥ · ∥ denotes the Frobienus norm. So, our reliance of max i {∥ B l,i + W l ∥ 2 , ∥ W l ∥ 2 } is a significant improvement. Additionally, our bound suggests that increasing the number of modes does not necessarily increase the complexity of the model. However, it is important to note that selecting high-frequency basis functions to fit high-frequency noise can adversely impact generalization, as it leads to larger values of ∥ B l,i + W l ∥ 2 . We shall verify the advantage of our bound in Experiment 6.1.

Super-Resolution Error . From Theorem 3.3, super-resolution in NOs depends on two critical factors: (1) the accuracy of integration trained on low-resolution grids, and (2) the density of the low-resolution grid to facilitate generalization to other points. It is important to note that numerical integration accuracy does not necessarily imply grid density, especially in the case of low-precision integration formats. We shall experiment on the super-resolution error in Experiment 6.2.

Choice of Grid . The choice of grids depends on the specific function and its domain. In the case of a finite interval, a uniform grid is commonly used, and integration can be performed using the trapezoidal rule. Furthermore, selecting a grid that allows for accurate integration effectively captures the function's characteristics at a finite number of points. For instance, in density functional theory (DFT), where the domain is R 3 , the function typically involves a multi-center Gaussian mixture. In such cases, designing an integral scheme tailored to multi-center functions is more reasonable. For example, selecting points in the vicinity of each Gaussian center enables better characterization and integration accuracy for the input function. We will conduct the corresponding DFT experiment on molecules in 6.3.

Extending NOs to Irregular Domains . The limitation of FNO [19, 23] is its reliance on Fourier bases, which restricts its application to regular domains and limits its usage in real-world complex geometries. Geo-FNO [19] assumes that irregular domains can be mapped to regular ones through a bijection, which is not applicable for arbitrarily irregular domains. Fortunately, under our framework in equation (2), we can overcome this limitation by utilizing random Fourier features (RFFs) and polynomials on irregular domains. By employing Gram-Schmidt orthogonalization, we can obtain an orthogonal basis and perform numerical integration on a discrete grid for the inner product. Moreover, we can select a suitable orthogonal basis based on the domain, such as Gauss-Hermite polynomials for the unbounded whole space R . As a result, we successfully extend NOs to irregular domains, enhancing their applicability in various scenarios. In the first setting of Experiment 6.3, we validate our general NOs on unbounded domains.

Guiding the Choice of Orthogonal Basis . The choice of basis in NOs significantly impacts their expressiveness, as highlighted by Theorem 3.1. If the target function can be well represented by a finite combination of basis functions, a small number of modes ( N modes) is sufficient. This leads to fewer parameters and increased model efficiency. Conversely, if an infinite series of basis functions is needed to expand the target function, the model tends to generalize poorly. For example, Fourier bases are suitable for periodic functions, wavelets excel in capturing rapid changes and discontinuities, and polynomials (e.g., orthogonal Legendre polynomials) provide a versatile basis for all functions. These bases have distinct capabilities and cannot efficiently represent each other. Additionally, wavelets are effective in handling multi-scale and multi-physics problems. Combining multiple basis sets can yield superior results by leveraging complementary effects. The selection of the basis is guided by the characteristics of the function and the operator in the dataset. For instance, if the function exhibits periodicity in certain subdomains but not in others, a combination of Fourier and polynomial basis functions can effectively model both parts simultaneously, which is verified in Experiment 6.4.

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

5 Related Work

Orthogonal Basis in NOs . Various orthogonal bases are adopted for kernel transform in neural operators, e.g., Fourier NO (FNO) [20] and its variant [25, 27, 30] adopt Fourier bases, Geo-FNO [19] utilizes Fourier bases under deformation, [21] uses orthogonal Legendre polynomials, and [28, 8] use multi-wavelets.

NOs on Arbitrary Domains . Some operator networks different from NO are grid-free. DeepONet [22] encodes the input function and the grid, respectively, and then combines them by dot product as the operator network output. MIONet [12] extends DeepONet to multiple input functions. Transformer operators [3, 18, 9] directly process the inputs by the attention mechanism [26]. The NO we proposed can also be applied to any domain and incorporates its prior knowledge, so our NO generally outperforms these two approaches.

Theory of Neural Operators . DeepONet and FNO are analyzed theoretically in the literature. Theory on DeepONet [16, 7] relies on the discretization over function and the input grid to transform the model into mapping between finite-dimensional vector space. The generalization theory on FNO [14] relies on discretization and proposes Rademacher complexity bounds. [15] proves the universal approximation and errors bound for approximating Darcy type elliptic PDE and the incompressible Navier-Stokes equations. [5, 6, 23] proposes theories for NO's approximation. [4] proves convergence rates for linear operators.

6 Experiment

6.1 Validation of Theorem 3.1

This section provides empirical evidence demonstrating the superior tightness and quality of our generalization bound presented in Theorem 3.1 compared to related works such as [14]. It is customary in the literature to compare the numerical values of various generalization bounds as a means to showcase their tightness, e.g., [24, 1, 11, 2]. In Figure 1, we provide numerical values of the generalization bounds for FNO models trained on four distinct datasets (1D Burgers, 2D Darcy Flow, 2D+time Navier-Stokes equation, 3D Navier-Stokes equation) as provided by FNO [20]. Following FNO's experimental setup, we normalize the value of our proposed bound to 1 for clarity. Remarkably, our robustness-based bound outperforms existing bounds by 2-3 orders of magnitude, underscoring its superior tightness and reliability.

1D Burgers

2D Darcy

2D+Time NS

3D NS

Datasets

10 0

10 1

10 2

10 3

10 4

Bound Values

1789.59

5623.29

626.22

732.52

1.00

1.00

1.00

1.00

Our bound is much tighter than previous bound.

Rademacher Complexity Bound

Our Robustness Bound

6.2 Super-resolution Error

We validate Theorem 3.3 that super-resolution error is affected by both the integration scheme and 300 the grid size. We conduct experiments on the previously mentioned Burgers dataset from FNO 301

Ngrid

=2 6

Ngrid

=2 8

Ngrid

=2 10

10

4

10

3

10

2

10

1

Super-res Gap:

Ltest

reg

Ltest

sr

1e-01

9e-02

6e-02

2e-02

6e-03

2e-03

4e-03

9e-05

5e-05

Super-Resolution Error and Integral Scheme

Random:

O

(1/

N

)

Darboux:

O

(1/

N

)

Trapezoid:

O

(1/

N

2

)

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

√

[20], where we use the random scheme (error: 1 / N , blue), Darboux rule based on a uniform grid (error: 1 /N , green), and trapezoidal rule based on a uniform grid (error: 1 /N 2 , red) as integration schemes in FNO [20]. The grid sizes are chosen to be 2 6 (first column), 2 8 (second column), 2 10 (third column) points. The super-resolution performance is evaluated by the super-resolution gap L test-reg -L test-sr originally defined in Theorem 3.3. Figure 2 illustrates the following findings: (1) increasing the grid size for the same integration scheme improves super-resolution performance, and (2) for the same grid size, integration schemes with higher accuracy yield lower super-resolution error. These results confirm the validity of our theory. It is important to note that the original FNO framework cannot operate on random grids. Our reinterpretation of NOs, utilizing random grids as the integration format and Fourier bases, enabled this capability.

6.3 NOs on Unbounded Domain

In this subsection, we select several Density Functional Theory (DFT) Hamiltonian operators to test our NOs with suitable orthogonal bases with other strong baselines.

Baselines . (1) FNO [20]: cannot be adopted on an arbitrary domain. (2) Geo-FNO [19]: uses a bijection to map the irregular domain to a regular one and perform FNO. Here the unbounded domains can be mapped to the regular domain by the bijection tan -1 , which can conduct super-resolution. (3) DeepONet [22] can operate on arbitrary domains but accepts fixed-length discretized input function, which restricts its super-resolution performance. (4) Linear Transformer (LT) [3] and (5) OFormer [18] are all transformers for operator learning, they can handle arbitrary domains and grids.

In Density Functional Theory (DFT), the Hamiltonian operator plays a crucial role in characterizing the ground state energy through its spectra. In this subsection, we evaluate different NOs' performance in learning Hamiltonian operators defined on unbounded domains across various dimensions.

QHO . In the quantum harmonic oscillator (QHO), the Hamiltonian operator, given the wave function ϕ , is ( ˆ H QHO ϕ ) ( x ) = -1 2 ∇ 2 ϕ ( x ) + 1 2 x 2 ϕ ( x ) , x ∈ R . We use random linear combinations of the 1D Hermite polynomial for data generation: ϕ i ( x ) = 1 π 1 4 2 i 2 √ i ! H i ( x ) e -x 2 2 , x ∈ R , The grid for training is the points in the Gauss-Hermite quadrature with 32 points, and the super-resolution testing grid is with degree 64. Geo-FNO adopts the grid generated by the tan -1 transform of uniform grid over ( -π 2 , π 2 ) . The orthogonal basis in our NO is Gauss-Hermite polynomials.

Molecules . Following D4FT [17], we consider Hamiltonian operators in real-world 3D molecules, which take the wave function ϕ ( r ) , and the density ρ ( r ) as inputs is given by four different terms (kinetic, external potential, Hartree and exchange-correlation):

KS-DFT offers natural basis sets that can be linearly combined to generate the training functions. 333 Additionally, grids with varying resolutions, characterized by levels, are provided based on the 334 multi-center Gaussian nature of the functions in DFT. In our experiments, we select level 1 for 335 training and level 2, which has more points, for testing. In Geo-FNO, the grid is generated using 336 the tan -1 transform of a uniform grid over the domain ( -π 2 , π 2 ) 3 . To ensure a fair comparison, the 337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

number of grid points in Geo-FNO is similar to the quadrature used in D4FT. For the D4FT setting, we consider three molecules: CO 2 , water, and CH 4 .

Results . Relative L 2 errors for DFT experiments are shown in Table 1. (1) NO-Ours exhibits slightly superior performance on regular testing and significantly outperforms in super-resolution tasks. (2) The uniform grid deformation in Geo-FNO is less efficient compared to quadrature, resulting in NO-Ours surpassing Geo-FNO. (3) As the molecule size increases, atom distribution extends throughout the R 3 domain. Consequently, the input wave function and density function disperse near the atoms rather than concentrating near the unit box. Thus, using a unit box grid in Geo-FNO becomes inefficient. (4) NO-Ours adapts to various grids based on the problem, utilizing efficient quadrature points in D4FT to accurately represent the input wave function and density. As a result, NO-Ours achieves super-resolution with minimal additional error. This highlights the significance of selecting appropriate integral points based on the characteristics of the input functions.

6.4 Conbining Multiple Bases

We try the advection equation u t + u x = 0 , x ∈ [0 , 1] , t ∈ [0 , 1] with/without periodic boundary conditions taken from [23]. Given the initial condition u 0 ( x ) , we aim to learn the nonlinear operator G : u 0 ( x ) ↦→ u ( x, t ) 2 , ( x, t ) ∈ [0 , 1] × [0 , 1] with the hybrid input functions u 0 ( x ) = h 1 1 { c 1 -w 2 ,c 1 + w 2 } + √ max( h 2 2 -a 2 ( x -c 2 ) 2 , 0) where c 1 , c 2 , w, h 1 , h 2 are randomly chosen to generate samples. The full problem is named Advection (1), which is periodic in both t and x axis, i.e., u ( x, 0) = u ( x, 1) and u (0 , t ) = u (1 , t ) . To construct a non-periodic problem, we truncate the target function on ( x, t ) ∈ [0 , 1] × [0 , 0 . 5] , so that it is periodic on the x -axis but not the t -axis. We use sinusoidal and/or polynomial bases. Specifically, there are two axes, for NO-Sin/Sin, we use sinusoidal bases on both axes; for NO-Poly/Poly, we use Legendre polynomial bases on both axes; for NO-Sin/Poly, we use sinusoidal on the x -axis and polynomial on t -axis. More specifically, if we have a set of basis functions { ϕ i ( x ) } ∞ i =0 on the domain Ω x along the x -axis, and another set of basis functions { ψ j ( t ) } ∞ j =0 on the domain Ω t along the t -axis, then the set of tensor product basis functions { ϕ i ( x ) ψ j ( t ) } ∞ i,j =0 forms a basis for the two-dimensional space Ω x × Ω t . We train all models with 1000 epochs.

Table 2 verifies the effectiveness of sinusoidal bases for periodic functions and polynomials for general non-periodic functions. Additionally, the combination of multiple different bases has been shown to be effective, taking into account the properties of the data. This approach deviates from previous works that typically focus on utilizing a single basis.

7 Conclusion

This paper proposes a novel perspective for studying NOs. We have provided a comprehensive understanding of NO through a detailed analysis of the infinite sequence space under orthogonal basis projection. Based on this versatile framework, we have proposed a method to adapt NO to arbitrary complex domains. We have also analyzed the generalization bound of NO, demonstrating its superiority over previous works. Furthermore, we have explained the importance of selecting the type and quantity of basis functions in NO, emphasizing the benefits of using multiple bases in a complementary manner based on operator characteristics. Additionally, we have examined the impact of grid points on super-resolution error, highlighting the crucial role of the integration format associated with the grid and the density of the grid itself. All the theoretical analyses have been extensively validated through experiments on multiple data sources, including numerical PDE and DFT. We shed new light on understanding NOs and improving them in practical applications.

381

382

383

384

385

386

References

Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning , pages 254-263. PMLR, 2018.

Peter Bartlett, Dylan J Foster, and Matus Telgarsky. Spectrally-normalized margin bounds for neural networks. arXiv preprint arXiv:1706.08498 , 2017.

Shuhao Cao. Choose a transformer: Fourier or galerkin. Advances in Neural Information 387 Processing Systems , 34:24924-24940, 2021. 388

Maarten V de Hoop, Nikola B Kovachki, Nicholas H Nelsen, and Andrew M Stuart. Conver389 gence rates for learning linear operators from noisy data. arXiv preprint arXiv:2108.12515 , 390 2021. 391

Tim De Ryck and Siddhartha Mishra. Generic bounds on the approximation error for physics392 informed (and) operator learning. arXiv preprint arXiv:2205.11393 , 2022. 393

Nicola Rares Franco, Stefania Fresca, Andrea Manzoni, and Paolo Zunino. Approximation 394 bounds for convolutional neural networks in operator learning. Neural Networks , 161:129-141, 395 2023. 396

397

398

Pulkit Gopalani, Sayar Karmakar, and Anirbit Mukherjee. Capacity bounds for the deeponet method of solving differential equations. arXiv preprint arXiv:2205.11359 , 2022.

399

400

401

402

403

404

405

Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet-based operator learning for differential equations. Advances in neural information processing systems , 34:24048-24062, 2021.

Zhongkai Hao, Chengyang Ying, Zhengyi Wang, Hang Su, Yinpeng Dong, Songming Liu, Ze Cheng, Jun Zhu, and Jian Song. Gnot: A general neural operator transformer for operator learning. arXiv preprint arXiv:2302.14376 , 2023.

Francis Begnaud Hildebrand. Introduction to numerical analysis . Courier Corporation, 1987.

Zheyuan Hu, Ameya D Jagtap, George Em Karniadakis, and Kenji Kawaguchi. When do 406 extended physics-informed neural networks (xpinns) improve generalization? SIAM Journal on 407 Scientific Computing (SISC) , 2022. 408

409

410

Pengzhan Jin, Shuai Meng, and Lu Lu. Mionet: Learning multiple-input operators via tensor product. SIAM Journal on Scientific Computing , 44(6):A3490-A3514, 2022.

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

Kenji Kawaguchi, Zhun Deng, Kyle Luh, and Jiaoyang Huang. Robustness implies generalization via data-dependent generalization bounds. In International Conference on Machine Learning (ICML) , 2022.

Taeyoung Kim and Myungjoo Kang. Bounding the rademacher complexity of fourier neural operator. arXiv preprint arXiv:2209.05150 , 2022.

Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. The Journal of Machine Learning Research , 22(1):13237-13312, 2021.

Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis. Error estimates for deeponets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications , 6(1):tnac001, 2022.

Tianbo Li, Min Lin, Zheyuan Hu, Kunhao Zheng, Giovanni Vignale, Kenji Kawaguchi, A.H. Castro Neto, Kostya S. Novoselov, and Shuicheng YAN. D4FT: A deep learning approach to kohn-sham density functional theory. In The Eleventh International Conference on Learning Representations , 2023.

Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning. arXiv preprint arXiv:2205.13671 , 2022.

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. arXiv preprint arXiv:2207.05209 , 2022.

Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. International Conference on Learning Representations (ICLR) , 2021.

Ziyuan Liu, Haifeng Wang, Kaijuna Bao, Xu Qian, Hong Zhang, and Songhe Song. Render unto numerics: Orthogonal polynomial neural operator for pdes with non-periodic boundary conditions. arXiv preprint arXiv:2206.12698 , 2022.

Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence , 3(3):218-229, 2021.

Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering , 393:114778, 2022.

Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring generalization in deep learning. arXiv preprint arXiv:1706.08947 , 2017.

Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. In International Conference on Learning Representations , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Gege Wen, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, and Sally M Benson. U-fno-an enhanced fourier neural operator-based deep-learning model for multiphase flow. Advances in Water Resources , 163:104180, 2022.

Xiongye Xiao, Defu Cao, Ruochen Yang, Gaurav Gupta, Gengshuo Liu, Chenzhong Yin, Radu Balan, and Paul Bogdan. Coupled multiwavelet operator learning for coupled differential equations. In The Eleventh International Conference on Learning Representations , 2023.

Huan Xu and Shie Mannor. Robustness and generalization. Machine learning , 86(3):391-423, 2012.

Jiawei Zhao, Robert Joseph George, Yifei Zhang, Zongyi Li, and Anima Anandkumar. Incre459 mental fourier neural operator. arXiv preprint arXiv:2211.15188 , 2022. 460

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the generalization error of neural operators that contain kernel operations. Under the basic setting of neural operators (such as FNO), this paper establishes upper bounds of the excess risk of neural operators. How to apply NO to irregular domains, and the error analysis of super resolution are also discussed. The techniques of this paper is rather standard. Some parts of the upper bounds are not clear. I think the contribution of this work is incremental.\n\nSTRENGTHS:\n1. This paper provides an upper bound for the excess risk of neural operators.\n\n2. The upper bound in this paper improves the one in [14]\n\n3. Extension of NOs to irregular domains is discussed.\n\n4. The super resolution error is studied.\n\nWEAKNESSES:\n1. The technique used in this paper is pretty standard. The author should emphasis their novelties. \n\n2. Theorems in this paper is not impressive and is unclear. For example, the error bound in Theorem 3.1 depends on the network structure, parameters, and the covering number of the space $B$, which is infinite dimensional. If the norms of network parameters are all larger than 1, the upper bound can be very large. Since the space $B$ is infinite dimensional, the covering number can also be very large, which makes the result less attractive. On the other hand, the magnitude of the training loss is also unclear. I believe the training loss depends on the network's width and depth, which relates closely to the upper bound in this paper. It will make the paper much stronger if these relations are analyzed clearly and a more clear upper bound is derived.\n\n3. For applying NOs to problems with irregular domains, the authors only give an example on unbounded domain. The case for arbitrary irregular domains is only briefly discussed. However, the authors claim the construction of NOs on arbitrary domains as the second contribution. More details on this part should be given.\n\nQUESTIONS:\n1. The upper bound in Theorem 3.1 depends on the covering number of the set $B$, which is an infinite dimensional space. Could the authors discuss how to estimate and bound the covering number?\n\n2. The bounds in this paper are only for excess risks, which also depends on the training loss. Could the author discuss how to bound the training loss?\n\n3. For the super-resolution error, why the upper bound does not depend on N_{grid,test}?\n\n4. In Section 6.1, how the upper bound is computed? The upper bound in this paper requires a tradeoff between \\gamma and N_{train}. How this tradeoff is made and how the covering number is computed?\n\nLIMITATIONS:\nLimitations are not disucssed in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
A theoretical analysis of neural operators (NOs) is presented, which provides further insight into the construction and performance of NOs.  The theoretical insights are validated by numerical experiments.

STRENGTHS:
A thorough theoretical analysis of NOs is presented, with significant improvements in the tightness of generalization bounds compared to prior work.  The implications of the theory and insights that may be drawn are discussed.  While some of these may seem obvious, it is important to base such insight on robust underlying theory, as presented.  The insights gained from the theory are validated by numerical experiments.

WEAKNESSES:
The summary of the overall model is not the most clear (Equation 1), which suggests the non-linear activation is applied first, rather than following the kernel and linear transforms.

Typo: "project" -> "projection"

Typo: sometimes $\mathcal{L}$ used to represent loss, sometimes $L$.

Typo: "Conbining" -> "Combining"

QUESTIONS:
While Figure 1 shows the generalization bounds are much tighter than alternative work, how can one have confidence the bounds are valid?

LIMITATIONS:
No special societal concerns.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

comments: SUMMARY:
The image super-resolution (SR) is a recurrently used task, nowadays, since the SR images can improve the accuracy of downstream tasks like object detection. Many proposals rely on heavy Deep Learning models or lightweight models based on efficiently designed architectures. This work studies neural operators based on examining the orthogonal base. The operations proposed in the manuscript, according to their theorems and demonstrations with the appropriate orthogonal bases and the grip points, reduce the time of convergence and make the network adapt faster to the  irregular domains.

STRENGTHS:
* The proposal section in the manuscript  is easy to read and follow.
* Theorem 3.2 and 3.3 are properly defined. For theorem 3.3, this will assist in re-planning new proposals for improving image SR accuracy.

WEAKNESSES:
Despite a short evaluation performed that shows a better performance of the proposal, I feel it is not sufficient to validate the generalization ability in the super-resolution task. To this end is necessary to validate with the standard evaluation metrics used in the SR domain. Personally, I feel that more explanation and motivation is needed for equation 3.

Minor error:
The machine learning articles, mostly, are organized Introduction, related works, the proposal, material, and experiments. It should be convenient to use this structure in the manuscript.

QUESTIONS:
No questions, but receive the weaknesses

LIMITATIONS:
There are no limitations addressed in the manuscript
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
In this paper, the authors propose to analyze neural operators from the orthogonal bases in the kernel operators, which helps to guide designing kernel operators and choosing grid points, analyzing generalization and super-resolution capabilities, and adapting neural operators to irregular domains.

STRENGTHS:
This is an interesting paper providing not specific models but new analysis perspectives and design principles on Neural Operators.

WEAKNESSES:
- I feel there are still gaps between the overall claims, the theoretical results in Section 3, and the implication and experiments in Sections 4&5. For example, the "NOs on Unbounded Domain" section is not what I would expect for "adapting neural operators to irregular domains"; the "Combining Multiple Bases" section seems more like analyzing kernel operators, instead of providing insight and principles in "designing kernel operators".
- The writing is not always good. For example, in the abstract, similar contents reappear three times.

QUESTIONS:
None.

LIMITATIONS:
This work is a bit beyond my capability. I think it is an interesting and important work, but I don't feel like it has reached its perfect state.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider a new perspective to neural operators, by examining the role of orthogonal bases. The kernel operators are constructed such that their eigenfunctions are predefined orthogonal bases, with the eingenvalues as trainable parameters. That is, a neural operator can be seen as a mapping from input coefficients to output coefficients of the orthogonal basis functions. The authors show several theoretical results using this new perspective, backed by empirical results:\n- Improved generalization bounds.\n- Super-resolution error bounds.\n-  Irregular domains - they show that neural operators can be extended to irregular domains using random Fourier features and polynomials on irregular domains.\n- Choosing other orthogonal bases. The authors show that Fourier bases can be combined with wavelet or polynomial bases.\n\nSTRENGTHS:\nThis paper can be impactful both because of the four concrete results they show (generalization bounds, super resolution bounds, irregular domains, and choice of orthogonal bases), but also because of the novel perspective of neural operators. In particular, the novel eigenvalue / orthogonal basis-based perspective of neural operators can be a useful view for studying neural operators in the future, both theoretically and empirically.\n\nWhile several other works have studied neural operators for irregular domains / topologies, the other results by the authors are much more novel. Not much prior research has studied generalization bounds, super resolution bounds, or combining bases before this work.\n\nWEAKNESSES:\n- Some of the results are fairly obvious or have only limited empirical value. For example, the super-resolution theorem/experiments, although it is novel to study super resolution, the main punchline is that super-resolution depends on the accuracy of the integration method, and the density of the low-resolution grid (i.e., the results in Figure 2), which is not so surprising.\n- Section 4 shows implications of the theory, motivating the experiments in the next section. However, some of these experiments have only tangential ties to the theory. For example, “Guiding the choice of Orthogonal Basis” is justified because Theorem 3.1 impacts their expressiveness. I would argue that we would decide to research the choice of orthogonal basis even if we didn’t see Theorem 3.1 first.\n- It would be nice if there was a bit more intuition for each of the proofs, in the main text.\n- Did not discuss limitations.\n- The authors could have released the code (anonymously during submission).\n\nQUESTIONS:\n- It would be good to discuss the relation of this work to other works on the theory of neural operators (beyond the discussion in Section 5). For example, how does your work relate to the JMLR paper https://arxiv.org/pdf/2108.08481.pdf, e.g. your super-resolution results compared to discretization invariance in Definition 4 and Theorem 8, and your perspective of neural operator compared to the set of neural operators used in their theory, defined in Section 9.1?\n- Wavelet bases are not necessarily discretization invariant. The definition used in the above JMLR paper says that we fix a finite set of weights for the architecture, and then we can take any discretization of the domain as input. The FNO accomplishes this by truncating to a fixed number of frequency modes $x$, which stays the same at higher resolutions. But it is not clear how Wavelet basis can satisfy this.\n- What are the limitations of your work (see below)?\n- Can you add a bit more intuition to all the theoretical results?\n\nLIMITATIONS:\nThe authors did not discuss the limitations of their work, and they answered “no” without explanation in the OpenReview paper checklist to that question. The NeurIPS call for papers states that authors can answer no to a checklist question, provided they give a good explanation. But I cannot think of any explanation why an author would be justified in not describing the limitations of their work; I think it is strictly better to do so.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper attempts to provide theoretical analyses of Neural Operators (NOs) mainly considering the following aspects: 1. The generalization bound of NOs; 2. The discretization error of NOs; 3. The \"super-resolution\" (trained on low-resolution grid then evaluate on the high-resolution grid) error of NOs on the uniform grids. Several numerical experiments are carried out to validate the theory. The theory results also motivate the authors to come up with improved designs on existing NOs, including bases/integration schemes that suit specific PDE better, and generalization to unbounded domains\n\nSTRENGTHS:\n* A tighter and more general generalization bound compared to prior works;\n* Advance in the understanding of discretization errors and super-resolution errors in NOs;\n* The improved design of NOs is validated by extensive numerical experiments.\n\n\nThe paper in general has analyzed several crucial properties of NOs, which is timely, and its practical implication I believe will benefit the scientific ML community. Specifically given that the concept of doing super-resolution with NOs is often vaguely studied in many relevant works.\n\nWEAKNESSES:\nThe overall presentation of the paper is fairly clear and easy to follow. However, some of the discussion in the experiment section is relatively vague, especially in section 6.3. As revealed by Theorem 3.3, the integration scheme along with basis selection has a crucial effect on the super-resolution error. The authors only briefly covered what basis and quadrature rule are used for the harmonic oscillator example, but then skim through the 3D DFT experiment, which makes it difficult to interpret the improvement in the results shown in Table 1. These details might be trivial for someone who is an expert in DFT, but they can help other audiences better understand the practical implication of the theorems.\n\nQUESTIONS:\n* One advantage of FNO is its superior computation efficiency on a uniform grid thanks to FFT. What is the rough estimate of the increase in the computation cost when using other integration schemes and bases?\n* Following theorem 3.3, will the increase in truncated modes harm the super-resolution performance?\n\nPossible typos:\n* line 197, page 5: , $e_{grid}(N_{grid}) = 1/N^2_{grid}$ -> $e_{grid}(N_{grid}) = 1/N_{grid}$\n* line 353, page 9: $u^2(x, t)$->$u(x, t)$\n\nLIMITATIONS:\nThe paper did not discuss any limitation. While this is somewhat understandable as a theory paper, the authors can discuss the limitation of their theory results in terms of the application scope and its assumption.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}