Infinite Neural Operators: Gaussian processes on functions
Authors: Daniel Augusto de Souza
Yuchen Zhu
Harry Jake Cunningham
Yuri Saporito
Diego Mesquita
Marc Peter Deisenroth
Venue: NeurIPS 2025 poster ()
Keywords: infinite-width neural networks
Gaussian process
neural operator
Primary area: probabilistic_methods

Abstract: A variety of infinitely wide neural architectures (e.g., dense NNs, CNNs, and transformers) induce Gaussian process (GP) priors over their outputs.
These relationships provide both an accurate characterization of the prior predictive distribution and enable the use of GP machinery to improve the uncertainty quantification of deep neural networks.
In this work, we extend this connection to neural operators (NOs), a class of models designed to learn mappings between function spaces.
Specifically, we show conditions for when arbitrary-depth NOs with Gaussian-distributed convolution kernels converge to function-valued GPs.
Based on this result, we show how to compute the covariance functions of these NO-GPs for two NO parametrizations, including the popular Fourier neural operator (FNO).
With this, we compute the posteriors of these GPs in regression scenarios, including PDE solution operators.
This work is an important step towards uncovering the inductive biases of current FNO architectures and opens a path to incorporate novel inductive biases for use in kernel-based operator learning methods.

Infinite Neural Operators: Gaussian processes on functions
Daniel Augusto de Souza' University College London
Yuchen Zhu University College London
Harry Jake Cunningham University College London
Yuri Saporito Fundação Getulio Vargas
Diego Mesquita Fundação Getulio Vargas
Marc Peter Deisenroth University College London
Abstract
A variety of infinitely wide neural architectures (e.g., dense NNs, CNNs, and trans- formers) induce Gaussian process (GP) priors their outputs. These relation- ships provide both an accurate characterization the prior predictive distribution and enable the use of GP machinery improve the uncertainty quantification of deep neural networks. In this work, we extend this connection to neural operators (NOs), class of models designed learn mappings between function spaces. Specifically, we show conditions for when arbitrary-depth NOs with Gaussian- distributed convolution kernels converge function-valuce GPs. Based on this result, we show how to compute the covariance functions of these NO-GPs for two NO parametrizations, including the popular Fourier neural operator (FNO). With this, we compute the posteriors of these GPs in regression scenarios, including PDE solution operators. This work an important step towards uncovering the inductive biases of current FNO architectures and opens a path to incorporate novel inductive biases for use in hernel-based operator learning methods.
Introduction
Neural Operators (NOs, Kovachki et al. 2023) are deep learning architectures designed to learn mappings between function spaces- with direct applications in many areas of science and engineering (Pathak et al. 2022 Li et al., 2024). NOs generalize conventional convolutional neural networks using kernel integral operators, which integrate the input function against learnable kernel each layer. Importantly, unlike CNNs. NOs can be trained with inputs of mixed, arbitrary resolutions and output predictions in discretizations of arbitrary granularity.
Despite their growing adoption, most works on NOs are primarily empirical, and most of the theoretical properties of NOs are still unexplored. In contrast, the convergence of Bayesian neural networks to Gaussian processes as their width goes to infinity has been amply studied (Neal 1995] Novak et al. 2019 Yang 2019). However, due the infinite dimensionality of function spaces, it is unclear whether GPs are limiting case for NOs and, if this is the case, how to characterize them.
In this work, we elucidate this question and present set of assumptions that guarantee the existence of the infinite limit of NOs as Gaussian elements in the space of operators. Additionally, we present how to derive the covariance function for infinite-width NOs in an analogous fashion to the covariance functions of infinitely wide, densely connected NNs. Finally, we characterize the infinite-widtt limit of Fourier neural operators (FNOs) and propose a novel Bayesian NO architecture based on Matérn distirrri integral kernels.
Corresponniing author: daniel. souza. 210ucl. ac. uk
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Our experiments reinforce our theoretical results, showcasing the agreement between increasingly wide NOs and our derived expressions for the infinite limit at initialization. Additionally, we compare the performance of these models in regression setting.
2
Background
This section provides brief background NOs (Section 2.1 along with basic notions of probability in Hilbert spaces (Section) 2.2 and Gaussian processes on functions.
2.1 Operator learning and neural operators
Kovachki et al. (202)) propose neural operators, family of parametrized operators. Recall that multilayer perceptrons transform vectors using successive layers of sums of linear transformations followed by element-wise non-linear activation functions. Analogously, Kovachki et al. (2023) define the building layers of neural operators (NOs) as sums of both point-wise wise linear operations and kernel integral operators, possibly followed by point- wise element- wise non non-linear activation functions.
Well-defined dot products in function spaces are central to coherently defining NOs. Thus, we will often assume functions lie in vector space in which their dot product is finite wrt some measure fax over their domain X. We define the Lebesgue space T,2 T,²(X, f,x;; TR²)) as the equivalence classes of functions in this vector space that agree almost everywhere in X' with respect to f.x- When clear from context, we will simply denote this vector space by (x). Whenever needed to evaluate functions point-wise, we further assume the function lies in an appropriate Reproducing Kernel Hilbert Space (RKHSs). In this work, we will be using both the Lebesgue space (x') and RKHSs, when adequate.
Point-wise operators. These operations are carried over from standard neural networks. Thus, given a function f: X - Fdd we consider dense layer-operations, with parameters W E Rxxd defined as (Wf) (z) X' + Rb = Wf(x), and element-wiss activations, with a given a R + R. to be defined as (z) == (f)( By composing and adding results between layers, we can build neural operators that basically act just on the output of the functions.
Kernel integral operator. The majority of interesting behaviors require expanding the receptive field and aggregate results from different function evaluations into one. The kernel integral operator Ak: : (X' + )²) - (y/ + R²",, parametrized by matrix-valued kernel function K y x + R together with measure y.x on X, is defined
A (v): y - R RR / K(y. a)) f((r) (a))
Under this operation, the function evaluated at a single evaluating point y linearly aggregates information on all evaluating points in the domain X' as modulated by the kernel K and the measure yx. Note, that this function may not converge for all values of y. but, for any kernel K I.2 (y x Ny X px: Px: R the operator Ak: L (2 px) - L L2 Myy well-defined.
Constructing neural operators. Given these building blocks, Kovachki et al. (2023) describe a neural operator as a three-part layered model. Firstly, sequence of point-wise operators applied to pre- process the function and change the dimension of its output. This is called the Lift layer. The second component is combination of point-wiss wise and kernel integral operators, in the so-called Neural Opera- tor layer. Finally, the Projection layer, sequence of point-wise operators is applied to the final result.
Specifically, neural operator layer combines matrix-valued kernel K and matrix W into
(x)) X + R Ak (m)) + Wf(x) f K(æ, 2) f(z) dzx((=) + Wf(x). Wf(m)
Setting the matrix-valuel kernel to zero recovers the lift and projection layers. Therefore, a neural operator with depth d and scalar output can be written succinetly as the composition:
[]f]()): X R (wi o H . H))
2.2 Probability in Hilbert spaces
Given a probability space (52, E, P), and Hilbert space H. random elements in H are functions x: Ω + H. such that the inner product w Ω + (y, x(w)) z is real-valued random variable, for

---

any y E H. As usual, we follow the standard notation of denoting the random elements/variables not as functions z but as elements z. Likewise, expectation is defined in terms of the random variables (y. ((w),, for each y E H. We say that the expectation of 2t, when it exists, is the element of H. denoted by R[z]], such that By,, x>] = (y. [[]])), for any y E H.
We denote the space of Hilbert-Schmidt (HS) operators mapping elements from a Hilbert space A to B by HS(AA; B). This space is the completion of the span of rank-one operators of the form a @ b: A + B. defined as (a @ (x) = (ar, a) A b for all a E A and E B. For L spaces, we have the isomorphism HS (L2 (x; L (D: = (2 x ); Vx under which (f @ g [h](-) = fx (.)=x)))(()) (x), where f E (*; g L (V; and gfT L2 x Rbx).
Moreover, the (cross-)covariane expectation of the tensor product E[z & y/. When this expectation exists, it is also HS operator denoted as Cov (z. y). From these definitions. we have that (22, Cov(z, y)[21]} cov( (22, y) (21, ()). for any 21, 2/ E H. In L spaces, we will make use of the isomorphism above and represent the covariance operator by its integration kernel. So, for any random elements E 1,2 (x: IR") and g g (2();R)) we introduce the function Clf, g): X x y + Rbxd such that Cov [b](() = fx C(f. g((,,))(()) x))(() (zx(
operator of two centered variables x and y is defined as the
In this work, we will make use of an extension of the strong law of large numbers to random elements:
Theorem 2.1 (Strong law of large numbers (Mourier 1956).. Let H be a separable Hilbert space and {0, jeN be countable sequence of identically distributed random elements. Consider the sample average yN = (1/N) Σ Fy. If. for any j. the expected norm 11I exists, then, the sequence {yN (uN)Nes NeN converges almost surely to the constant random element yoo E[xz]-
2.3 Operator valued kernels and Hilbert space valued Gaussian processes
Now, given a set X' and a separable Hilbert space H. an operator-valued valued kernel C: X x X' + HS(H; H) is any Hermitian positive-definite function, i.e., for all æ; E X, C(x, x²) C X' x H and {ai H C R. we have that E any (y, C(x,, C(æ,, (Bnl) > 0 (Kadri et al. 2016).x
((x',))² and, for any n > 0, {(x; }==1
Consider an operator-values valued kernel C: X' X HS(H; H) such that x + C(æ,, x)) is of trace-class. We say I: X x Ω + H is a centered Gaussian process with covariance function C if, for any n > and [(0;, yt) } C X x H. the vector ((y1, ()),,,)),.... ,),,..,, (Vn, (( ))) is a random element
distributed as an n-dimensional Gaussian with covariance
(12, f(22, )) (un, f(11, )) = (V2, C(x1, (vnl)-
We denote this by f ~ GO, C). For simplicity, we also define f(æ) == f(xx.))
3 Infinite-width neural operators as Gaussian processes
It is well known that infinite- width limits of various Bayesian neural networks are Gaussian processes (Neal 1995; Matthews et al.] 2018). We generalize this connection and show that infinite-width neural operators are ceaccion-valledd Gaussian processes.
Analogous to Novak et al. (019)) who place Gaussian priors on the convolution kernels of a CNN, the natural step towards function-valued GPs is to put independent GP priors on the component operators. Similarly, we require the weights and kernel for any component operator to be i.i.d. and with covariance shrinking with width. Theorem 3.1 states the main result of this work.
Theorem 3.1 (Infinitewwidd neural operators are Gaussian processes). Let X C Rdd be a measurable space and let (X;Rd) C RR;)) be an RKHS for any J N+. Then, for a given depth D E N+, consider a vector of positive integers J = [Jo, JI....,JJD. JD-1; T] E ND+11 and a J- indexed neural operators DPD) of depth D:
2\D) = (D)
(D)
ZJ- E (X' + Rd) + (X + R),
(5)
where,
-ー ()) L L' (x + H((;
(6)
3

---

H = (ARi) + E + H(()))))
(7)
w 1
(8)
K(9 E Hl( x X; Re)
and a: + R such that o f) E (x) for any E (2)
When all parameters are independently distributed a priori according to
w ~ (0, of Jt-I(), and ( ~ ~GP(O, Qkty) (Je-
for E d].
(10)
then, the iterated limit lim valued GP 2(D) GP(O, where Coo [fo[f. g]. is available in closed-form.
2 in the sense of Definition B.1 is equal to a function- lim
Jn-1-10
J. J10000
where we present the explicit formula for Coc. which depends on the conditional covariance function between layers. Before delving into these details, we introduce the compositionality property of covariance functions in Section 3.1 This property enables the closed- formm computation of the conditional covariances, thereby fully characterizing the limiting covariance function Co..
An outline of the proof is presented in Section 3.2
3.1 Operator-valued covariance functions
We realize the following crucial points: i) The covariance function only depends on the inner product of the values of the imput functions, and ii) Using the strong law of large numbers, the covariance of the composition of operators can be described by composing its covariance functions. This is presented in the next lemma, with proof postponed to Appendix [B.2]
Lemma 3.2 (Compositionality of covariance functions). Let B₁: L' (X) - (2; be
random operator and B2: : L² (X;; R²) + (X)) be centered the following assumptions hold:
finctton function-valued Gaussian process. If
For all f E (x(;)R and æ E X, each component of B, (x) E R is independent and identically distributed such that the covariance function Cp, [f. g. = CB, [f.
The covariance function of B2 can be expressed, for all f,g E 1,2 (2; R2) as Cn2 [f.g]
CB, [] and the function h + CB, [h] is continuous map from I.² x X) to itself.
Then, B2 B1 converges in distribution to a function-valued Gaussian process
oc, and
CB [n, f2] CBy [CB, f2])
(11)
For each operator discussed in Section 2.1 below we state the conditions under which they are function-valued Gaussian processes, and derive their covariance functions.
Spint-wise linear operator. Given vector w E Id and a function f: X + IRd, then, define the linear operator (w f): X' + R such that (w. (=)) = Σø= (a)p If the entries of the weight vector follow an i.i.d. Gaussian distribution, i.e. w N of then, this a centered Gaussian process taking values from (2 R2) R2) to (2) with covariance function:
Cuo [f,, (2 222 R ff (x2) fi (x1), such that,
(12)
cov((12, w TT) (bi, w (f)))) // (2 h (x1) G. [fi, (2) dux (x) (x2)
(13)
Note that e. only depends the function f. X x X - IR, we abuse notation and write cw [f. fz] = cw f.) Moreover, this function is homogeneous: ac [f] fil Cw fi). for a > 0.
Kernel integral operator. As defined in Section 2.1, given a function k: y X + Rd and an input function f: X' + Rd, we consider the linear operator Akr [f]: 3 + R. If follows an i.i.d. GP such that k E (22 x X) ~ GP, Gk), then we have that Akr a centered function-vall valued GP mapping from 2 (x(:RR)) to L (2') with covariance function, denoted here by:
CArT [f, (y), y2) = 1/ ck æT, U2, x2) f.J (x2) fg (xx)) dax (x; (x2)
(14)

---

Acx (11 y2).
(15)
Note CA&T also only depends the inner product of f2 and fi and is homogeneous. Thus, denote CA&T [f, [2] = CALT (1]
Point-wise element-wise activation. Given non-linedd function a: R - R, abuse the notation and define the non-linear operator o|:]: (2) + L (x) as
a [ff] = off(x))
(16)
Note that some restrictions on a need to be placed for this to be a well-defined operator in (2)) As example of such condition, for their theoretical analysis, Kovachki (oovachi, al. (2233 restricts activations
to measurable linearly bounded functions, noting that the popular ReL.U ELU, tanh, and sigmoid activations satisfy this condition. In Appendix B.I we provide a proof that this condition is sufficient for finite measure domains.
Consider centered Gaussian operator B: (x) + H(2) with covariance function eg such that H((x) C (x)) an RKHS with reproducing kernel ky. When of.] is a well-defined operator, the operator (ar B) is random operator in (x)) + L2 with covariance function:
C)ooB) (m1: x2) = cov((o (() (a (x2))
(17)
Now, since B(f,] and B[fg] are Gaussian processes with outputs in RKHS H. we can consider the following bivariate Gaussian r.v. blr = [=1,) (x22](x2)]
b .V [
x
Tc [,, f (m1, ca [2, (2) mi)
cB Ca, fo (21, ca [22, (22,
(18)
(22, 222
This random variable is well-defined due to the reproducing property, B(f ())
Thus, we can continue to conclude
B) (11, x2)
(br,) N N fa) 0, LT) (b
(19)
o( WE 0, T)dE
(20)
R²
c. [en fi, (z)] x2),
(21)
where L is square-root of the covariance matrix of bs) and 4; is the i-th row of this matrix.
The expected value C as a function of I and l2 in Eq. (20) is known as the dual kernel of The
dual kernels for many activation functions have closed-form solutions (e.g.. sigmoid (Willimss) 1996
Eq. 10) and ReLU (Cho and Saul 2009, Eq. 1)) or can be efficiently approximated (Han et al. 2022). Any of these solutions can be directly used in our context by computing the covariacce matrix of b(e,,5) and applying the rows of its square-root as arguments.
In conclusion, we construct covariance function Co: H(X' X) + L2 X) such that, for given covariance function cn: 12 x X) + H(X x X)):
(hzh1, Colbicc/cgti [cglf, fell) cov((hi, (a B)ffi) (b2, (a ()()))),
(22)
for all f1, f2, h1, h2 E (2')
3.2 Outline of the proof for Theorem 3.1
We now describe sketch for the proof, refer the readers to Appendix B.3 for the complete proof. Step We start by showing that, under the conditions of Theorem 3.1 each linear layer in a neural operator layer is function-valued Gaussian process when conditioned on its inputs. Moreover, as discussed Section 3.1 the conditional covariance function of each node on each layer only depends on the empirical covariance function of its inputs cff, g)(=', x) = (1/J) Σ (x) (x')) We denote this dependency by writing the conditional covariance function as c [ x').
Step 2. Due to the chosen prior distribution of each layer, we know that each node Hz|- E R is i.i.d. and, therefore, we apply Lemma 3.2 to conclude that, as Je-1 so, the covariance ( ( [g] H [f]/ Jc-1] converges almost surely to (ff(e-1) [CH, [f, gl].
5

---

Step 3. Combining both steps, we show, by induction on up until & = d, that, as J + oc, the covariance function of Zy(f] is simply the composition of all the previous covariances as denoted in Step 1. So, we have that the covariance function of 7oc is:
c () f. gl el c clddd")(((--1d--22 .... [f,, ....
(23)
Finally, denote cld) Coo.
4 Parametrizations and computations
To apply the results of Theorem 3.1 we must specify a covariance function for the integral kernel operators Ag. This choice corresponds to selecting a particular neural operator parameterization, following the approach of Kovachki et al. (2023).
In this section, we derive the operator-valued covariance functions for Ak under two parametrizations of the integral operator. The first is based on the band-limited Fourier Neural Operator (Section [.1 the second models the kernel as a non-stationary process, with prior distribution derived from the classical Matérn family of covariance functions (Section 4.2).
A common assumption for both cases is that the input domain is compact. This ensures that samples of the kernel components k, reside in L,2 space. By further choosing the domain to be the dz- dimensional flat torus T Rd= / 2d= we are able to exploit Fourier analysis tools. In partieular, by assuming that the input functions are band-limited enables tractable computations through the connection of Fourier series with discrete Fourier transforms for evaluations in regular grids.
4.1 Fourier neural operator
Out of the parametrizations proposed by Kovachki et al. (2023), the Fourier neural operator the most popular due to its computational benefits. By imposing three assumptions into the convolutions kernel - periodicity, shift-invariance, and band-limitedness we can use the convolution theorem to compute the integrals using sums up to the chosen band-limit of the kernel in the Fourier space.
Concretely, assuming periodicity equivalent to choosing the domain be some d.-ddimensionl flat torus X' = T and shift-invariance means kernels satisfy kj (w, æ) = kj (w x). where we abuse notation and represent the kernel as a univariate function of the same name ky T - Rd. Under these conditions, any kj admits Fourier series representation:
k x)
Σ (S x)), BE24r
(24)
where FS, is the (s1,..., )4,) coefficient of the Fourier series and v)a = expl-i-sTa]] with
i V-1 being the imaginary unit. Moreover, to have a band-limited kernel implies that only finitely many Fourier coefficients are non-zero, i.e. there is some B3 E N, 1 < j S d,, such that FS. [k] = |s5| > Bj, for all 1 S 5 dz.
Under these conditions, despite all input functions f being represented with a (potentially infinite) Fourier series, by the convolution theorem, the NO layer 11, [f] is band-limited and its Fourier series coefficients can be computed directly from the product of Fourier coefficients of the kerell function k and the input function Thus, we have that:
FS, H [f] = [k,] [F.[[F] wh!' [S.[f].
(25)
when is a aed-valued GP, the kernel integral operator Ak is a macciio-vvluued Gaussian process with covariance function of Ag in terms of the covariance function of k, Ck:
Parameterization of an FNO. Following Section 3.1
CANT [fn f2] =2) Ac, [[] Z2).
(26)
The most popular choice proposed by Kovachki et al. (203) is to directly parametrize the Fourier coefficients of the kernel. Thus, we let these 2B + Fourier coefficients follow i.i.d. centered complex Gaussian distributions with variance of (Appendix A. obtaining the covariance function C₁:
k(x xx)= Σ
FS. [k v x) GP Ck),
(27)
.E ( B..... B)

---

Ck((z x2, x'))
LI E v--laz- 2) - =²)),
(28)
*E(-B,..., B)
where a hyperparameter of the model controlling the band-limit of the integral kernel. This allows us to derive finite-sum representation of the covariance of Ak parameterized by o
CAT [fi (2,, (27))
Σ
x²).
(29)
aE{ oE(-B...,B)
4.2 Toroidal Matérn operator
In this section, we propose model in which the kernel does not admit shift-invariant decomposi- tion. Another popular decomposition used in the Gaussian process literature the tensor-product factorization, where the covariance function of GP factorizes over the input dimension. That is, f: 2 R (O, c), where c(a, b) IT d. c (a,, by): although the covariance factorizes over the input dimensions, in general, samples from f do not.
Our proposal will make use of the ubiquitous Matérn family of covariance functions, which characterized by the smoothness parameter Following Borovitskiy et al. 2020), we define the Matérn covariance functions in the d, dy-dimensional flat torus T T Tas
de
c(x; x'; v, 0) = (2T) d, Σ
(n(x) (x'') (x')
(30)
nEZda
where l is the lengthscale hyper-yaramene parameter and the spectral density è defined as: ==
exp - -
6(A M 2)
(31)
( +
otherwise.
In general, this kernel is not tensor-product factorized, but for the special case of v = co, the squared exponential covariance function, the factorization holds (Appendix A.2.1) Thus, in general, enforce the censor-produc factorization:
d2 II
de
c(x, a'c, (1,
c(a,,)fi,,))) =2mm) d. Σ (z) (z)) j=1
II c(n 4j (32)
nEZda
where l is the automatic relevance determination (ARD) lengthscale hyper-parameter.
Parameterization of a toroidal Matérn operator. So, we consider a convolution kernel k: X' x X' + Rd defined as the product of Matérn covariance functions:
C Cz x, 2 x') = c(2, z'; (x) c(æ, My, La) la-
x';
(33)
where c(.,.v,, ;2 e) X X' + R is the Matérn covariance functions with smoothness parameter and length-scale
Again, following Section 3.1
we express the covariance of the operator as:
CANT (2 #2) Ac, [21,
(34)
Thus, by using the identity Eq.
33 we can derive:
de [fc] co(33 Un, las). (35)
CAur (f,, = c(z, uus Rx) (2x)da
Σ
[f]
FS_
II
nE244
J=1
Experimental validation
In this section, we empirically show i) the agreement between finite width neural operators with increasing width and their corresponding infinite-width neural operator, and ii) evaluate our model against FNO in regression task.

---

= 1 (TVD= 0.483)
= (TVD
/ = 100 (TVD 0.008)
/ 1000 (.005
0.6
1
^
p(2if((0)
0.4
0.4
0.2
0.2
0.2
0
0.0
0.0
0.0
-2.5
0.0
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
Z[]((0)
Zif(())
Figure 1: A density estimation of the O empirical distribution of the output of increasing channel dimension compared the t infinite width distribution. On top of each plot we show the total variation distance of the empirical distribution against the infinite width distribution.
Section 5.1 explores the distribution of untrained randomly initialized Fourier neural operators of varying width and the distribution the infinite-width FNO (oo-FNO). As expected from the theoretical results, these distributions should eventually match the hidden dimension increases.
Section 5.2 considers two tasks: a synthetic regression example, where the task is to predict the output of non-linear operator, and the task of predicting the final evolution of Burgers' equation given the initial state. This situation not covered in our theory, since only applies to the distribution of the neural operators at initialization. but experiments show the behavior of the posteriors of infinite-width neural operators against Adam trained finite- wiith neural operators of increasing width.
Throughout this section, our stating point is single hidden-layer neural operator Z[[]: + R (w) ReLU (Ak + Wi))((]. More details for each experiment can be found in Appendix C All experiments were implemented in Python, mainly based on the GPyTorch (Gardner al. 2018) library, and run desktop computer using Titan RTX. Code is avaliable
httttps://gittt t/tp:bbb/gttu...... www.sppetraaaaanni......maaaaaae www.dddanniiniiiieeeeeeeeeeeeeee mew.aannnn-eeaaloperrrtto...
5.1 Empirical demonstration results
In this experiment, we demonstrate that our analytical computation of the variance for a neural operator layer H agrees with empirical estimates, and we validate Theorem 3.1 by showing that the output of neural operator Z converges
Gaussian distribution the number of hidden channels J increases.
Throughout all experiments, the input function f: T + R has band-limit = 3, with its output values ((x) sampled from uniform distribution 1). Both operators evaluated we analyze the empirical distributions (0) and 2 (0), respectively.
so
Following Section 4.1
parametrize the integral kernel operators using band-limited functions. The band-limit of the kernel is set equal to that of the input f, and the kernel coefficients are drawn from Gaussian distribution with unit variance scaled by the inverse of the number of hidden channels.
1.340
Theoretical value
1.316
Estimated (mean std +
Variaace
1.292
1.268
As shown in Fig. 2 the empirical estimate of the variance converges to the theoretical value as the number of Monte Carlo samples increases, supporting the correctness of our variance computation. Furthermore, Fig. shows that, as the number of hidden channels grows, the total variation distance (TVD) between the empirical distribution and Gaussian distribution approaches zero, thereby further verifying the validity of Theorem 3.1
1.244
1.220
10'
10'
10
10a
10'
# Samples
Figure 2: Plot of the MC estimate for the variance of H (f] against our analyti- cal computation (Sec. 3.I.
5.2 Regression tasks
In this task, we're given pairs of ID functions [fe, gi J evaluated in grid with m 2B,,. + points. We consider FNOs increasing width E N as well as oc-FNOs, both with increasing kernel band-limite B (1,5, 20). These models will be trained two datasets: (a) A operator

---

10
10
EXD
FXOUB=5)
Test L error
10
Test 12 arrr
PND 30
=3ND10
10
10-2
10
(50 201
109
10
10a
100
10
10a
J
(a) Synthetic data, FN-FNO with B = (--) 20 (..) overlap.
(b) ID Burgers' equation 0.002). 0o- FNO with B (--) and (--) overlap.
Figure 3: Results for the regression experiments. Mean and std. of test L loss as a function of width for different Sand-limity B.
generated by a randomly-inittalized ground truth FNO Zirue with band-limit B = 5 and width / = 1. with iniformily-distribine
outputs d(-1,,1 and band-limit Bm = 5. (b) ID Burgers' equation dataset from Takamoto et al. (2022) with = 0.002. The task is predict the end state (t 2) given the initial condition (t 0). Due memory constraints, we subsample the total dataset data to 100 functions and grid size of m = 103.
We sample 100 input functions f;: T
The hyperparameters of the FN-FNO estimated using L-BFGS, while the parameters of the FNOs are optimized with Adam using a step size of 0.001. We evaluate all models using 5-fold cross- validation and report the average and standard deviation of the empirical I,2 norm of the prediction error. For ON-FNOs, we use the posterior mean as the prediction.
In general, we do not expect close agreement between the predictive performance of oo-FNOs and finite-width FNOs, as the former corresponds to Bayesian estimate while the latter are trained by minimizing an empirical risk, nonetheless, observed in Figs. 3a and 3b there is consistency between the gap hyperparameters in the same model class.
In the synthetic case, we know the band-limits of the ground truth operator, Fig. 3a shows that the models are only able accurately predict the output when their band-limits exceed that of the ground truth.
Related works
Infinite limits of stochastic NNs. The study of infinite-width Bayesian neural networks began with
the seminal work Neal (1995) and was later extended to deep architectures (Lee et al.) 2018 Yang, 2019 Matthews et al. 2018). Our analysis builds on the ideas developed by Matthews et al. (2018). From the outset, these sninite-width models were considered "disappointing" (Neal 1995),
View reinforced by findings that neither the Bayesian limit nor the neural tangent kernel limit learns features from data (Aitchison) 2020). However, recent work shows these models still reflect the different inductive biases of their finite-width counterparts (Novak et al. (201)). and that alternative initialization distributions enable feature learning in the infinite- width setting YYngg and Hu 2021).
Bayesian neural operators. Several works have investigated approximate Bayesian uncertainty quantification finite-width neural operators using function-valued Gaussian processes, Magnani al. (2022 2024) both employ last-layer Laplace approximations to construct GP approximations of the posterior distribution. In addition, Magnani et al. (2022) considers the case where the kernel K of the integral operator Ak follows Matérn GP prior. However, their analysis is restricted the
finite-widhe wiite- regime on compact subsets of Euclidean space, whereas our work focuses on the flat torus.
Kernel methods for operator learning. Batlle et al. (024) propose the use of kernel methods for operator learning, leveraging operator-value veluue kernels and representer theorem in their correspond- ing RKHS. Their results are promising and highlight the potential of kernel-based approaches in this

---

domain. Our contribution introduces an additional way to construct operator-valued kernels based on neural operators, enabling new kernel-based models for operator learning.
7 Discussion
In this work, we formalized the concept of infinite-width Bayesian neural operators, established their existence (Theorem (.I). and described how to compute their associated covariance functions (Section We validated these results empirically (Section 5.11 and further assessed the performance of these models regression setting (Section 5.2).
Our contributions lay a foundation for future investigations, particularly in bridging the gap between SGD-trained finite-width neural operators and their infinite-width counterparts. Addressing this challenge will require extending the neural tangent kernel framework (Jacot et al. [018 Lee et al. 2019) settings involving Hilbert space-valued functions. Morcover, while we focused on the ubiquitous FNO architecture, deriving covariance functions for other architectures, such as the graph neural operator (Kovachki et al. 2023), remains an open direction.
Limitations, Our current implementation for computing the required kernel quantities scales with cubically in both the evaluation grid size and the number of training functions. We anticipate that future work can improve computational efficiency by leveraging advances from the Gaussian process literature improve scalability and efficiency (Borovitskiy al. 2020 Gilboa et al. 2015).
Acknowledgments and Disclosure of Funding
YZ acknowledges support by the Engineering and Physical Sciences Research Council with grant number EP/S021566// YS was supported by Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro (FAPERJ) through the Jovem Cientista do Nosso Estado Program (E- 26/201.375/2222 (272760) and by Conselho Nacional de Desenvolvimento Científico Tecnológico (CNPq) through the Productivity in Research Scholarship (306695/2021-9 305159/2025-9,, DM was supported by the Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro (FAPERJ) (SEI-260003/000000002333 and the Conselho Nacional de Desenvolvimento Científico Tecnológico (CNPq) (404336/2023-0, 305692/2025-9).
References
Laurence Aitchison. Why bigger not always better: on finite and infinite neural networks. In Proceedings the 37th International Conference on Machine Learning (ICML), 2020. URL httph:://pooceediiiiii mlr
ppesssv119/ attp1/////cchhioo200 html
Pau Batlle, Matthicu Darcy, Bamdad Hosseini, and Houman Owhadi. Kernel methods are competitive for operator learning. Journal of Computational Physics, 496, 2024. URL /RRlittts:///oi.orga. http://o.....//10.......0000 jep.
2023. 2023. 112549
Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, and Marc Deisenroth. Matérn Gaussian processes on Riemannian manifolds. In Advunces in Neural Information Processsnn Syxtems (NewrIPS), 2020. URL
https://papers. nips. whtppprw7000nnaa22266eee00000e00 html
Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. In Advances in Neural Information Processing Systems (NewrIPS), 2009. URL
Jacob R. Gardner, Geoff Pleiss, David Bindel, Kilian Weinberger, and Andrew Gordon Wilson. GPyTorch:
Blackbox matrix-matrix Gaussian process inference with gpu acceleration. In Advances in Neural Informa- tion Processing Syxtems (NeurlPS), 2018. httpps//////ppppet www.nii..pppaaaaam htc//pppr_fflleeeeeeeeeeeeeeeeee fwwesf paper///pp Abst//eataabttttt......m html
Elad Gilboa, Yunus Saatçi, and John P. Cunningham. Scaling multidimensional inference for structured Gaussian processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37, 2015. URL http://dx, doi, org/ 1109/TPAMIII. 2013. 192
10

---

Insu Han, Amir Zandieh, Jaehoon Lee. Roman Novak, Lechao Xiao, and Amin Karbasi. Fast neu-
ral kernel embeddings for general activations. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2022. URL https://proceeeeingg neurips http:ipppccpppppeeeeeeee wwtppppr..ille//ppeeeepeeeeeeee htwpperrfflleeeppee/22ee/ 20222////222hhhh
html
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kemel: Convergence and generalization in neural networks, In Advances in Neural Information Processing
Systems (NeurIPS), 2018. URL. https: /appa/////ccccce..... wewprddnnnnnnn.n...eeeeee ceuripa.cc/ppiee 2t////211////a///
Hachem Kadri, Emmanuel Duftos, Philippe Preux, Stéphane Canu, Alain Rakotomamonjy, and Julien Audiffren. Operator-valued kernels for learning from functional response data. Journal of Machine Learning Research
(JMLR), 17, 2016. URL http::ttp::/altt org/papprssvvl7/11-111 htmm
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Arizradenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to PDEs. Journal of Machine Learning Rexearch (JMLR), 24, 2023. URL http:///m/ org/ prg/papers/v24/ 21-1524. html
Jaehoon Lee, Jascha Sohl-Dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks as Gaussian processes. In International Conference on Learning Representations (ICLR), 2018. MRLLtttps://opeerev/// wwt.:oottttoottttteee... B1EAAMMOZZ
Jaehoon Lee. Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. In Advances in Neural Information Processing Systems (NeurIPS), 2019. URL URL aattpe://proceedingg Abstract html
Zongyi Li, Hongkai Zheng. Nikola Kovachki, David Jin, Haoxuan Chen, Burigede I Liu, Kamyar Avizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. ACM IMS Journal of Data Science, 1(3), 2024. httph::/to//. org/ 10.
Emilia Magnani, Nicholas Krämer, Runa Eschenhagen, Lorenzo Rosasco, and Philipp Hennig. Approximate
Bayesian neural operators: Uncertainty quantification for parametric PDEs. 2022. URL httth://aaxxxii org/ aba 22088.0066
Emilia Magnani, Marvin Pförtner, Tobias Weher, and Philipp Hennig. Lincarization turns neural operators into function-valued Gaussian processes. 2024. URL httttt:t///trtttt crg. hargg/abs24065 011400600777
Alexander G. de G. Matthews, Mark Rowland, Jiri Hron, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. 2018. URL htttpe//aarr...rrr///////11111.. 11271
Edith Mourier. L-random elements and I'random elements in Banach spaces. In Contributions to Probability Theory, pages 231-242. University of California Press, December 1956. htLLhttts///dd..rrg//1 1525/ 9780620350670-017
Radford M. Neal. Bayesian Learning for Neural Nenworks. PhD thesis, University of Toronto, 1995.
Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang. Jiri Hron, Daniel A. Abolatia, Jeffrey Pennington, and Jascha Sohl- Dickkstenn... Bayesian deep convolutional networks with many channels are Gaussian processes. In International Conference on Learning Representations (ICLR), 2019. URL ///pennevvew.net///eee www.//orooiddtt/ieee... g30
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: global data-driven high-resolution weather model using adaptive fourier neural operators. 2022. URL pattts://arrzss www.oogg///a/ao..oooooo...o....a abtp///a.........//// 2202 11214
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Dan MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. PDEBench: An extensive benchmark for scientific machine learning. In Advances in Neural Information Processing Syxtems (NeurIPS), 2022. URL htttts: neurips. neurips.ccppaper.. files/ Benchmarks
html
Christopher Williams. Computing with infinite networks. In Advances in Neural Information Pro- cessing Systems (NeuriPS), 1996. URL https://proceeddii neurips. http://wwwwnaaooooo.ooo.cocopppe httpppps..../pppe////////s///// Sbb3eeefooo4444cccccoooo66665553 html

---

Greg Yang. Wide feedforward recurrent neural networks of any architecture are Gaussian processes. In Advances in Neural Information Processing Systems (NeurIPSS, 2019. URL httpp: neurips https///oooooooooooooooooaaaaaaa
Greg Yang and Edward J. Hu. Tensor programs IV: Feature learning in infinite-width neural networks. In
Proceedings of 38th International Conference on Machine Learning (ICML), 2021. URL https
//proceedinge. http:/ewnnnneeeeeeeeeeeeeeeeeeee wttppe///119///mg21........
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Our contributions consist of i) a major theorem (Theorem 3.1) characterizing the infinite-width limit of Neural Operators (NOs), and ii) closed-form equations (Sec- tion for computing the kernel induced by NO architectures, which allow for practical implementations.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have included the limitations of our work in the Discussion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "L_imitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification: The main assumptions are placed inside the theorems, the main text includes a proof sketch of our main result, and the full proof for it and related lemmas provided in the supplemental material.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The main text include a summary of how the dataset was generated and how to compute the necessary quantities for implementation. The supplementary material include further details on hyper-parameter and initializations alongside the full source code for the experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
14

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will provide code as part of the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
guwdes/ httc://uuulllgggaudddo///oebaaaa
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe this in the experimental section and include more detail in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We include mean and standard error for experiment 5.1, representing the different sample used for estimation and experiment 5.2 contain mean and standard deviation of representing each of the 5-fold splits.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have included the specs of the machine used for experiments in the main text and in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htteeestttt:////t htttpp/:/boarrrss.c/////////tttt
Answer: [Yes]
Justification: This work deals with theoretical results and synthetic data, thus, we believe that it conforms the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
16

---

This work deals with theoretical results and synthetic data, we believe there is no direct path for direct negative application of this work.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The models used in this work do not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The authors of frameworks used for implementation are credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
17

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, datasets.paperrsttt wwppppsppwaaaae thcode com datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19

---

Infinite Neural Operators:
Gaussian processes on functions
(Supplemental Materials)
A Covariance Function Computation
In this section, we will work out in detail the computations of Section 4 first for the Fourier neural operator (FNO) case and later for the toroidal Matérn operator.
A.1 Fourier neural operator
Under the direct parametrization of the integral kernel operator, the coefficients of the kernel's Fourier series (FS) are parametrized and randomly sampled at initialization. Therefore, our first step is to derive what the Gaussian process distribution of band-limited function with i.i.d. Gaussian FS coefficients is.
Fourier series. Given a function on the de-dimensional torus f(-): T + Rd f= (fl.... fa), it can be represented terms of Fourier series:
fx(x)
Σ #6
FSa sz)(z),
(36)
for p {I,...,d}, d}, where,
FS, (2xx) do
(t) (t)
(37)
[-x,0]0=
(x) expl-i- TT0), ex])_
(38)
and V-I is the imaginary unit.
Note that, as fp is real-valued function, also have that FS FSSolffp, where & is the
complex conjugate.
Gaussian distributed band-limited functions. Consider the sequence f: [-B..... Bld + defined as:
RRfo ~ N((0.
3 =0 0, 3/. O(0,02/22)
(39)
Rf. ~ 00, Rf - Rf..
(40)
3
(41)
where Rz and 32 are the real and imaginary parts of the complex number z, respectively, and all random variables are independent each other, except the conjugate duals fa and f-.. For s + 0, the equations above can also expressed as:
R f
0
o²
0
0
S
000
0
2
0
2
R
2
o2
0
(42)
3 f
0
-
0
o²
With this in mind, the expectation of the product of two elements is:
Blfa fal] Ef E.SA.ff
HE 3f.1 +
-ERf.fff
(43)

---

(44)
f..
if = 8. if - otherwise:
(45)
a²
if s' 0 and = 0.
0²/2 02/2
if s' 8,
(46)
0²/2 02/2 if s' otherwise;
o²
if -8.
(47)
0
otherwise.
Thus, we can define the Gaussian process f: T + through a Fourier series representation:
f(a:) Σ (-
(48)
=E(-B...,BR*
We compute the covariance function of f as:
(x) B[f(x) f(x')
(49)
E
Σ
So (=)
E
for (=))
(50)
#E ( By.
BEI-BB....BI'
Σ E[Ss 1/ (xx) Sar 1/ (xx')
(51)
8 '( (BB....,Bdd
Σ
Elss fw (x))) 3- (xx)
(52)
8.8' a. ((-B....,.)) B)dz
2² Σ
V ( (x)'
(53)
3E (- B B,..., B)ds
2
f- x¹).
(54)
SE{ sE(-B...,BB"
A.1.1
Covariance after convolution CART
Let us place a centered Gaussian distribution on the Fourier series of the band-limited convolution kernel k: X x X R2d so that:
Cn((2 (z' - x²)) = o Σ
w-a((z xx) - (2' - x²))
(55)
#E{ ( B)d
I Σ
v (=(2-x))aa x¹)
(56)
.E( B,.., B)da
- Σ
w- 21) x')
(57)
eE(-B..... B)
So, let us consider the quantity CA2 [f], fb] for arbitrary functions fi and fg:
// 1 ((ad'))(((
Ao (1f =
(z' - x')) dxda.
(58)
x
| fl (x') 02 Σ v-o(z 2z) o()){(z)) dxdx'
(59)
OE(-B....

---

0²
E
vz - x²)
x' x'){ (x (x) dxdæ'
(60)
(E(-B....,)) Bide
0² Σ
vz z') (2T 2dd [1
(61)
051 BB..B...BB)
= (2 2do
(2 2¹) FS- [f]
(62)
sE B1...
A.2 Toroidal Matérn operator
Definition A.1 (Matérn family of kernels on a closed manifold), The Matérn family of kernels c with lengthscale l in a d-dimensional closed manifold M are described as:
b(a,b;,,e)
Ö(Aii 1) (a) (b),
(63)
exp
A
if = 0o.
8()
(64)
+ + otherwise.
where, A and du are the k-th eigenvalues and eigenfunctions, respectively, of the Laplace- Beltrami operator of the manifold M.
For a 1-dimensional flat torus, an orthonormal eigensystem for the Laplace-Beltrami Beltrami operator is:
1/V2T cos VAL)//TT k = 2n, sin sin(vAr.)/ if k = 2n + 1.
1,
A = 1
du (x)
(65)
Additionally, for de-dimensional flat torus, an orthonormal eigensystem for the Laplace-Belin Belltram operator is given by the sum and product of the 1-dimensional eigensystem such that, given an index de k = ]. we have that A Σド=
Ax; and dh (ax) [ dhy (zz)).
Expression of 1 1-dimensional toroidal kernel using complex exponentials. For convenience, we will rewrite the series expansion of this kernel to use exponentials of complex numbers.
Start by noting that:
22 (a) (a))2n(b)))))))))))))))))))))( = cos(na) cos(nb),
22n+1 (2)+1(4))2n+11)) = π
sin(na) sin(nb),
(66)
Therefore,
cos(na) cos(ni)) 1
22n (2) (b) + 62n41 (12m+((a)22m.) (a)d2n+1 (b)
sin(na) sin(nb)
(67)
cos(nía - 6))
(68)
(explin(a b)] + expl-in - b)])
(69)
2n
(explina]expl( + expl_inna exp (nb))
(70)
2 (vh ())_
(71)
Now, we can rewrite the Matérn kernel expression as:
b(a,b; Σ d(Ai (x(((a)))))))))))))))))))))))))) (x)))(()))))))))))) (b)
(72)
1=1
(() +
Σ 8(hani U. () (a))2n (b) + ob (a)dcn+ (b))
(73)
2x
3

---

1
1
+
6n² M, (V-,(()))))))
(b) +
(a)a//- (b))
(74)
2x
2nT
n=1
2(0:2,0)+
+
Σ
[n= + Σ 2x
8( cnn v, (a)a
2x
n=1
(75)
1 6(0:2,,)+ +
Σ
000
v, +
Σ
c(n 2 v, ()
2x
2x
2x
n=1
(76)
1 6(0)6(aa)0-00: +
Σ
c(n": v, (0))((0000)0000
(77)
2%
2π
nEZ](U)
1 2x
Σ DEZ
(78)
Product kernel for T In order to have one lengthscale per dimension, we will make tensor product kernel where the kernel of a dy-dimensional torus T is the product of the 1-d torodaal kernel for each dimension:
dz II 5=1
c(a. b. k,)
(79)
II
(80)
2T
-2T) d
c(""; ()
(81)
j=1 nEZ
= (2m) de Σ II
c( v, (ay)) (by)
(82)
nE2d> ==1
d,
= =2m) de
(a) (b)
II
à a(nf u, eg)
(83)
REZ
j=1
A.2.1 == 0o lets Matérn kernel be a product kernel Notice that when v= = oo and E = L, we have
de II j=1
c(a, by e) = (2T) de Σ
(n(a)
exp -nn3/2)
(84)
nEZda
(2 de
Σ
(n(a)) (b) exp
e Bng12
(85)
n nE2ds
C2 de
=(2m de
(n(a)vin exp
ng
(86)
2
ngzdz
d=
=(2TT dh
(n)avin (b)c
n3 v,
(87)
TI
With the proper rearrangement, we can see that this fits the definition of a Matérn kernel in the T as the eigenvalues of its teltrami-Laplace operator can be expressed as the sum of the eigenvalues for the I-dimensional flat torus T.

---

A.2.2 Covariance after convolution CAk.
Let us place centered factored Matérn prior in the convolution kernel k: 2 x X + RR,, that: Ck x', x')x") c(z, Mz, c(z, æ'; Mzi La
(88)
When clear from context, we will suppress the dependency on the hyper-parameree of the Matérn kernel.
So, let us consider the quantity CAulf, [a] for arbitrary functions fn and f2:
Acpr [ [z](2,22) z²)
// Cx,,))C((,, x²) 2') (2) dæda''
(89)
I/s (](())) ((zz l,)a al: va, (e) fi (x) drdx
(90)
= x(z,z') / fy (a) x²) dadar
(91)
d²
Σ (2m) d= /1 (() dodæ' II
= =(2x22)
cn (2)
(92)
nE2da
j=1
de
c(z z'; (2T))
Σ
FS m.
[] II
(93)
nEZd*
j=1
(2m) 22,
Σ
FS [ FS, [
FS,
è (n]:
(94)
nE2d*
B Proofs
In this section, we include the proofs for Lemma 3.2 Theorem 3.1 and a short lemma on the well-defined-ness of the activation operator.
Well-defined-ness of the point-wise element-wise activation operator B.1
Lemma B.1 Let (X, E, ux) be a finite measure space, i.e. px S oc, and a: IR
Borel
measurable function such that
|a(x)]
sup = 1+ |z]
(95)
for some constani C R. Then, the operator [[]: (2 (2 = o is well defined.
Proof. Remember that a function f in (22 if, and only if,
/ |[(=)]2 d(x((x) < co.
(96)
Now, from the linear boundedness condition, we know that for any f (²())) and any x E X: [a(f(=))) < C(1 [f(x)),
(97)
by squaring both sides and taking integrals,
/ [(((()))) dux(x) Ca / (1 ([+1E(=))) dxx(②)
(98)
Now, note that the constant function is in (xX)) since fx 1 dux(w)) ux(x) < so and that [f(-) in L.X X. Thus, from linearity, 1 [f(-) is also in L and fx 1+ |fxx) Therefore,
/ [[(((=))) dpx(x) < co.
5

---

B.2
Compositionality of covariance functions
Lemma 3.2. Let :1: (x; + (2) be random operator and B2: (2(
(x) be a centered function-valued Gaussian process. If the following assumptions hold:
For all f (2; TR.)) and E X, each component of Bı (f)(aa) E RR is independent and
identically distributed such that the covariance function CB, [f, gl CB. g.
The covariance function of B2 can expressed, for all f,g E L (X;
[f,[
[ and the function h CB2 [h] continuous map from 1.2 X) itself.
Then, B2 o B1 converges in distribution to a function-valuue valued Gaussian process as - oc, and
BB [,. [2] = CB2llng [2]
(100)
Proof. Consider set of size N N+, hf,,, C (2(X;R (x()), then define the N N-dimensional vector:
(h1, (B2 B1) B.)(f]),,... (hx, (B2 Bi)[fx])]] E R
(101)
Additionally, define the function:
CB [C, f:]: L2 (x; L3 (x; (x x x) BB
(102)
Then, the conditional random variable and covariance:
{ūB, [f., fyl = is Gaussian distributed with zero mean
cov(2;, æj CB, [f., fJ) = (hy, (h., cn, C.n [f, f])))
(103)
We want to show that every converges in distribution Gaussian distribution when J oo, thus, it is useful to remember the following facts:
Multivariate Levy's continuity theorem. A sequence of random variables {x, 1- converges to another one Ixc if and only if the sequence of characteristic functions oz, (t) = Ejexp(i : tTx,)], where i = 1-1, converges point-wise to Ox
Characteristic function of N-dimensional Gaussian distribution. If N(O,E), then @ (t) = Et) and @u (t) I 1, for all t E R.W.
Strong law large numbers. As / + oc, the random element K(f,, fy] converges strongly to the constant CB, [f,, fy], for all f.
Portmanteau theorem. Given a sequence of random elements in H converging in distribu- tion (x; } _ + Toc, then, lim, (x:)) (xx)),, for all bounded and continuous functions f: H - R.
Thus, begin with the characteristic function of the variable
dz (t) E[exp(i t tTz)],
(104)
by the tower rule, write
EE [[exp(i xTx) (CB, [f, f.])) Etexp(tT Σt),,
(105)
where, Σl = cov (2). 2* č32 [fy, f,]) is a random variable.
Now, because of the continuity of inner products and the assumption that ca, is continuous, we know that the mapping h + cov (2,, CB, [f,, fg] = h) is continuous. With this we take the limit:
lim dx (t) lim Elexp(t=EE)) 0
(106)
-0000
using the portmanteau theorem, we get that:
(hI, (h CB.
lim T3 [f. ((.fil))) +o
lim dz(t)= E exp tT J+00
1
(107)

---

and, finally, by the strong law of large numbers, we write the expectation as:
lim dx(() exp (eT (hi, (hi, Cn (en (G, )
(108)
1-000
Therefore, we have shown that converges to a centered Gaussian distribution with:
cov(zj, =x) cov(2j, zi CB, [G, fx] cB, [f, fi]] (hx, (hy, CBa [B,[C,, (fl))
(109)
Since the (fn, h W is arbitrary, have shown that B2 B1 is centered Ainctionvvalued Gaussian process with covariance function [B,[C, -1-
B.3 Infinite-width neural operators are Gaussian processes
Definition B.1 (Iterated convergence distribution). Let Xg be random variable for each i
[1. . C N The iterated limit
lim Ag +00 Vix 1-000
lim (... lim
Xx)
..)
(110)
+2000
whenever exists, is defined as the iterated limit in distribution. That is, suppose there random variables (ooofaa.:...a)) Ak) (oo, Aels" X (000,00.. .o] such that for every d2, 11 4x
Aiaatadam de-
X 100 d-ndd
i1 0o.
(111)
X Xpoodaiam 1ddl
4 X [po0 de 1866
(112)
X 110,000000... co,iel S X 0100,00.0.....oomm 000000
(113)
Then we define the iterated limit of X; as
lim lim (... lim (xi) ..))...)) +00
0100,00,--..00)
(114)
1ig-1-000
11-000
Theorem 3.1. Let X C Rd= be a measurable space and let H(X;R') C L²(X;R") be RKHS for any J E N Then, for given depth D E N consider vector positive integers
-|oo, JJ, J D-1, 1|_ E ND+1 and a J Jiindexed neural operators 2 of depth D:
T (D) 000 DDD-1) E (X + B²)) + (8 R),
(115)
where,
2 = H()) E (x,;) + H( and
(116)
H (Akio w()) E L (2; H(
(117)
with ( E Bee 1, and (()) E H(X X: Rex.e. 1).
When all parameters are independently distributed priori according
W ~ (0. Je and (t) ~ ~GP(O, Cyt Jt-
for E .......d. (118)
then, the iterated limit lim lim 2))) in the sense Definition B.l is equal to a function- Jp-1 J00
valued GP (PP) GP(O, coc), where coolf, [f. g] is available in closed-form.
Proof. First, we note from Section 3.1 that the covariances cwcc [f.g] and CA [f, (f.g] are equal to:
Cyrc [.g] = o J
gTfl, and, CA [f.g] Asuee
gff I
(119)
Je-
7

---

such that both depend on the empirical covariance J,,, for all g, E (2 1) and & E N Therefore, since H(4) is the sum of these two independent function-valued Gaussian processes, we have that H7() (O, cle(te1) such that:
(
ff c( (g-f/e-- Ac2oo [gTT/Je-1] oig
(120)
With this in mind, we proceed the proof by induction on the depth D.
Base case. For the base case D 1, we consider the operator z Therefore, there are limits to consider in this step. Nonetheless, as discussed in the previous paragraph, this quantity is a valued GP with covariance:
[f, [f.g] g] = c (, [r = -Acoo (//// + oigTt/J-
(121)
Therefore, our claim is proven.
Inductive step. Our inductive hypothesis says that, for specific & E N we have that the iterated
limit J+-
lim lim
2 converges in distribution a z GP (0, c(0) I1.)
J1000
As first step, we would like to prove that
lim ((++)) o 2 -1000
lim 100
(122)
converges in distribution to
H(++1) oo z(O)
(123)
Consider an arbitrary set of size N E N
(F, = ((f,, hy), (fN. hN)} C (2( x L22 I
(124)
and define the variables [| E R and D (8) [ E L'3 (B such that:
[x[F, Hl]m = (nn H(C+)) (zzz)) [fm]) and,
(125)
[z]] Fllm 2 [f,]
(126)
By definition, 2[F, H| conditioned on [F) follows a multivariate centered Gaussian distribution
(o (ZZf((())) with covariance matrix:
[{))} (be.
(127)
Thus, by the tower rule, the characteristic function the marginal distribution of =[F, H is:
*z(F (t) E/exp(itTz]/ 2(J, H4)
[F]] E (xxp(ETE(Z5 (F)+t]
(128)
Now, consider the point-wise convergence of the characteristic function:
lim
lim 2z[F. (t) =: Φx (t) +00
(129)
1
Using the portmanteau theorem and continuity of E()), have that:
1
do (t)
F. exp
lim lim m mo
(130)
Now, our inductive hypothesis says that 20 converges in distribution to function- Gaussian process 2.6 with each output being i.i.d. With this fact, we can conclude that ( 2 [r,)] This means that:
[F] also converges
in distribution to the corresponding variable: (Z [F]]
toe (t) = R. exp E 2 (F) 1))
(131)
8

---

which is the characteristic function of variable defined as:
[ [F, H|Ja = hn, (((+1))))))))))) 6 (n))
(132)
Therefore, =[F, H. iteratively converges in distribution E|F, HH,, as Jr for every S E. Since the set (F, H) arbitrary, we can conclude that (Hr(++1) o 7 also converges distribution to (H+++)) = random operator.
From the induction step, we know that the entries in (or 2 are i.idd since the entries of Z are also i.d. Therefore, we use Lemma 3.2 to show that lim (H(e+1) o 2 converges in distribution function-valued Gaussian process with covariance function
J,-00
Clet1) gl CHæ+t (0o25)))
c(
[f
[f.
(133)
Therefore, we just proved by induction that the iterated limit lim lim zP_)
converges in
J 11000
distribution to a 2 GP((O, Coo I,,) and this covariance function equal to:
Coo [[.g] c [f, gl = cldid 1/d -.
(134)
C Experimental details
In this section, we describe the setup for our experiments. As previously mentioned, all experiments were run in desktop machine with 3.8GHz Intel Core i7-9800X CPU and 24GB NVIDIA Titan RTX (TU102) GPU. More details for each experiment can be found below.
C.1 Empirical demonstration of results
For both experiments, the input function f:: TR has band-limit B 3, with its output values f(x) sampled from uniform distribution ((-111) In other words, we can express this band-limited function as:
3
3
1
2T x
f(x)
4/
(135)
7
where each fa ~ 1(-1,1) is independent and identically distributed.
In the first experiment of Fig. 2 we construct the operator layer under the usual formulation:
H[f[()): LT(TT) - (T) [x[f(()) + wf(x),
(136)
where 1) 1) and k follows the
Gaussian process distribution (Section and Appendix (. with with band-limit = 3 and variance o² 1/77. Then, the operator on evaluated zero H (f) with increasing sample sizes.
4.1
For the second experiment of Fig. I we construct the single-layer neural operator:
[1]](x): LT L3 (T: -L)
(T) = (w) ReLU (Ax + +ww))
(137)
where J is the width of the hidden layer, and Wy ~ -/ 1/J), wy ~ WN 1), and k follows an i.i.d. band-limited Gaussian process distribution (Section 4.1 and Appendix A. with band-limin B and variance 2² 1/7. For varying widths E (1, 100, 1000), we evaluate 10,000 samples of the operator at (0) and show the density of the empirical distribution using kernel density estimation (KDE) with Gaussian kernel.
These experiments are implemented in the file experimentssfno-linie fim ipynb.

---

C.2 Regression
We consider FNOs of increasing width, E {1, 10, 100) and J {1.3, 3, 10, 100,0000000 5003 for the synthetic and ID Burgers' respectively, as well as oc- Do-FNOs, both with increasing kernel band-limits B E {1.5, 2.0.. These single-layer neural operators are constructed as:
1, x)) (T) (T - (T)
(w) o ReLU o (Ak + w1 (1))
(138)
where J is the width of the hidden layer, and (N(0. 1/J). w2 ~ (O, 1), and k follow an i.idd. band-limited Gaussian process distribution (Section 4.1 and Appendix A.I with variance o² = 1/(2B + 1).
The hyperparameters of the oc-FNO estimated using L-BFGS, while the parameters of the FNOs are optimized with Adam using a step size of 0.001. We evaluate all models using 5-fold cross- validation and report the average and standard deviation of the empirical L norm of the prediction emor. For oc-NNOs, we use the posterior mean as the prediction.
This experiment is implemented in the file experimenss/traie .y.
Synthetic regression
start by defining the ground truth Fourier neural operator (FNO) which will generate our training and test data:
Zru [ (T(T)) + (T) (T) = (wy ReLU (Ax + (x))
(139)
where the hidden layer's width is 1 and the band-limit of is equal to Next, we sample n = 100
input functions f,: T -> E with the same band-limit B 5 and 4(-1111) 1), so that we have:
outputs
1
2x
(z)
fie
(140)
11
11
s'=-5
where each /1s 1)), independent and identically distributed. We then compute Zurue [G] an equally spaced grid given by 2 2m
..... 5
C R
1D Burgers' equation
This dataset provided from PDEBench Burgers' equation:
(Takamoto et al. 2022). which includes solutions to the 1D
0 0t
1 0
02 Fz ut,,:)),
utt, x) + (t. 2 3r
(141)
where E (0, 1) and t (0,2) are independent variables and v is the diffusion coefficient.
The regression task set up with v 0.002 and a collection initial conditions {u(0, -) = f. ) and their respective end states {u(2,-) .) = gi |_ Due to memory constraints when creating the covariance matrices for oc-NNO, we subsample the original dataset to n = 100 functions and grid size of 103. The original data can be downloaded at https: /daruudd..//////// uni stuttgart.de/ app/aaccess/dattafeee/88833
10


Reviews:
[{"review_id": "wgvU2fN8E2", "reviewer": "Reviewer_PtA6", "summary": "The paper studies neural operators in the infinitely wide limit. The paper provides a thorough background on (neural) operators alongside probability theory (including the strong law of large numbers and Gaussian processes).\n\n\nThe main contribution is a proof that the (a priori) infinite-width limit of neural operators is a function-valued Gaussian process under certain conditions. The paper thus extends results established for the infinite-width limit of Bayesian neural networks priors being Gaussian processes. It does so under similar assumptions of shrinking covariance with width and Gaussian assumptions (on the operators in each layer of the neural operator). The provided proof involves establishing each operator layer as a function-valued GP and that the covariance functions of these layers compose. Following this, the authors show closed-form representations for the covariance of an infinite-width Fourier neural operator and a Toroidal Matérn operator. \n\n\nFinally, experimental validation shows that the empirical variance and output distribution match theoretical predictions as neural operators become wider in a simple, synthetic problem. An example regression task is studied, showing the difference between the infinitely wide FNOs and trained FNOs in approximating a simple, predefined ground truth FNO.", "questions": "See points (a) and (b) above, primarily.\n\nDo you think there is a connection between your results and the experimental results showing optimum widths (or diminishing returns) for FNOs in, for example, Section B.2 of Georg et al. (2024) [2], or the analogous results regarding modes in Section C of Qin et al. (2024) [3]? I am just curious, and you do not have to have or provide answers to this: do you think the initialization behaviour of the infinitely-wide FNO could be related in any way to the findings on \"trainability\" by Koshizuka et al. (2024) [4], who note that stable training requires initialization \"at the edge of chaos\"?\n\nWhy did you choose to investigate a J=1 ground truth FNO in Section 5.2, rather than, for example, a J=10 configuration? Again, out of curiosity, does the infinite-depth perspective, in your view, offer any insights into why increasing the depth of an FNO appears to worsen performance (disregarding the main issue of the discrepancy between the behaviour of newly initialized and trained models)? Similarly, out of curiosity, do you have any intuition as to whether, e.g., Eq. 26 might inform the limitations encountered by the infinite-depth FNO in the context of Figure 3?\n\nl.70: \"matrix-valued\"?\nl. 164: \"... whose image *is an RKHS...\" (?) and \"a* covariance function\"\nl. 174: \"... we apply know...\"\nl. 186: undefined reference\nl. 241: “…evaluate compre our model...\"\nl. 325: \"...scales with cubically...\"\n\n[1] Takamoto, Makoto, et al. \"PDEBench: An extensive benchmark for scientific machine learning.\" Advances in Neural Information Processing Systems 35 (2022): 1596-1611.\n[2] George, Robert Joseph, et al. \"Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs.\" Transactions on Machine Learning Research (2024).\n[3] Qin, Shaoxiang, et al. \"Toward a better understanding of Fourier neural operators: Analysis and improvement from a spectral perspective.\" CoRR (2024).\n[4] Koshizuka, Takeshi, et al. \"Understanding the Expressivity and Trainability of Fourier Neural Operator: A Mean-Field Perspective.\" Advances in Neural Information Processing Systems 37 (2024): 11021-11060.", "limitations": "Largely, yes, authors discussed the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is generally concise and very clearly communicates the contributions. The paper concisely motivates the study and provides a good, if relatively thorough and dense, review of necessary background material. Instead of the level of detail in Section 2 towards establishing the setting, some of the details could be considered for the supplementary material, thus possibly allowing for more detail in, e.g., Section 3.2.\n\n\nThe theoretical contribution builds on well-established approaches to studying the infinite-width limits of neural networks and adds, to my knowledge, novel theoretical results applicable to, e.g., widely-adopted Fourier neural operators. While outside my immediate area of expertise, the provided proof appears correct and the claims on the theoretical results seem well-founded, and the results are presented clearly.\n\n\n(a) On experimental results: In the introduction, the authors refer to the data that is analysed as realistic scenarios and real-world data (ll. 11 & 35). Given that the experimental validation in Sections 5.1 and 5.2 is relatively simple “synthetic” settings, I find this misleading. I recognise that the contribution is theoretical and that limited experimental validation is not a big weakness, yet I would suggest to soften the language in the introduction (simple, synthetic problems) - or, alternatively, the present language necessitates adding an analysis of actual realistic/real-world data, for instance the smallest (or a simplified version of) problems in PDEBench [1] or the simplest problems studied Sec. 6 in Kovachki et al. (2024).\n\n\n(b) For the theoretical results, I find that the implications of the results are under-explored in the paper’s presentation. The contributions can stand on their own, and as the authors conclude, their contributions \"lay a foundation for future investigations\". However, the consequences of the infinite-width view of Bayesian neural networks' priors, as highlighted early on by, e.g., Neal (1995), could be considered for the infinite-width neural operator setting by analogy. Neal, for one, considers the necessity of infinite variance priors and heavy-tailed assumptions (calling the Gaussian process priors \"disappointing\"). While the theoretical results are sufficiently novel and well-founded (and supported by some experimental results), the paper would, in my opinion, be much stronger if it also discussed current experimental findings on the limitations of, e.g., FNOs in the context of the paper's theoretical results (see question related to empirical studies below).\n\nI find the paper to be a strong submission as it is, but I would strongly consider increasing my score if a valuable discussion of the points in (b) were added or if a more realistic setting, as discussed in (a), were added.", "was_revised": true, "final_justification": "The authors partially addressed one of my concerns by exploring results on the 1d Burger's equation, and their added preliminary results does provide evidence towards their relatively strongly worded claims related to realistic scenarios and real-world data; I would still suggest some softening but I appreciate and recognize the supporting evidence provided. \n\nSimilarly, while the authors provide a discussion of the implication of the theoretical work, and the theoretical contributions are strong in their own right, I still find the part (b) in \"Strengths And Weaknesses\" of this review to be a relevant critique of the work even if the weakness were to be ameliorated by adding a discussion similar to the one provided by the author's in the rebuttal to my review. \n\nI am raising my rating to a weak 5 in recognition of the fact that this work constitues a necessary theoretical foundation for the further study of infinite-width limit of neural operators.", "rebuttal": "Thank you for your valuable input and pointing out places where our text can be improved and other typos.  We hope our answers sufficiently address your concerns. If we have left blank spots or you would like additional clarifications, we will be happy to discuss further.\n\n> Connections with existing FNO literature, further theoretical results.\n\nThank you for highlighting these recent empirical and theoretical results on FNOs. Indeed, we believe that the conclusion of Neal (1995), that, compared to finite-width Bayesian neural networks, their infinite-width counterparts are \"disappointing\", has become the common sense in the community, as it has been shown in both the Bayesian setting and in the empirical risk minimization setting (e.g. neural tangent kernels) that such infinite-width networks do not learn features from data. This analysis has been expanded by the muP / mean-field perspective, where it has been shown that there exists an optimal way to initialize networks to induce feature learning in the so-called “edge of chaos”.\n\nAs Neal (1995) laid the foundation for the analysis of infinite-width neural networks, our paper establishes the sufficient conditions for the existence of the infinite-width limit of NOs and characterizes their covariances at initialization. While we are very interested in continuing this analysis into infinite-width NOs trained by stochastic descent, similar to neural tangent kernels, we couldn't have done so without this foundation.\n\nRegarding connections between our paper and the works of Gerog et al. (2024) and Qin et al. (2024), we would not expect our theoretical results to shed light on questions about optimal width or band-limits of FNOs, in the same way that we do not expect NN-GPs to be very illuminating about the specific post-training behavior of finite-width NNs. \n\n> Why a J=1 ground truth FNO was investigated, whether an infinite-depth perspective explains why increasing FNO depth worsens performance, and if Eq. 26 sheds light on the limitations of the infinite-depth FNO in Figure 3.\n\nWe chose J = 1 as the ground truth to stress-test the infinite-width models, as we would expect this model to be the most different from the infinite-width neural operator, as shown in Section 5.1, compared to other values of J.\n\nAdditionally, we would like to clarify that our experiments only consider one-hidden-layer deep FNOs and infinite-width FNOs. That said, we speculate that a possible reason behind the performance decrease of FNO as the width (J) increases is that our training set has n = 100 functions, giving a total of (2*5+1)*100 = 1,100 data points, which is much less than the number of parameters. Nonetheless, we would like to recall that the main point of Figure 3 is showing that, regardless of the model's width, the model’s band-limit needs to at least cover the data’s band-limit for accurate predictions and that, even as the width increases, the predictive performance of finite-width models is not directly related to that of infinite-width ones.\n\n> Claims around experimental results\n\nThanks for pointing this out. We agree that we have oversold our experimental results. As suggested, we will soften the tone of lines 11 and 35 in our revised manuscript. To move our results toward a more realistic setting, we set up an additional experiment with the same set-up as in the paper but using the 1D Burgers' equation dataset with $\\\\nu = 0.002$ from PDEBench. The task is to predict the end state (t=2) given the initial condition (t=0). Due to memory constraints while inverting the kernel matrix, we subsample the total dataset data to n = 100 functions and a grid size of 103.\nOur preliminary results are as follow:\n| Model | B  | J     | Test L² norm |\n|-------|----|-------|--------------|\n| FNO   | 20 | 10  | 8.05±0.63    |\n| ∞-FNO | 20 |  ∞  | 11.31±0.85   |\n| FNO   | 20 | 100 | 20.72±4.00   |\n| FNO   | 20 | 1   | 24.02±3.73   |\n\nAt a glance, the finite-width FNO has a different optimal width for this experiment and still there is no correlation between results from models with increasing width and the test loss from the predictive posterior mean of the infinite-width model."}, {"review_id": "YGJt295fjc", "reviewer": "Reviewer_oSXr", "summary": "This paper extends the classic result that finite width neural networks converge to Gaussian processes as the width of the network width tends to infinity. The results are are extended to the operator learning setting, where the input and output of a network are infinite-dimension functions. The result briefly states that as the output dimension of intermediate functions in the operator tend to infinity, the output of the network converges to a function-valued Guassian process with a computable kernel.", "questions": "- Line 186 has a missing reference.\n- Do the authors think it worth discussing Deep Gaussian Processes [1]? They seem a related method that applies in the Bayesian inference setting, and therefore potentially a better comparison than finite-width FNOs.\n- Do the authors see any routes to applying these methods in practice, given they perform less well than finite width FNOs in this simple regression setting? Do they see any impact on finite width FNO design that could be gleaned from this paper?\n- Can the authors think of any classic FNO settings (PDE learning, weather modeling) where a more in detail experiment might be possible?\n- Can the authors explain why lower J seems to return better results for finite FNOs? This seems unexpected to me.\n\n\n\n[1] https://arxiv.org/pdf/1211.0358", "limitations": "None.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The contributions of this paper are reasonable, from the mathematics point of view.\n- The mathematics are well presented and clear.\n- The position in the literature and relation to other works is well discussed.\n\nWeaknesses:\n- As a practical method, the $\\infty$-FNO is not well evaluated, and not in the settings typical of neural operators.\n- As a method of guiding architecture design, it is not clear to me the paper provides insight to guide FNO design", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for your feedback. Below, we address your concerns. We hope our answer elevates your appraisal of our work. Please let us know if you have any other questions or need more clarification. We would be happy to engage further.\n\n> Do the authors think it worth discussing Deep Gaussian Processes [1]? They seem a related method that applies in the Bayesian inference setting, and therefore potentially a better comparison than finite-width FNOs.\n\nTo the best of our knowledge, DGPs have not yet been applied to operator learning and it is not clear how it could be done. In terms of our theoretical results, previous work on neural networks indicates doing either infinite width limits or infinite depth and width limits recovers Gaussian process models, but not DGPs. Our current paper generalizes these results to the operator learning setting, thus, we did not find a link with DGP models.\n\n> Do the authors see any routes to applying these methods in practice, given they perform less well than finite width FNOs in this simple regression setting? Do they see any impact on finite width FNO design that could be gleaned from this paper?\n\nThe purpose of the regression experiment is not to evaluate the performance of FNOs; it is rather to stress-test the FNOs when the true width and bandlimit are violated in the model specification and show that their performance is not correlated, as our paper is not focused on advocating for infinite-width FNOs as a modelling tool. A priori, this does not mean that infinite-width FNOs are not applicable in more realistic scenarios.\n\nHowever, based on previous results on infinite-width neural networks, we see promising avenues of application of our work. For example, Adlam et al. (2023) [arxiv:2303.05420] has successfully scaled infinite-width NNs on the millions of data points setting and achieved competitive results against SotA methods and Meronen et al. (2020) [NeurIPS 2020, arxiv:2010.09494] derived activations for finite-width NNs based on infinite-width models,  achieving better out-of-distribution detection and uncertainty quantification.\n\n> Can the authors explain why lower J seems to return better results for finite FNOs? This seems unexpected to me.\n\nWe speculate that a possible reason behind the performance decrease of FNO as the width (J) increases is that our training set has n = 100 functions, giving a total of (2*5+1)*100 = 1,100 data points, which is much less than the number of parameters."}, {"review_id": "XhV8N2IQsI", "reviewer": "Reviewer_bNVY", "summary": "This paper studies when infinite neural operators behave like Gaussian processes. In particular, the authors find that under certain conditions, arbitrary-depth neural operators with Gaussian distributed convolution kernels converge to GPs. These conditions include infinite width, randomly initialized weights. This is demonstrated empirically through empirical estimates of the variance converging to the theoretical value.", "questions": "- How is the infinite width neural operator implemented in practice, especially in the case of the results for Fig. 3?\n- Is the infinite width definition applied to both the projection layers as well as the kernel $K$ and matrix $W$?\n- Does the theory still hold if the number of Fourier modes used in each neural operator layer is fewer than what is necessary to represent the data? Clarity on this and other limitations would potentially increase my score.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "(+) Well written and clear paper.\n\n(+) The proofs and proof sketches are also clear. \n\n(+) The empirical results demonstrating the density estimation and variance converging are compelling. \n\n(-) The definition of both NO and infinite-width neural operator are unclear. It looks like the lift and projection layers are ignored, but it could also be interpreted to meaning that $A_{K_1} = A_{K_d} = 0$. Please state this more clearly and define the infinite-width NO.\n\n(-) It is not clear exactly what the that need to be met are for the infinite with neural operator to behave like a GP.", "was_revised": true, "final_justification": "This paper provides a strong theoretical framework for understanding neural operators with empirical experiments that demonstrate this as well. The authors have addressed the concerns of most of the reviewers and have addressed my main concerns. I will keep my score as is.", "rebuttal": "Thank you for attentively reviewing and appreciating our work. We hope our answers below appropriately address your concerns. If this is the case, please consider raising your score. Otherwise, we will be more than happy to discuss our contributions in further detail. \n \n\n\n> How is the infinite width neural operator implemented in practice?\n\nAs discussed in Theorem 3.1, an infinite-width neural operator is equivalent to a Gaussian process with a specific kernel $c\\_\\\\infty$. In Section 4, we show how to compute this kernel in two cases: Eq. (32) and Eq. (36). Given this, we can implement the infinite-width NO just like any kernel method. For instance, given some training and test data, we can compute the Gram matrix $\\\\boldsymbol{C}$ of the kernel by evaluating it on every pair of training points. We can sample from the corresponding GP by multiplying independent Gaussian samples with the Cholesky decomposition of $\\\\boldsymbol{C}$, and the inverse of $\\\\boldsymbol{C}$ can be used to compute the posterior for prediction.\n\nAdditionally, our supplemental material includes an implementation of the infinite-width model in PyTorch.\n\n> Assumptions behind the theoretical results.\n\nWe apologize for the possible confusion regarding the lift and projection layers. Kovachki et al. (2023) separate the layers of their architecture into three sections: lift, NO layer, and projection components. However, as we discuss in Section 2.1, both the lift and projection operators can be represented by just the NO layer by simply setting $\\\\boldsymbol{K}$ to zero. As such, in Theorem 3.1, when we defined $\\\\boldsymbol{K}\\_\\\\ell$ and $\\\\boldsymbol{W}\\_\\\\ell$ to map functions from $J\\_{\\\\ell-1}$ channels to $J\\_{\\\\ell}$ channels, this includes the possibility of either of them being zero. Our theorem requires that all $J\\_\\\\ell$ values approach infinity, encompassing all layers, both $\\\\boldsymbol{K}$ and $\\\\boldsymbol{W}$, thus the hidden layers of the lift, NO layer, and projection components all go to infinity.\n\nAs for the number of Fourier modes, our theorem is applicable to any number of Fourier modes, regardless of whether they are sufficient to represent the data. As stated in Theorem 3.1, the only requirements are that the weights and kernel for each component operator are independent and identically distributed (i.i.d.) and that their covariance shrinks with increasing width. In the FNO model, Section 4.1 demonstrates that the number of Fourier modes (2B+1) acts as a configurable hyperparameter in kernel computation. Additionally, our theorem is general enough to encompass different NO architectures which might not even have a limited number of Fourier modes, such as the Matérn kernel of Section 4.2 which inherently possesses an infinite number of Fourier modes, though in practice, it is truncated by the input data's bandlimit."}, {"review_id": "cUJz2z6IvF", "reviewer": "Reviewer_BQMy", "summary": "The authors show that infinite-width neural operators converge to function-valued Gaussian processes. The resulting covariance functions are computed for two neural operators, including the Fourier neural operator. This result is validated numerically.", "questions": "- Most importantly, in Section 3.1, what is meant by the covariance function associated with a particular operator? From the description in the paper it is not at all clear what $C_{B_1}$ or $c_{B_1}$ are supposed to be. Is the idea that $B_1: L^2(\\mathcal{X}, \\mathbb{R}^{J_1}) \\to L^2(\\mathcal{X}, \\mathbb{R}^{J_2})$ is itself a Hilbert space-valued Gaussian process and $C_{B_1}$ is its covariance function? (This seems to be what is implied by the following discussion about covariance functions of neural operator layers under Gaussian assumptions.) If so, this needs to be clarified, as it would only make sense specifically if $B_1$ is a Gaussian process rather than an arbitrary operator. I am also not clear on what the distinction between $C_{B_1}$ and $c_{B_1}$ is supposed to be. \n- Is the closed form of the limiting covariance $c_{\\infty}$ in Theorem 3.1 only known for certain activation functions $\\sigma$? The discussion in Section 3.1 seems to imply that the Gaussian process result only holds for Lipschitz $\\sigma$ with $\\sigma(0) = 0$ and closed-form covariance is only known for some activation functions such as ReLU and sigmoid. These assumptions should be explicitly stated in Theorem 3.1. \n- What is the purpose of the regression experiment in section 5.2? It doesn't seem like it has anything to do with the infinite-width GP result, but rather it just seems to be a comparison of the predictive performance of FNOs and standard finite input/output neural networks on a particular regression task. I don't see how this contributes to the paper.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The fundamental result that neural operators induce Gaussian processes over their outputs is interesting. As the authors note, the theoretical properties of neural operators are still largely unexplored, and this result suggests a natural means of uncertainty quantification for neural operators with Gaussian parameters/kernels. \n\nOne important limitation of this result (which is also applicable to similar known infinite-width NN/GP equivalence results) is that the characterization only applies to neural operators with i.i.d. Gaussian parameters/kernels. Although neural networks in practical scenarios often begin with randomly initialized parameters, how these are affected by training in practical regimes is still largely an open question, which means that any connection between these infinite-width Gaussian process/NN equivalence type results and neural networks (or operators) actually used in practice is still largely unclear. \n\nI have some particular confusions on Section 3.1 which seem fundamental to the paper's result (see Questions below). If these can be answered and clarified in the paper's presentation, I will consider raising my score.", "was_revised": true, "final_justification": "I thank the authors for their detailed response; they have satisfactorily answered my questions and I have raised my score accordingly. I do encourage them to make some of the points in their rebuttal more explicit in the paper. As they have noted, it should be clarified in Lemma 3.1 that $B_1$ and $B_2$ are Hilbert-space valued stochastic and Gaussian processes, respectively. They should also more explicitly discuss which assumptions on the activation functions are necessary for the various results to hold. I have no problem in principle with assumptions on activation functions, it should just be stated clearly what assumptions are required for an NO to be well-defined, what assumptions are necessary for a closed-form covariance function to be computed, etc. I also encourage them to specifically state that $C_{B_1}[f,g] = c_{B_1}[f,g] I_j$ means that the covariance is proportional to the $j x j$ identity matrix (I didn't grok on my initial read that $c_{B_1}[f,g]$ is supposed to be a constant depending on $f,g$) and to note explicitly that this is equivalent to the the outputs of $B_1$ being i.i.d. across dimensions (I understand that this can be inferred from all the definitions, but because the setup is already heavy on definitions and notation, this makes it more digestible).", "rebuttal": "Thanks for your thoughtful feedback and contribution towards sharpening our definitions and fleshing out the assumptions behind our theoretical results . Below, we provide clarifications regarding the content of Section 3, as well as our experiments in Section 5.2. \n\nWe hope our answers sufficiently address your concerns and elevate your appraisal of our work. Otherwise, we would be happy to engage further.\n\n> What is meant by the covariance function associated with a particular operator?\n\nWe apologize, the statement of Lemma 3.1 in the main text is missing some details that are only present in Appendix B.1. This will be corrected and improved in the main text. In summary, $B\\_1$ is a Hilbert space-valued stochastic process, which is not necessarily Gaussian, while $B\\_2$ must be a Hilbert space-valued Gaussian process. What we mean by the covariance function associated is meant to be the covariance function of these stochastic processes/random operators.\n\n> Distinction between $C\\_{B\\_1}$ and $c\\_{B\\_1}$\n\nInitially, it is important to note that $B\\_1$ maps functions landing in $\\\\mathbb{R}^{J\\_1}$ to functions landing in $\\\\mathbb{R}^{J\\_2}$. Consequently, as detailed in Section 2.2, lines 89-91, applying its covariance function to two input functions, $\\\\mathbf{f}$ and $\\\\mathbf{g}$, yields $\\\\mathbf{C}\\_{B\\_1}[\\\\mathbf{f},\\\\mathbf{g}]$. This result is a function that takes two domain elements and outputs a $J\\_2 \\\\times J\\_2$ matrix. This matrix structure accounts for potential cross-covariance between the output dimensions of $B\\_1[\\\\mathbf{f}]$ and $B\\_1[\\\\mathbf{g}]$. Therefore, when we state that $\\\\mathbf{C}\\_{B\\_1}[\\\\mathbf{f},\\\\mathbf{g}] = \\\\mathrm{c}\\_{B_1}[\\\\mathbf{f},\\\\mathbf{g}]\\\\boldsymbol{I}\\_{J\\_2}$, we are indicating that this matrix is proportional to the identity matrix. This condition is equivalent to requiring that the output dimensions of $B\\_1[\\\\cdot]$ are independently and identically distributed (i.i.d.) for all input functions.\n\n> Assumptions on the activation function.\n\nWe would like to clarify that some assumptions on the activation function are necessary for any neural operator to be mathematically well-defined, including finite-width one, thus are included when we assume $Z\\_J$ is a NO. For example, Kovachki et al. (2023) assume measurable, linearly bounded activation functions, a condition slightly weaker than Lipschitz continuity, which includes activations like ReLU, ELU, tanh, and sigmoid.\nNonetheless, our statement in line 150 (Section 3.1) is overly restrictive and could be misleading; it should say, \"if $\\\\sigma$ is Lipschitz [...] then this is a well-defined operator,\" rather than \"only if.\" We will amend the main text and add to the supplementary material a proof that, for a toroidal domain, any measurable linearly bounded activation induces a well-defined $L^2$ operator.\n\nAs for the functional form of the activation function, closed-form solutions are only known for certain activation functions but these do not invalidate the result theorem. However, it is not limited to just ReLU and sigmoid. We have cited these solutions because they are the most popular activations, but the infinite-width neural network literature has continued to explore more closed-form solutions and numerical approximations to Eq (16). For example, Han et al. (2022) [arxiv:2209.04121] provides ways to compute exact and approximate solutions for a wide class of activations, and these have been implemented in the [neural-tangents Python package](https://neural-tangents.readthedocs.io/en/latest/stax.html#elementwise-nonlinear).\n\n>  What is the purpose of the regression experiment in section 5.2?\n\nIt is true that Section 5.2 does not relate directly to our results in Section 3, however, it shows that, regardless of the model's width, the model’s band-limit needs to at least cover the data’s band-limit for accurate predictions and that, even as the width increases, the predictive performance of finite-width models is not directly related to that of infinite-width ones."}]

Author rebuttal:

Meta-review: This paper studies the infinite-width limit of randomly initialized deep neural operators and finds a (Hilbert space-valued) Gaussian process limit that is analogous to the limits of deep neural networks. A major justification for the paper is that the inductive biases and functional properties of neural operators are largely unexplored, and this analysis of the infinite-width limit provides a strong theoretical foundation for further analyses of this important class of models. The background and mathematical content are clearly explained, and the proofs themselves are nontrivial contributions of significance. The experiments, though limited, provide empirical justification for the main theoretical results.

Beyond minor points of confusion that the authors have promised to clarify, the key limitation of this paper is the lack of discussion on how to translate these theoretical results into practical insights. In particular, the authors don't offer suggestions for insights into future neural operator design, nor do they discuss how their theory relates to known empirical limitations like optimal width findings from other papers. I encourage the authors to include such a discussion in the camera-ready version. Nevertheless, this paper is a strong and important theoretical contribution that will be of interest to the NeurIPS community, and therefore I recommend acceptance.
Decision: Accept (poster)