Neural Stochastic PDEs: Resolution-Invariant Learning of Continuous Spatiotemporal Dynamics
Authors: Cristopher Salvi
Maud Lemercier
Andris Gerasimovics
Venue: NeurIPS 2022 Accept (2022)
Keywords: operator learning
PDEs
neural differential equations
spatiotemporal data

Abstract: Stochastic partial differential equations (SPDEs) are the mathematical tool of choice for modelling spatiotemporal PDE-dynamics under the influence of randomness. Based on the notion of mild solution of an SPDE, we introduce a novel neural architecture to learn solution operators of PDEs with (possibly stochastic) forcing from partially observed data. The proposed Neural SPDE model provides an extension to two popular classes of physics-inspired architectures. On the one hand, it extends Neural CDEs and variants -- continuous-time analogues of RNNs -- in that it is capable of processing incoming sequential information arriving at arbitrary spatial resolutions. On the other hand, it extends Neural Operators -- generalizations of neural networks to model mappings between spaces of functions -- in that it can parameterize solution operators of SPDEs depending simultaneously on the initial condition and a realization of the driving noise. By performing operations in the spectral domain, we show how a Neural SPDE can be evaluated in two ways, either by calling an ODE solver (emulating a spectral Galerkin scheme), or by solving a fixed point problem. Experiments on various semilinear SPDEs, including the stochastic Navier-Stokes equations, demonstrate how the Neural SPDE model is capable of learning complex spatiotemporal dynamics in a resolution-invariant way, with better accuracy and lighter training data requirements compared to alternative models, and up to 3 orders of magnitude faster than traditional solvers.

Neural Stochastic PDEs: Resolution-Invariant Learning of Continuous Spatiotemporal Dynamics
Cristopher Salvi Imperial College London & The Alan Turing Institute c. wsssslliiin aeliiiiiierilaaa ac uk
Maud Lemercier University of Warwick
Andris Gerasimovits University of Bath ag26160bath. ac uk
maud. lemercier@warwickk
Abstract
Stochastic partial differential equations (SPDEs) are the mathematical tool of choice for modelling spatiotemporal PDE-dynamics under the influence of random- ness. Based on the notion of mild solution of SPDE, we introduce novel neural architecture to learn solution operators of PDEs with (possibly stochastic) forcing from partially observed data. The proposed Neural SPDE model provides an exten- sion to two popular classes of bhysics-inspires architectures. On the one hand, it extends Neural CDEs and variants continuous-time analogues RNNs in that it is capable of processing incoming sequential information arriving arbitrary spatial resolutions. the other hand, extends Neural Operators generalizations of neural networks to model mappings between spaces of functions in that it can parameterize solution operators of SPDEs depending simultancously on the initial condition and realization of the driving noise. By performing operations in the spectral domain, we show how a Neural SPDE can be evaluated in two ways, either by calling ODE solver (emulating a spectral Galerkin scheme), or by solving a fixed point problem. Experiments on various semilinear SPDEs, including the stochastic Navier-Stokes equations, demonstrate how the Neural SPDE model is capable of learning complex spatiotemporal dynamics in resolution-invarinn way, with better accuracy and lighter training data requirements compared alternative models, and up to orders of magnitude faster than traditional solvers.
1 Introduction
Stochastic partial differential equations (SPDEs) are the mathematical formalism used to model many physical, biological and economic systems subject the influence of randomness, it intrinsic (e.g. quantifying uncertainty) Notable examples of SPDEs include the Kardar-Parisi-Zlinn interface growth such as the propagation of forest fire from burnt region to an unburnt region [12], the Günzburg-Landler equation describing phase transitions of ferromagnets and superconductors near critical temperature [34], the stochastic Navier-Stokes equations modelling the dynamics of turbulent fluid flow under the presence local random fluctuations [31]. For an introduction to theory of SPDEs see Hairer [1 [11]; comprehensive textbook is Holden et al. [15].
extrinsic (e.g. modelling environmental random perturbations).
(KPZ) equation for modelling random
Classical numerical approaches for solving SPDEs include finite difference methods and spectral Galerkin methods [27] among others. To ensure accuracy and stability of numerical solutions of complex SPDEs, computations must be carried out at high resolution using fine discretization grids, rendering the resulting schemes computationally intractable. This limitation motivates the study of data-driven methods that can learn solutions to differential equations from partially observed data.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Related work There has been an increased interest in recent years to combine neural networks and differential equations into a hybrid approach [5, 17. 37].
Newral controlled differential equations (Neural CDEs), as popularised by [3, 18, 32), are continuous- time analogues to recurrent neural networks (RNN, GRU, LSTM etc.). The input to Neural CDE model is a multivariate time series interpolated into a continuous path X [o. 7] - IR:e; the model consists of a matrix-valued feedforward neural network fo : Rd + (gdi parameterizing the
vector field of the following dynamical system (and satisfying some minimal Lipschitz regularity to ensure existence and uniqueness of solutions)
21 = 20 + L. folzs)dX., u =0)
2u = Co(u)),
(1)
where lo Rdd + Rd and ITo : IR + Rdd are feedforward neural networks. The output response u: [0, 2] + gd is then fed to (possibly pathwise) loss function (mean squared, cross entropy etc.) and trained via stochastic gradient descent in the usual way. In practice, the term "dX," " means that the solution 2 of the equation (e.g. attention required from doctor) change of an external stream of information X, (e.g. heart rate of a patient).
change in response to
Depending on the level of roughness of the control path X, the integral in eq. (1) can be interpreted in different ways. In [18], X assumed differentiable and is obtained in practice via cubic splines interpolation of the original time series. In this way, the term "d.X," " can be interpreted as " "X,ds" so that eq. (1) becomes an ODE of the form & = fo(=,) Xx, that can be evaluated numerically via a call to an ODE solver of choice (Fuler, Runge-Kutte Kutta, implicit, adaptive stepsize schemes etc.). More generally, if X is of bounded variation then the integral above can be seen as classical Riemann-Stielljes or Young integral [38]. Neural SDEs [19, 20, 22, 26] are special subclass of Neural CDEs where the control is a sample path from a de- dd-dimensional Brownian motion (which is not of bounded variation), and eq. (1) is understood via stochastic integration (Itô, Stratonovich etc.). Neural RDEs [32] allow to relax even further the regularity assumptions on X by treating the integral using rough integration [10, 30]. In practice, Neural RDEs are particularly well suited for long time series. This is due to the fact that the model can be evaluated via a numerical scheme from stochastic analysis (called the log-ODE method [32]) over intervals much larger than what would be expected given the sampling rate of the time series. However, the space complexity of the numerical solver increases exponentially in the number of channels de, thus model complexity becomes intractable for high dimensional time series. Despite offering many advantages for modelling temporal dynamics, these models are not designed to process signals varying both in space and in time such as physical fields described by SPDEs. In particular, although these models are time-resolution invariant, they invariant, and are not well suited to capture nonlinear interactions between
are not space-resolution the various space-time points typically observed in SPDE-dynamics. Similar to CDEs, the solution u an SPDE is characterized by an initial condition uo and driving noise X. However, in the case of CDEs, (Mo, X,, 4z) are vectors, while in the case of SPDEs they are functions.
Neural Operators [21, 23, 24, 28] generalizations of neural networks capable of modelling mappings between spaces of functions and offer an attractive option for learning with spatiotemporal data [21]. Among all kinds of Neural Operators, Fourier Neural Operators (FNOs) [25] stand out because of their easier parametrization while demonstrating similar learning performance compared to other Neural Operator models. However, Neural Operators generally fail to incorporate the effect that an external (possibly random) spatiotemporal signal might have on the system they describe. In the case of SPDEs, the external signal is indeed random (e.g. sample from a Wiener process) and its presence leads to new phenomena, both at the mathematical and the physical level, often describing more complex and realistic dynamics than the ones arising from deterministic PDEs.
Contributions To overcome the above limitations faced by Neural CDEs and Neural Operators, we introduce the neural stochastic partial differential equation (Neural SPDE) model, capable of learning solution operators of SPDEs from partially observed data by processing, continuously in time and space, incoming sequential information arriving at an arbitrary resolution. We propose two separate algorithms to evaluate our model: the first reduces the Neural SPDE to a system of ODEs in Fourier space, which can then be solved numerically by means of any ODE solver of choice (emulating a spectral Galerkin scheme); the second rewrites the Neural SPDE fixed point problem, which is solved via classical root-finding schemes. For both choices of evaluation, the Neural SPDE model inherits memory-efficied backpropagation capabilities provided by existing
2

---

udjoint-based and peplict-ddiffereniiatie experiments on various semilinear SPDEs, including the stochastic Ginzburg-Landau, Korteweg Vries, Navier-Stokee equations. The empirical results illustrate several useful aspects of our model: 1) it is space and time resolution-invariant, meaning that even if trained on a lower resolution it can be directly evaluated on higher resolution; 2) it requires lower amount of training data to achieve similar or better performance compared to alternative models; 3) its evaluation is up to 3 orders of magnitude faster than traditional numerical solvers.
methods respectively, Finally, we perform extensive
The outline of the paper is as follows: in Sec. 2 we provide a brief introduction to SPDEs which will help to define our Neural SPDE model in Sec. 3, followed by numerical experiments in Sec. 4. In Appendix A, we provide an overview of the computational aspects of SPDEs used to design the model and solve SPDEs numerically. Additional experiments can be found in Appendix B.
2 Background on SPDEs
Let T > 0 and d, du, de E N. Let D C Rd be a bounded domain. Let Hu : - Rd,) and 14g [f : - R be two Hilbert spaces of functions from D to Rd and IR respectively. We consider a large class of SPDEs of the following type
du = (Lu, F(we)) di + G(u))dW,
where W, is either an infinite dimensional Q- Q-Wiener process [27, Def. 10.6] or a cylindrical Wiener process [11, Def. 3.54] with values in Hgg F' H. + H. and G H. - L(Hg, H.)) are two continuous operators, L(Hg, H.) is the space bounded linear operators from 7g to Hu. and L is a linear differential operator generating a semigroup' exE Hy + Hu. For further details on Wiener processes see Appendix A.2, and for primer semigroup theory see Hairer [11, Section 4]. A function u : [0, T + H. is said to be a mild solution the SPDE (2) if for any E [0, I` it satisfies
U1 el + S elt Fltt))FFFudddd + S' cltt
where the second integral a stochastic integral interpreted in the Itô sense [11, Def. 3.57]. Thus, an SPDE can be informally thought of as an SDE with values in the functional space H., and driven by an infinite dimensional Brownian motion W. Assuming global Lipschitz regularity on F and G, mild solution u to (2) exists and is unique [11, Thm. 6.4), at least for short times.
We follow Friz and Hairer [9] and consider regularization W = W of the driving noise W with mollifier?, where * means convolution. As done in Kidger et al. [18] for Neural CDEs, we can rewrite the mild solution of the mollified version of eq. (2) as the following randomly forced PDE
Ut = f + S/ olt He(w))dd He(w.) = F(v)) + G(u.).r.
(3)
where & = W. H. = (dd,).dd.) and Hg = (DD Rdc). We will refer to & as white noise W cylindrical Wiener process and coloured noise if W is Q-Wiener process.
In view of machine learning applications, one should think of W continuous space-time embed- ding of an underlying spatiotemporal data stream. In this paper we are only going to consider W to be a sample path from Wiener process, we emphasise that the Neural SPDE model extends, in principle, beyond the scope of SPDEs and could be used for example to process videos in computer vision applications, which we leave as future work. Next we introduce the Neural SPDE model.
A strongly continuous semigroup S on H.. {S, : Ha + Ha.1oo with the properties that: 1) So = Id, the identity operator on Hr. 2) St S, = St+z, for any 8, > 0, and 3) the function + S,u continuous from [0, T] Hu. for any u H..
family of bounded linear operators S
AA mollifier is smooth function on Rd+1 that is: 1) compactly supported, 2) fgai spx)dx = 1, and 3) lim. 10" (z) = lim e -dd-1) (dx/e) = ((x). where d is the Diract delta function and the limit must be understood in the space of Schwartz distributions.
3

---

oodii = (5 (8711)855.0 (adl + +
Fixed Point Solver:
co R4. [0.0] - L2(2) R44)
2: [0.7] - L2
Latent Space
ODE Solver: 4= 4=F7 (v)
4 %
Av, 3 A.Hag (11))))) vo (io
LR: RA
Readout Tl. R4 -a R4
(40
20
R4,
[0,07 - (2(2)
10.71 (2),, R%)
Physical Space
Solution Operator
Space
M
Space
Loss wtar
Time
Time
727
A signal & and an initial condition du are observed on a, possibly irregular, spatiotemporal grid. to is lifted to 2o living a latent space. solution an SPDE latent space driven by the signal & and initial condition is obtained using either fixed point solver ODE solver. The solution then projected back the physical space by linear readout. The output signal u is then fed to pathwise loss function.
Figure I:
3
Neural SPDEs
For large class differential operators L, the action of the semigroup eLC can be written as an integral against kernel function K, D D Rduxdu
xdu such that
(e h))) = (((zz)))(
for any Hu, any x DD and E [0. and where j4r Borel measure on D. As in Kovachki al. [21], here take μr be the Lebesgue measure on Rdd but other choices can be made, for example to incorporate prior information. We assume that I. is stationary so that eq. (3) can be rewritten in terms of the spatial convolution
Ut = Kr * uo 1'' Ke-s Hg(u,)ds.
For large class of SPDEs of the form (2), both F and G are local operators acting function e Hu. In other words, the evaluations (x))(x) and G(h) at any point æ e only depend h(*), and not the evaluation h(y) at some other point y D in the neighbourhood of æ.
3.1 The model
Let Hi. (dd) for some latent space dimension dn du, Let
Lg IRd Rdh, Fe RR Rdh, Go : Rd Rd x II Rdd + Rdd be four feedforward neural networks. For any differentiable control E [0,7] Hg. define the map Ho. Hh + Hy so that for any h E Hu. x E D, E [0, He.e(h)(x) (h(x))) + Go(h()))
A Neural SPDE is defined as follows
[ Kr_s (2. u(xx) (x)) (4)
(x) To (uo())), 2 =
We note that globally Lipschitz conditions can be imposed by using ReLU tanh activation functions in the neural networks Fg and Gg. In sees. 3.2 and 3.3 we propose two distinct algorithms to evaluate the Neural SPDE model (4) which are based on two different parameterization of the kernel K.

---

3.2 Evaluating the model by solving system of ODEs
Levereraging the convolution theorem, we can rewrite the integral in eq. (4) follows
2t == (F(K))(( (FGGFEOO ((Lo)))((
where F,F+1 are the d-dimensional Fourier transform (FT) and its inverse (see Def. A.1). If one further assumes that is a polynomial differential operator, can be shown that there exists a map A: Cd + Cdaxds such that (v) = e*A(2) (see appendix B. 1.1 for a derivation). It follows that
= F (e-A(())((z)) + / ((He,))A((( =F
where 24 Cd + Cd is the solution the following ODE
2/ vo + { Av, + F(Ho, (F (v.))).
Hence, 2, can be obtained by applying the inverse FT to the output of an ODE solver [0, with initial condition F(zu), vector field Vo. == A F Ho.e F-1, i.e.
(ODES)lve(F(z)) Vo,c (0,
24
This approach naturally be seen "neural version" of the classical spectral Galerkin method for SPDEs as described in Appendix A.3.
We note that this numerical evaluation of the Neural SPDE model (4) allows to inherit memory- efficient adjoint-bun -based backpropagation capabilities as in the evaluation of a Neural CDE, For further details adjoint-basee based backpropagation refer the reader to Chen et al. [5], Kidger [17].
3.3 Evaluating the model by solving fixed point problem
In our second approach of model evaluation, we make use of three different versions of the FT: the time-only FT Fi and its inverse Fi-, the space-only FT Fa and its inverse Fa and the space-time FT Fa+1 and its inverse Fa41 (see Def. A.1 for details). Denoting by the space-time convolution, the integral in eq. (4) can be rewritten as
2 Kg zu (K
120))o,(2.));
where 120 is the indicator function restricting the temporal domain to the positive real line. Using again the convolution theorem we obtain
2, FT (Fa(K,)Faaz))) + F (Fasi (K) (4) (120Ho,
where all multiplications are vetrix-vector multiplications. Using the trick introduced [25], one parameterize Fa+1 (K) directly in Fourier space complex tensor B, that the solution of eq. (4) can be obtained by solving the fixed point problem = == =0).e(2) with
(x), = = (FT (B), + F (BFa+ 177441(120ll (1
and where used the fact that Fa(K.) (Fa This can be solved numerically using classical root-finding schemes (e.g. by Picard's iteration)
z FixedPointSolve(2a &o.5).
Analogously to adjoint-basee based backpropagation for the evaluation approach mentioned in Sec. 3.2, there mechanism that leverages the implicit fincction theorem allowing to backpropagate through the operations of fixed point solver in a memory-efficient way. See Bai et al. [2] for further details.
We note that the FT's are numerically approximated using the discrete Fourier transform (DFT) and selecting a maximum number of frequency modes (see Appendix A.1 for details).
'The DFT approximates the Fourier series expansion truncated at a maximum number of modes. This allows to specify the shape of the two complex tensors A (Kiux x Ad x da. x da) in See. 3.2 and B (Ad) X kd+1 x da. di, Sec. 3.3. The kf treated as hyperparameters of the model.
5

---

3.4 Space-time resolution-invariance
As depicted in Fig. 1, the input to Neural SPDE corresponds (possibly irregularly sampled) time-indexed sequence of (possibly partially-obseree spatial observations recorded on space-time grid. The data is then interpolated into a continuous spatiotemporal signal ç and initial condition No. By construction, a Neural SPDE operates in continuous time and space on the tuple of functions (uo, {) and produces a spatiotemporal response u, which also continuous in space and time; the function u can then be evaluated at an arbitrary space-time resolution, possibly different from the one used during training. In the next section we will demonstrate empirically that even trained on coarser resolution, a Neural SPDE can be evaluated on a finer resolution without sacrificing performance, a property known as zero-shot super-resolation.
3.5 Comparison of the two evaluation methods
Number of parameters For fixed dimension da of the latent space, the majority of trainable parameters in the ODE parameterization lies in the complex tensor A, which consists of - dz parameters, where each kmax is the maximum number selected frequencies in the Fourier domain. Regarding the Fixed Point parameterization, the bulk of the parameters is in the complex tensor B, which consists of k- .. 1 max dr,, where the additional frequency is due to the fact that we are taking the FFT in space-time rather than just in space done in the ODE Solver approach. Hence, the latter would in principle have the advantage of using a lower number of parameters than the former; however, to achieve similar performance, we found that the dimensionality of the latent space has to be roughly 20 times higher, which offsets the aforementioned advantage.
Time complexities The time complexity of the ODE Solver approach O log(Nz)), where N is the number of time steps taken by the ODE solver and N, is the number of points on the spatial grid, while the complexity of the Fixed Point approach is O N,,N,,, (log( (N,) + log(N.)) where is the number of Picard iterations and N₂ is the number of points on the temporal grid. In our experiments we choose N₂ R N₂ and IN, R N, making the two complexities comparable.
Speed of computation We found that the ODE approach is approximately 10 times slower than the Fixed Point approach. We believe this is largely an implementation issue of the tor corcffiffe library, while the FFT is highly optimised transform in Pytorch.
3.6 Considerations about convergence
We follow Friz and Hairer [9] and consider regularization W = W of the driving noise W with a compactly supported smooth mollifier p², It is a classical result (Wong-Zakai [35]) from rough path theory [10, 30] that, for the case of SDEs, the sequence of random ODEs driven by the mollification of Brownian motion converges in probability a limiting process that does not depend on the choice of mollifier and agrees with the Stratonovich solution of the SDE. Furthermore, the solution map (ug, W) + continuous in an appropriate rough path topology. This result nicely extends to the setting of SPDEs driven by finite dimensional noise [9, Thm. 1.3]: if u" denotes the random PDE solutions driven by W dt (instead of odW)), then u converges in probability to limiting process corresponding to the Stratonovich solution of the SPDE. In our setting though, the driving noise is infinite dimensional and the resulting integral cannot be interpreted in the Stratonovich sense because otherwise the corresponding Itô- vd-SStrttonvvch correction would be infinite. Nonetheless, Hairer and Pardoux [14, Thm. 1.1] show that, in the case of the heat operator and under appropriate renormalization and drift correction, the random PDE solution uf converges in probability to the Itô solution of the SPDE, and that the solution map is continuous in an appropriate regularity structures topology. We note that extending this result to a generic differential operators would require a similarly rigorous proof, which goes beyond the scope of this article and that we leave future work.
4 Experiments
In this section, we run experiments on three semilinear SPDEs: the stochastic Ginzburg-Landan cquation in 4.1, the stochastic Korteweg-De De Vries cquation in 4.2, and the stochastic Savier-Stoker equations in 4.3. We note that although the assumption of globally Lipschitz vector fields might be

---

violated for the following SPDEs, well-posedness (i.e. existence of global solutions) can be shown using equation-specifi arguments. We consider three supervised aperator-leaming settings:
My assuming the noise & is not observed;
ç - u. assuming the noise is observed, but the initial condition uo is fixed across samples;
(wo, <) - assuming the noise & is observed and Mo changes across samples.
We note that learning the operator Ug - of an SPDE without observing the driving noise & unavoidably yields poor results for all considered models only partial information about the system is provided as input. However, we find informative to include the performances obtained in this setting, as this provides sanity check that emphasizes the importance of the noise in all the experiments we consider in this paper. Moreover, the ability to process the initial condition uo on its own (in absence of noise) testifies that Neural SPDEs can also be used learn deterministic PDEs. We provide an example on the deterministie Navier-Stokes equations in Appendix B.5.
Neural CDE, Neural RDE, FNO and DeepONet [28, 29] will be the main benchmark models. In addition, we also propose an additional baseline Neural CNE-FNO, which is a hybrid model consisting of a Neural CDE where the drift is modelled by FNO and the diffusion by feedforward neural network. The motivation for using FNO to represent the drift comes from the universal approximation properties of FNOs studied in Kovachki al. [21, Thm. 4].
An interesting line of work to tackle SPDE-learning is provided in [7, 16]. The authors construct a set of features from the pair (uo, <) following the definition of model from the theory of regularity structures [13]. They then perform linear [7] and nonlinear [16] regression from these features to single time point. Therefore, these models would have to be retrained for any new prediction. In addition, both [7. 16] assume knowledge of the differential operator governing the dynamics, while Neural SPDE learns representation of L via the parametrization of the associated kernel. For these reasons these recent models are not included in our benchmark.
the solution of the SPDE
For all the experiments, the loss function is the relative pathwise 1,2 error. The hyper-paramedd for all the models are selected by grid-search (see Appendix B.2 for further experimental details). Experiments are run Tesla P100 NVIDIA GPU. The code for the experiments is provided in the supplementary material. Additional experiments may found in Appendix B.
4.1 Stochastic Ginzburg-Landal equation
We start with the stochastic Ginzburg-Lande equation, reaction diffusion equation in 1D given by u(t, utt,, 1), u(0, x)) uo(x), (t, j0, [0. 1].
Ô Au 3u nd c.
This equation is also known the Allen-Cahi equation in 1-dimension and is used for modeling various physical phenomena like superconductivity [34]. Here & denotes space-time white noise with sample paths generated using classical sampling schemes for Wiener processes detailed in A.2.
Table 1: Ginzburg-Landau Relative L2 errrr on the test set. indicates that the model not applicable.
N = 1 000
N = 10000
Model
20
& +
(uo. {) H
Ug
++
(uo, {) H
NCDE
0.112
0.127
0.056
0.072
NRDE
X
0.129
0.150
X
0.070
0.083
NCDE-FNO
X
0.071
0.066
0.066
0.069
DeepONet
0.130
0.126
0.126
0.061
FNO
0.128
0.032
X
0.126
0.027
NSPDE (Ours)
0.128
0.009
0.012
0.126
0.006
0.006
We consider two data-regimes: low data regime where the total number of training observations N = 1 000, and large data regime where N = 10000. In both cases, the response paths are generated by solving the SPDE along each sample path of the noise using finite difference
7

---

scheme described in Appendix A.3 using 128 evenly distanced points in space and time and step size At 1001 Following the same setup as in Chevyrev al. [7, eq. (3.6)), we solve the SPDE until T = 0.05 resulting in 50 time points. We choose as initial condition (z(z) = xx1 = kn(x), with (=x) do + -k=-10 d1 [k[2) sin (kTz) where Og ~ VV(00 1). We take A. 0 and = 0.1 to generate a dataset where the initial data is either fixed or varies across samples. We provide extra experiments on this SPDE for larger time horizons T and multiplicative forcing in Appendix B.3. We report the results in Table 1. The Neural SPDE model (NSPDE) yields the lowest relative error for all tasks, reaching one order of magnitude improvement on the main task (uo, {) + u in the large data regime compared to all the applicable benchmark models (NCDE, NRDE, NCDE-FNO), In all settings, even with a limited amount of training samples (N 1 000), NSPDE achieves 1% error rate, and marginally improves to < 1% error when N = 10 000.
4.2 Stochastic Korteweg-De De Vries equation
Next, we consider the stochastic Korteweg-De Vries (KdV) equation, a higher order SPDE given by 0,u y2uu 6ud, u E, u(t, 0) u(t, 1), x0 = uo(2)) (t,z) E [0, T] [0, 1].
This equation is used describe the propagation of nonlinear waves at the surface of fluid subject to random perturbations (another wave equation is studied in Appendix B.4). We refer the reader Wazwaz [36] for an overview on the KdV equation and its relations to solitary waves. The stochastic forcing given by g W for W being a partial sum approximation of Q- Q-Wiener process as per Example 10.8 in Lord et al. [27] with Ny j j and (z(x) = sin (jTæ) (see eq. (6) in Appendix A.2). Taking small æ > guarantees that We is twice differentiable in space for every t 2 0. To generate the datasets, we solve the SPDE with = 0.1 until T = 0.5.
Table 2: Stochastic KdV. Relative L2 error on the test set.
The stochastic forcing is simulated using 128 evenly distanced points in space and a time step Afrer 10-3 We then approxi- mate realizations of the solution of the KdV equation using time step A/ 10-22 un- til T = 0.5. Here, the initial condition is given by un(x) where is defined as in Sec. 4.1. Similarly See. 4.1 we either take x to generate datasets where the initial condi- tion is either fixed or varies across samples. Each dataset consists of N = 1 000 train- ing observations. As reported in Table 2a, Neural SPDEs outperforms the second best model FNO by full order of magnitude in the task { - and the second best model NCDE-FNO by almost two orders of mag- nitude in the task (wo.E) {) + We also perform the same tasks for a larger time horizon T 1 and report the results of comparison against FNO in Table 2b.
The symbol indicates that the model is not applicable.
(a) N 1 000 and ' 0.5.
Model
Ug + u
E ++ u
(uo: {) ++ u
NCDE NRDE
X
0.464
0.466
sin (2xx) + kz)(z).
0.497
0,503
NCDE-FNO
x
0.126
0.259
DeepONet
0.874
0.235
x
FNO
0.835
0.079
NSPDE (Ours)
0.832
0.004
0.008
(b) N = 1 000 and T T= 1.
Model
Ug ++ a
{ ++ u
(up. {) +
FNO
0.913
0.112
X
NSPDE (Ours)
0.904
0.009
0.012
(c) Subsampling (N 1 and T 0.5).
Partial observations Neural SPDEs able to process signals that are irregularly sampled both in space and in time by in- terpolating between observations. Yet, the ability of model to process irregular data does not guarantee its robustness when some observations are dropped. Robust- ness can only be guaranteed if the signal is regular enough so that replacing dropped observations by interpolation results new signal that is close, some suitable norm, to the original signal. To illustrate this point, we run two additional experiments where we drop uniformly at random 1) 10% of the data in time and 2) 50% of the data in space. As it can be observed in Table 2c, the performance of Neural SPDE remains roughly unchanged when data is dropped in space but decreases when data
Subsampling rates
| u
(uo: {)
Time 0%
Space: 0%
0.004
0.008
Time 10%
Space: 0%
0.076
0.059
Time 0%
Space 50%
0.005
0.008
8

---

is dropped in time, which is to be expected since the driving signal is a Q- Q-Wiener process, which is rough in time, but smoother in space. We also note that ensure good upproximation of the FT by the FFT. the interpolation must translate the irregular data to (possibly finer) regular grid.
4.3 Stochastic Navier-Stokes equations in 2D
Finally, we consider the vorticity form of the Navier-Stokes equations for an incompressible flow
0ww vAw
Vw + f of,
w(0,x) wo(x), (t,x) E [0, V x u. These
[0, 1]? (5)
where w is the unique divergence free (V . u 0) velocity field such that equations describe the motion an incompressible fluid with viscosity y subject to external forees [34]. The deterministic forcing f. defined as in Li et al. [25], is a function of space only. The stochastic forcing & is given by & = W for W being a Q-Wiener process which is colored in space and rescaled by a 0.05 (see Appendix A.2). The initial condition is generated according to wo (0, ( (- À 491)-3 with periodic boundary conditions. The viscosity is set 10-4
5000
t 1625
2730
3875
t-50000
8
50
50
30
50
Glannd
25
25
25
50
50
30
50
0
50
anscuaton podlectonn
50-
50
50-
30-
50-
25-
25-
25
10
50
50
30
Figure 2: Top panel: Solution of the vorticity equation for one realisation of the stochastic forcing between the 500 and the 000 time steps. Bottom panel: Predictions with the Neural SPDE model given the initial condition at the 500" time step and the forcing hetween the 500 and the 000 time steps. The model is trained 16 16 mesh and evaluated 64 x 64 mesh.
For each realization of the Q-Wiener process (sampled according to the scheme in Appendix A.2) we solve eq. (5) with pseudo-spectral solver described Appendix A.3, where time is advanced with a Frank-NNcolson update. We solve the SPDE on 64 x 64 mesh in space and use a time step of size 10-3 For the tasks Mg -> u and E r+ u, we generate the datasets by solving the SPDE up to time T = and downsample the trajectories by factor of 10 in time (resulting in 100 time steps) and in space (resulting
16 16 spatial resolution). The number of training samples is N ==11 000. To generate the training set for the task (uo, {) + u, generate 10 long trajectories of 15 000 steps each up to time 7 15. We partition each trajectory into consecutive sub-trajectories of 500 time-steps using a rolling window. This yields total of 2 000 input-output pairs. We split the used GPU.
data into shorter sequences 500 time steps so that one batch fits memory
As shown in Table 3. Neural SPDEs marginally outperforms FNO on the task Ç + u, but with significantly larger gap on the task (ug, {) + u from the sec- ond best model NCDE-FNO. Fig. 2 indi- cates that our model is capable zero-shot super-resolution in space-time, achieving good performance even when evaluated on larger time horizon and pled spatial grid. Finally, we report in Table 4 some time statisties indicat- ing that NSPDEs can be up to 3 orders of magnitude faster than traditional numerical solvers.
Table 3: Stochastic Navier-Stokes. Relative L.2 the test set. The symbol x indicates that the model is not applica- blc, and indicates that the model does fit in memory.
Model NCDE NRDE
uo u ++ u
(uo, {) ->
0.366
0.843
upsam-
NCDE-FNO
0.326
0.178
DeepONet
0.432
0.348
FNO
0.188
0.039
X
NSPDE (Ours)
0.155
0.034
0.049

---

5 Conclusion
We introduced Neural SPDEs, a model capable of learning solu- tion operators of PDEs with (possibly stochastic) forcing from partially observed data. Our model provides an extension to two classes of physics-inspired models. It extends Neural CDEs in that is iisolution-invarian both in space and in time, and it extends Neural Operators as it can be used to learn solution operators of SPDEs depending simultaneously on the initial condition and driving noise. We performed extensive experi- ments illustrating how the model achieves superior performance while requiring lower amount of training data compared to other models, and its evaluation is up to 3 orders of magnitude faster than traditional numerical solvers.
Table 4: Ratio of the inference time of the trained NSPDE over the runtime of the numerical solver. We the same spatiotemporal discretization for both NSPDE and the numerical solver.
Dataset
Speedup
Ginzburg-Lande Korteweg-DD Vries
59
80
Navier-Stokes
300
Limitations and future work Similarly to other neural operator models, parameterising the kernel in Fourier space is by means the only available option; other parameterisations could mitigate some of the disadvantages of the FFT (irregular grids, aliasing effect ...), see for example [23]. A question we leave to future work is how to construct a discrepancy between probability measures supported on spatiotemporal signals, generalizing for example the signature kernel MMD in [33]. Neural SPDEs paired with such discrepancy would allow the design of new generative models for spatiotemporal signals. Another research direction will be to assess whether Neural SPDEs can be used in computer vision process videos at arbitrary resolution.
Acknowledgments and Disclosure of Funding
This project was supported by G-Research and by DataSig under the grant EP/S026347/1.
References
[1] Sh A Alimov, RR Ashurov, and AK Pulatov. Multiple fourier series and fourier integrals. In Commutative Harmonic Analysis IV, pages 1-95. Springer, 1992.
[2] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32:690-701, 2019.
[3] Alexis Bellot and Mihaela Van Der Schaar. Policy analysis using synthetic controls in continuous-time. In International Conference Machine Learning, pages 759-768. PMLR, 2021.
[4] William L Briggs and Van Emden Henson. The DFT: an owner's manual for the discrete Fourier transform. SIAM, 1995.
[5] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary dif- ferential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6572-6583, 2018.
[6] Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911-917 1995.
[7] Ilya Chevyrev, Andris Gerasimovics, and Hendrik Weber. Feature engineering with regularity structures. arXiv preprint arXiv:2 2021.
[8] James W Cooley and John W Tukey. An algorithm for the machine calculation of complex fourier series. Mathematics of computation, 19900)297-301, 1965.
[9] Peter K Friz and Martin Hairer. A course on rough paths. Springer, 2020.
[10] Massimiliano Gubinelli. Controlling rough paths. Journal of Functional Analysis, 216(1): 86-140, 2004.
10

---

[11] Martin Hairer. An introduction to stochastic pdes. arXiv preprint arXiv:0907.4178, 2009.
[12] Martin Hairer. Solving the kpz equation. Annals of mathematics, pages 559-664, 2013.
[13] Martin Hairer. A theory of regularity structures. Inventiones mathematicae, 198(2):299-004,
2014.
[14] Martin Hairer and Étienne Pardoux. A wong-zakai theorem for stochastic pdes. Journal of the Mathemalical Society of Japan, 67(4): 2015.
[15] Helge Holden, Bernt Øksendal, Jan Ubøe, and Tusheng Zhang. Stochastic partial differential equations. In Stochastic partial differential equations, pages 141-191. Springer, 1996.
[16] Peiyan Hu, Qi Meng, Bingguang Chen, Shiqi Gong, Yue Wang, Wei Chen, Rongchan Zhu, Zhi-Ming Ma, and Tie- Yan Liu. Neural operator with regularity structure for modeling dynamics driven by spdes. arXiv preprint arXiv:2204.06255, 2022.
[17] Patrick Kidger. On neural differential equations. arXiv preprint arXiv:2202.02433. 2022.
[18] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. arXiv preprint arXiv:2005.08926 2020.
[19] Patrick Kidger, James Foster, Xuechen Li, and Terry Lyons. Efficient and accurate gradients for neural sdes, arXiv preprint wrXiv:2105.13499 2021.
[20] Patrick Kidger, James Foster, Xuechen Li, Harald Oberbauser, and Terry Lyons. Neural sdes as infiite-dimennionaal gans. arXiv preprint arXiv:2102.03365, 2021.
[21] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces, arXiv preprint (xXi:2200800888) 2021.
[22] Xuechen Li, King-Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradi- ents for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pages 3870-3882. PMLR, 2020.
[23] Zongyi Li, Nikola Kovachki, Kamyar Azizzadeneshelii Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.0344 034455, 2020.
[24] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33, 2020.
[25] Zongyi Li, Nikola Borislavov Kovachki, Kamyar Arizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar, al. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2020.
[26] Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. Neural sde: Stabilizing neural ode networks with stochastic noise. arXiv preprint arXiv: 090600235, 2019.
[27] Gabriel J Lord, Catherine E Powell, and Tony Shardlow. An introduction computational stochastic PDEs, volume 50. Cambridge University Press, 2014.
[28] Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218 229, 2021.
[29] Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393:114778, 2022.
[30] Terry Lyons. Differential equations driven by rough signals. Revista Matemática Iberoameri- cana, 14(2):215-310, 1998.

---

Communications in [31] Remigijus Mikulevicius and Boris L Rozovskii. Stochastic savier equations for turbulent flows. SIAM Journal on Mathematical Analysis, 35(5):1250-1310, 2004.
[32] James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In International Conference on Machine Leurning, pages 7829-7838. PMLR, 2021.
[33] Cristopher Salvi, Thomas Cass, James Foster, Terry Lyons, and Weixin Yang. The signature kernel is the solution of goursat pde. SIAM Journal on Mathematics of Data Science, 3(3): 873-899, 2021.
[34] Roger Temam. Infinite- lnfinite-dimensional dynamical systems in mechanics and physics, volume 68. Springer Science Business Media, 2012.
[35] Krystyna Twardowska. Wong-zakai approximations for stochastic differential equations. Acta Applicandae Mathematica, 43(3):317-359, 1996.
[36] Abdul-Majid Wazwaz. Solitary waves theory. In Partial Differential Equations and Solitary Waves Theory, pages 479-502. Springer, 2009.
[37] Weinan. A proposal on machine learning via dynamical systems. Mathematics and Staristics, 1(5):1-11, 2017.
[38] William Henry Young. Vi. on the general theory integration. Philosophical Transactions the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 204(372-386)221 252, 1905.
12


Reviews:
[{"review_id": "fTXoxHcpL4c", "reviewer": "Reviewer_YFMX", "summary": "The paper introduces the neural stochastic PDE model to learn solution operators of PDEs, extending previous frameworks of neural CDEs and neural operators. The authors introduce two ways of evaluating the model, based on either solving a system of ODEs, or reformulating to a fixed point problem. The authors perform experiments on multiple stochastic PDE problems, showing that their neural stochastic PDE model is superior over previous baseline models in learning such dynamics. ", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Unfortunately, the paper is well outside of my expertise (due to a potential mismatch in the system), and I really do not have the relevant PDE background to analyze the techniques used in the paper. \n\nAside from the technical parts, I find the structure of the paper pretty clear - the authors provided good motivations for their method, and derived the theory for evaluating the model in an organized fashion. It would be nicer if the authors can summarize their final algorithm in an algorithm block, which would make it much easier for readers to understand the key steps. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aP8x7AYsVJ", "reviewer": "Reviewer_98nh", "summary": "- The paper introduces Neural Stochastic Partial Differential Equations (NSPDEs) for learning solution operators for SPDEs while being resolution invariant. \n- Two methods of performing inference for NSPDEs are proposed: (1) A method using Fourier Transformations to reduce the NSPDE to a Neural ODE, which can then be solved with existing techniques; (2) a method based on framing the NSPDE as a fixed point problem allowing the use of fixed-point solvers. \n- The model is evaluated on well-known SPDEs and shown to outperform various Neural Differential Equation models as well as the Neural Operators method. ", "questions": "- See all my questions from the weaknesses section above. \n- I am a bit confused about how $\\xi(t, x)$ is supplied to the Neural Operator models and all models more broadly when it is observed. \n   - First of all, what do we assume it is observed more precisely? Do we have full access to the function $\\xi(t, x)$ and we can sample it at any point or do we have access to some samples from this function for certain $(t, x)$ pairs. \n   - In the case of FNO, in its original paper, the model takes as input a function over the domain $\\mathcal{D}$ and outputs another function over the same domain. As far as I understand, in the setting when $u_0$ is fixed, the input function to the FNO model becomes $\\xi(t, x)$. So you are supplying to, FNO samples from $\\xi(t, x)$ on a regular grid over $[0, T] \\times \\mathcal{D}$. Is that correct? \n   - I see that the Neural Operator models were not evaluated in the setting where the input is $(u_0, \\xi)$. Couldn't this input be supplyed as function combining these two functions? For instance, the new function could concatenate the two over the spatial domain at $t = 0$ and concatenate $\\xi$ with some dummy values for $t > 0$ (or something else along these lines)? \n   ", "limitations": "TLDR: No \n- For limitations, the checklist mentions Section 2 (which is a background section), while Section 5 discusses future directions (not limitations). So I believe limitations have not been properly addressed. \n- Negative societal impacts are not discussed either. I believe this part is treated too superficially despite that fact that PDEs show up everywhere in mechanics and these sort of models could be applied for all sorts of military applications and what not. Therefore, I would invite the authors to put more thought about the potentially dangerous applications of these models.  ", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Stengths\n\n- The paper proposes a well-executed instantiation of Neural SPDEs continuing the recent research direction of combining deep learning models with various types of differential equations. \n- The two proposed methods for evaluating the NSPDE model provide some degree of choice/flexibility for practitioners. \n- The emprical evaluation shows that the proposed model performs well at predicting the solution of SPDEs. \n\n# Weaknesses \n\n- SPDEs are notoriously a very technical subject, however, I believe this is not a justification for the fact that the background section fails (and perhaps does not even attempt) to find the right level of technicality for a machine learning audience. Overall, the background section is not self-contained at all: \n     - The reader is invited to consult a primer on semigroup theory. But no semigroup theory is actually needed to read the main content of the paper. The word semigroup is used only to say that $\\mathcal{L}$ is a differential operator generating a semigroup $e^{t\\mathcal{L}}$. Why not simply say $\\mathcal{L}$ induces a map $e^{t\\mathcal{L}}$ and refer to it as a map. If you really want to mention it's a semigroup, you can add a footnote and even define what a semigroup is in that footnote (it's a very short definition). \n     - Line 115 mentions a \"mollifier\" without ever explaining what that is. Also, what does $\\delta^\\epsilon * W$ mean? Is that a convolution? \n     - This is followed by a very technical discussion in Lines 116-130 that is of little consequence for understanding the rest of the paper. This discussion could be perhaps moved to the appendix. The space of that paragraph could be saved for further explaining other concepts or adding some figures that help explaining the main concepts. \n     - For Wiener processes, the paper simply references definitions from papers/books. I understand these have been defined properly in the appendix, but I think a short definition or at least an intuitive definition would have been possible in the main text given they are the essence of SPDEs. This would make the main text more self-contained. \n- There is no comparison between the two evaluation methods. First, what is their computational complexity and what are their theoretical advantages and drawbacks? Second, how do they compare in practice / empirically? None of these two types of questions is addressed and from reading the paper, it is unclear which one of the two one should use in various settings. Instead, a minimal set of guidelines backed up by theory and experiments could be provided. \n- None of the tables contain error bars. I see that the checklist says this is because \"this would be too expensive for the baseline models NCDE, NRDE and NCDE-FNO\". First, the neural operator models do not contain error bars either despite being faster. Second, this claim is not really backed up by data. Could you report the training and evaluation time of those models that made evaluation across multiple seeds difficult/impossible? Realistically speaking, if there was enough compute to run these experiment once, then running them 3 or 5 times should be doable (unless it takes weeks or months?). At least on the systems with one spatial dimensions, I don't think that should be the case. \n- One of the main motivations of the model is that it generalises NCDEs and it can also be applied to irregularly sampled and partially observed data. Correct me if I am wrong, but I believe that none of the experiments in the evaluation section actually consider such a context. Here, I would have expected something like in Figure 1, where data is observed at some sequence of times $t_1, \\ldots, t_n$ and the samples might be missing some channels. \n\n======== EDIT =========\n\nRaised my score to Accept after discussion. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Bh4yx1Hp5zC", "reviewer": "Reviewer_QRvu", "summary": "This paper introduces a method to elucidate noise-conditioned spatiotemporal dynamics via the framework of Stochastic Partial Differential Equations (SPDEs). It consists in learning neural-parameterized SPDEs to fit the observed phenomenon thanks to the adaptation of two standard Fourier-based resolution methods used to handle the neural SPDEs (an ODE solver in the Fourier space and a fixed-point method). This allows the proposed model to work on arbitrary space and time resolutions.\n\nThe model is then evaluated against several neural operators and neural CDE baselines to solve standard SPDEs in three contexts: based on the initial condition only, conditioned on the intrinsic noise information, and both at the same time. The model especially shows significant improvement w.r.t. the considered baselines on the two latter settings.", "questions": " - Could the authors comment in the ability of the model to handle partially observed stochasticity and the applicability of the considered setting to real-world phenomena?\n - Can the experimental setting be clarified w.r.t. my concerns described above?", "limitations": "The authors adequately addressed the limitations of their work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThe paper presents, to my knowledge, a **well-motivated, novel model** for a **relevant task**, since handling an underlying noise in the observed dynamics remains a challenge for neural PDE-solving methods. **Numerical results support the relevance** of the proposed model, with the latter being able to accurately predict the studied phenomena, unlike tested baselines. Its **flexibility** is also appreciated, with its two different versions, its ability to handle multiple output resolutions and its versatility in the experiments.\n\n### Weaknesses\n\nI have three main concerns about this paper that are significant enough to nuance my positive recommendation.\n\nFirstly, **the significance of both the model and its results remains unclear**. This is in part due to its SPDE inspiration: the model is specifically designed to integrate the complete noise information in its predictions, but this use case seems limited. Indeed, accessing this noise may only be possible in synthetic controlled settings such as the conducted experiments in the paper. Assessing whether the model is able to integrate partially observed noise would greatly benefit the paper in this regard. Nonetheless, given the underperformance of the model in the absence of noise information, I suspect that this may be challenging for the model as currently presented here.\n\nSecondly, I find that **the evaluation is lacking in some respects**.\n - Some baselines on neural operators might be missing, in particular improvements of FNO: for example, the works of Cao (2021) and Gupta et al. (2021) seem relevant. The evaluation of FNO is also questionable: why is it necessarily unable to handle the $(u_0, \\xi) \\rightarrow u$ case? Couldn't $u_0$ be appended to $\\xi$ at any $t$ as a supplementary information?\n - Moreover, the impact of the two proposed resolution methods is not evaluated: an ablation study (in terms of performance, efficiency, etc.) might highlight their respective merits w.r.t. other standard resolution methods. On this matter, to my understanding, only one of these alternatives (the fixed-point one) is used in the experiments.\n - Finally, I am unsure of the relevance of the task $u_0 \\rightarrow u$ in which models are trained and evaluated using a regression objective while the phenomenon is stochastic by nature.\n\nThirdly, **the writing and presentation of the method should be improved**. The problem is threefold.\n 1. The SPDEs are contextualized by a dense background description which is hard to follow for non-experts. Providing more intuitions and simplifying the notations could improve this point: for example, the simplification of lines 148-149 could be done earlier in the document, and examples accompanying the introduction of SPDEs would be welcome.\n 2. The presentation of the method in unclear on some points, notably because of the heavy notations but also because important information is scattered across the paper. In particular, Section 3 should clearly state the possible inputs and outputs of the model as well as the learnable parameters (for example, the fact that \"Neural SPDE learns a representation of $\\mathcal{G}$\" is only explicitly stated in Section 4, line 235).\n 3. The paper misses a further discussion of related work, either by adding a dedicated section or by specifying in e.g. Section 3 the precise differences with previous work. This would make the original contributions of the paper clearer.\n\nCao. Choose a Transformer: Fourier or Galerkin. NeurIPS 2021.\\\nGupta et al. Multiwavelet-based Operator Learning for Differential Equations. NeurIPS 2021.\n\n### Overall Sentiment\n\nDespite its flaws which are not prohibitive, I believe that this paper introduces an interesting method for the NeurIPS community with appealing performance. Therefore, I am inclined to recommend this paper for acceptance. I am willing to change my evaluation following the discussion phase with the authors and the other reviewers.\n\n**Post-rebuttal update:** The authors' response partially alleviated my concerns (cf. [my answer below](https://openreview.net/forum?id=OGM9dXemmq&noteId=8iHe2W6IV1B)). I consequently raise my score from 5 to 6.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept