Title: Monte Carlo Neural PDE Solver
Venue: NeurIPS 2023
Accepted: no
Keywords: Neural PDE Solver

Feynman-Kac Formula

AI for PDE
Abstract: Training neural PDE solver in an unsupervised manner is essential in scenarios with limited available or high-quality data. However, the performance and efficiency of existing methods are limited by the properties of numerical algorithms integrated during the training stage (like FDM and PSM), which require careful spatiotemporal discretization to obtain reasonable accuracy, especially in cases with high-frequency components and long periods. To overcome these limitations, we propose Monte Carlo Neural PDE Solver (MCNP Solver) for training unsupervised neural solvers via a Monte Carlo view, which regards macroscopic phenomena as ensembles of random particles. MCNP Solver naturally inherits the advantages of the Monte Carlo method (MCM), which is robust against spatial-temporal variations and can tolerate coarse time steps compared to other unsupervised methods. In practice, we develop one-step rollout and Fourier Interpolation techniques that help reduce computational costs or errors arising from time and space, respectively. Furthermore, we design a multi-scale framework to improve performance in long-time simulation tasks. In theory, we characterize the approximation error and robustness of the MCNP Solver on convection-diffusion equations. Numerical experiments on diffusion and Navier-Stokes equations demonstrate significant accuracy improvements compared to other unsupervised baselines in cases with highly variable fields and long-time simulation settings.

Abstract

Training neural PDE solver in an unsupervised manner is essential in scenarios with limited available or high-quality data. However, the performance and efficiency of existing methods are limited by the properties of numerical algorithms integrated during the training stage (like FDM and PSM), which require careful spatiotemporal discretization to obtain reasonable accuracy, especially in cases with high-frequency components and long periods. To overcome these limitations, we propose Monte Carlo Neural PDE Solver (MCNP Solver) for training unsupervised neural solvers via a Monte Carlo view, which regards macroscopic phenomena as ensembles of random particles. MCNP Solver naturally inherits the advantages of the Monte Carlo method (MCM), which is robust against spatial-temporal variations and can tolerate coarse time steps compared to other unsupervised methods. In practice, we develop one-step rollout and Fourier Interpolation techniques that help reduce computational costs or errors arising from time and space, respectively. Furthermore, we design a multi-scale framework to improve performance in long-time simulation tasks. In theory, we characterize the approximation error and robustness of the MCNP Solver on convection-diffusion equations. Numerical experiments on diffusion and Navier-Stokes equations demonstrate significant accuracy improvements compared to other unsupervised baselines in cases with highly variable fields and long-time simulation settings.

1 Introduction

Neural PDE solvers, which leverage neural networks as surrogate models to approximate the solutions of PDEs, are emerging as a new paradigm for simulating physical systems with the development of deep learning [33, 31, 23, 12]. Along this direction, several studies have proposed diverse network architectures for neural PDE solvers [30, 33, 5]. These solvers can be trained using supervised [33, 30] or unsupervised approaches [59, 54, 32], employing pre-generated data or PDE information to construct training targets, respectively. The unsupervised training approach is essential for AI-based PDE solvers, particularly in scenarios with limited available or high-quality data. To address this, some studies [54, 32] borrow techniques from classical numerical solvers to construct training targets. For instance, the low-rank decomposition network (LordNet) [54] and physics-informed neural operator (PINO) [32] integrate finite difference method (FDM) and pseudo-spectral methods (PSM) with neural networks during the training stage, respectively. However, FDM and PSM require fine meshes or time steps for stable simulations in general. Therefore, the performance and efficiency of these neural PDE solvers are also limited by the discretization of time and space, particularly when handling highly spatial-temporal variations and simulating physical systems over long periods.

To this end, we propose Monte Carlo Neural PDE Solver (MCNP Solver) for training neural solvers 35

from a Monte Carlo perspective, which regards macroscopic phenomena as ensembles of random 36

movements of microscopic particles [62]. Consequently, for a PDE system with probabilistic repre37

sentation, MCNP Solver constructs its solutions as training targets via Monte Carlo approximation. 38

Compared to other unsupervised neural solvers, such as LordNet [54] and PINO [32], MCNP Solver 39 naturally inherits the advantages of MCM. On the one hand, MCNP Solver can tolerate coarse 40 time steps [11, 39], thereby reducing training costs and accumulated errors arising from temporal 41 discretization. On the other hand, it can efficiently handle high-frequency spatial fields due to the 42 derivative-free property of MCM [37, 1]. Moreover, the boundary conditions are automatically 43 encoded into the stochastic process of particles [2, 34], eliminating the need to introduce extra loss 44 terms to satisfy such constraints. In addition to inheriting the benefits of MCM, we also develop 45 one-step rollout and Fourier Interpolation techniques to improve performance and efficiency from 46 the perspective of time and space. Furthermore, we design a multi-scale framework to improve the 47 accuracy and robustness of the MCNP Solver in long-time simulation tasks. 48

49

50

51

52

53

54

Compared to traditional MCM, MCNP Solver enjoys a significantly faster inference speed once trained. Additionally, traditional MCM requires sampling excess particles to achieve high-precision results, which can lead to severe computational and memory issues. However, thanks to the involvement of neural networks, the MCNP Solver does not necessitate sampling as many particles per epoch during training. According to our experimental observations, the model can converge as expected using gradient descent with only a few particles.

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

In this paper, we conduct in-depth analyses of the MCNP Solver's performance theoretically and experimentally. In summary, we make the following contributions:

1. We introduce MCNP Solver, a novel Monte Carlo-based unsupervised approach for training neural solvers applicable to PDE systems that allow probabilistic representation. Additionally, we develop several techniques to enhance performance and efficiency, such as Fourier Interpolation, one-step rollout, and multi-scale prediction.

2. Theoretically, we compare the approximation error and robustness of two kinds of neural PDE solvers concerning variations in spatial conditions, temporal discretization steps, and diffusive coefficients. Our theoretical results reveal that MCNP Solver is more robust against the spatialtemporal variants when solving convection-diffusion equations.

3. Our experiments on the diffusion and Navier-Stokes equation (NSE) show significant improvements in accuracy compared to other unsupervised neural solvers for simulating tasks with complex spatialtemporal variants and long-time simulation. Furthermore, the MCNP Solver can obtain comparable or even better results than supervised neural solvers.

2 Related Work

Neural PDE Solver Neural PDE solvers have been proposed to learn mappings between functional spaces, such as mapping a PDE's initial condition to its solution [33]. Works like DeepONet [33] and its variants [15, 52, 57, 26] encode the initial conditions and queried locations using branch and trunk networks, respectively. Additionally, Fourier Neural Operator (FNO) [31] and its variants [29, 45, 56] explore learning the operator in Fourier space, an efficient approach for handling different frequency components. Several studies have employed graph neural networks [30, 5] or transformers [6, 28] as the backbone models of neural solvers to adapt to complex geometries. However, these methods require the supervision of ground-truth data generated via accurate numerical solvers, which can be time-consuming in general. To this end, some studies aim to train the neural PDE solvers without the supervision of data [59, 32, 54, 19]. For example, [59] proposed PI-DeepONets, which utilize the PDE residuals to train DeepONets in an unsupervised way. Similarly, [19] proposed MetaAuto-Decoder, a meta-learning approach to learn families of PDEs in the unsupervised regime. Furthermore, LordNet [54] and PINO [32] borrow techniques from FDM and PSM, and utilize the corresponding residuals as training loss, respectively. Compared to these unsupervised methods, the MCNP Solver incorporates physics information through the Feynman-Kac law, representing a Monte Carlo perspective. This approach allows the solver to efficiently manage diffusion terms, exhibit robustness against spatial-temporal variants, and be suitable for long-time simulations.

Physics-Informed Neural Networks (PINNs) PINNs have been proposed to solve PDE systems by approximating solutions using the PDE residuals, which involve point-to-point mapping between spatial-temporal points and solution values. They are widely employed for solving forward or inverse problems [46, 8, 22, 66]. Recently, PINNs have made significant progress in addressing scientific problems based on PDEs, including NSEs [47, 20, 36], Schrödinger equations [18, 27], Allen Cahn

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

equations [38, 21], and more. Instead of constructing the loss function directly via the PDE residuals, some works utilize the probabilistic representation to train neural networks [17, 14, 63], which can efficiently handle high-dimensional or fractional PDEs [16, 50, 14, 49, 41]. Furthermore, some studies design loss functions based on other numerical methods, such as the finite volume method [4], finite element method [40, 42], and energy-based method [61]. Notably, the aforementioned PINN methods require retraining neural networks when encountering a PDE with new initial conditions, which can be time-consuming. Moreover, the studies [3, 48] consider PDE families with varying initial conditions while requiring corresponding conditions can be represented by a low-dimensional vector. In this paper, we aim to learn operators between functional spaces that can generalize to different PDE conditions over a distribution. When applying Feynman-Kac laws to this new scenario, we encounter several computational challenges arising from corresponding tasks, such as higher generalization requirements, long-time simulations, and the non-linearity of PDEs. Therefore, we propose Fourier Interpolation, one-step rollout, and multi-scale prediction to overcome these issues. More detailed discussions of these Feynman-Kac-based PINNs can be seen in Appendix D.

3 Methodology

3.1 Preliminary

In this paper, we consider the general convection-diffusion equation defined as follows:

where x ∈ Ω ⊂ R d and t denote the d -dimensional spatial variable and the time variable, respectively, β [ u ]( x , t ) ∈ R d is a vector-valued mapping from u to R d , κ ∈ R + is the diffusion parameter, and f ( x , t ) ∈ R denotes the force term. Many well-known PDEs, such as Burgers' equation, NSE, can be viewed as a special form of Eq. 1.

For such PDEs with the form as Eq. 1, the Feynman-Kac formula provides the relationship between the PDEs and corresponding probabilistic representation [43, 44, 16]. In detail, we can use the time inversion (i.e., ˜ u ( x , t ) = u ( x , T -t ) , ˜ f ( x , t ) = f ( x , T -t ) ) to the PDE as:

Applying the Feynman-Kac formula [35] to the terminal value problem Eq. 2, we have

where ˜ ξ s ∈ R d is a random process starting at x , and moving from 0 to T , which satisfies:

where B s is the d -dimensional standard Brownian motion. Applying time inversion t → T -t to Eq. 3 and letting ξ be the inversion of ˜ ξ , we have

Furthermore, apart from Eq. 1, some other PDEs can also be handled via the Feynman-Kac formula after certain processing, like wave equations [9] and spatially varying diffusion equations [51].

3.2 Monte Carlo Neural PDE Solver

Given a PDE with the form of Eq. 1 and a distribution of the initial conditions D 0 , the target of MCNP Solver is to learn a functional mapping G θ with parameter θ which can simulate the subsequent fields for all initial fields u 0 ∼ D 0 at time t ∈ [0 , T ] . In detail, the inputs and outputs of G θ are given as:

𝒖𝒖

𝟎𝟎

𝒕𝒕

+

𝜟𝜟𝒕𝒕

𝒕𝒕

𝒖𝒖

𝒕𝒕

𝒖𝒖

𝒕𝒕+𝜟𝜟𝒕𝒕

Fourier

Interpolation

�

𝒖𝒖

𝒕𝒕

MCNP

C

A

B

A: Random walk

B: Project

C: Query and Average

MCNP

where D [0 ,T ] denotes the joint distribution of the field after t = 0 . Unlike other supervised operator 126 learning algorithms [27, 33, 5], MCNP Solver aims to learn the operator in an unsupervised way, 127 i.e., only utilize the physics information provided by PDEs. To this end, MCNP Solver considers 128 training the solver via the relationship between u t and u t +∆ t (where 0 ≤ t < t +∆ t ≤ T ) derived 129 by the aforementioned probabilistic representation. Considering Eq. 5, an expected neural operator 130 G θ should satisfy the following equation: 131

132

where ξ s ( s ∈ [ t, t +∆ t ] ) is the inverse version of stochastic process in Eq. 4 as follows:

Regarding Eq. 7 as the optimization objective, the neural Monte Carlo loss can be written as follows: 133

134

135

136

137

138

139

140

141

142

143

144

Equipped with the loss function Eq. 9, we sample the initial states u 0 from D 0 and the time t from [0 , T ] each epoch, and the MCNP loss L MCNP is given as follows:

where λ ∈ R + is a hyper-parameter, and L init ( G θ | u 0 ) ≜ ∥G θ ( u 0 , 0) -u 0 ∥ 2 2 denotes the loss at t = 0 .

3.3 Implementation Details of MCNP Solver

In this section, we introduce some important implementation details for MCNP Solver. We illustrate the framework and training process of MCNP Solver in Fig. 1 and the overall algorithm in Appendix A. We design one-step rollout and Fourier Interpolation trick to reduce the computational cost and error from the perspectives of time and space, respectively. Moreover, we conduct the multi-scale framework to improve the long-time simulation ability of MCNP Solver.

Temporal Discretization and One-Step Rollout When simulating the stochastic process in Eq. 8, we utilize the classical Euler-Maruyama method [58] to approximate corresponding SDEs, .i.e,

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

The stochastic integral of the force f in Eq. 7 is approximated via the Euler method, which aligns with [16]. Unlike other Feynman-Kac-based methods [16, 41] conducting random walks in Eq. 8 with multi-steps, we utilize one-step rollout technique to simulate SDEs, i.e., at each t +∆ t , MCNP Solver generates new particles from x , and moves them back to t according to Eq. 11. The one-step rollout trick can enforce all ξ t +∆ t starting at x share the same β [ u ]( x , t +∆ t ) during the simulation of SDEs and thus, reduce the computational cost, especially for the scenario when the calculation cost of β is expensive. For instance, when the drift β term depends on solution u , we have to utilize MCNP Solver to calculate β accordingly. Moreover, in the NSE conducted in this paper, the mapping u → β represents the transformation from the vorticity field to the velocity field, which involves a numerical integration over an entire domain.

Random Walks and Boundary Conditions Eq. 3 and Eq. 4 describe the random walks driven by stochastic processes of corresponding PDEs. For PDEs with periodical boundary conditions, particles should be pulled back according to the periodical law when walking out of the domain Ω . For Dirichlet boundary conditions, the random walk of particles should stop once they reach the boundary. Compared to other unsupervised neural PDE solvers, MCNP Solver encodes the boundary conditions naturally into the random walks of particles and thus does not need additional soft constraints in the loss function. Furthermore, for PDEs with the fractional Laplacian -( -∆) α u , where α ∈ (0 , 2) , we only need to replace the Brownian motion with the α -stable Lévy process [24, 65, 64].

Spatial Discretization and Fourier Interpolation In this paper, we are interested in the evolution of PDEs at fixed grids { x p } P p =1 ∈ Ω . Consequently, the inputs and outputs of the solver G θ are solution values at P coordinate points. Please note that in Eq .7, the particles ξ t need to query the value of G θ ( u 0 , t ) when approximating G θ ( u 0 , t +∆ t ) . To efficiently obtain the querying results, we project the locations of particles ξ t to their nearest neighbor grids in practice. To reduce projection errors, we utilize the Fourier transform to interpolate the fields u t = G θ ( u 0 , t ) to the high-resolution one ˆ u t before the projection. It is worth mentioning that the Fourier Interpolation technique can help the neural solver achieve high-accuracy training signals without the calls of solvers on the high-resolution PDE fields, thereby reducing the training cost.

Multi-Scale Framework for Long-Time Simulation When handling tasks with long temporal intervals, we design the following multi-scale framework to make the training process more robust. In detail, we divide the long-time interval [0 , T ] into K coarse subintervals, i.e., { [ T k , T k +1 ] } K -1 k =0 , with T 0 = 0 , T K = T and T k +1 -T k = ∆ T . Accordingly, we adopt K neural solvers {G θ k } K -1 k =0 with independent parameter θ k to approximate the solution in [ T k , T k +1 ] , respectively. In the training stage, the loss function for long-time simulation is given as follows:

Here, u T k = G θ k -1 ( u T k -1 , ∆ T ) can be calculated recursively with u T 0 = u 0 , and L init ( G θ k | u T k ) ≜ ∥G θ k ( u T k , 0) -sg[ u T k ] ∥ 2 2 denotes the initialization loss for G θ k , where sg[ · ] denotes the stop-gradient operator. In the inference stage, when predicting the PDE field with the initialization u 0 at t = T k +∆ t (0 < ∆ t < ∆ T ) , we first rollout with coarse step ∆ T to obtain u T k , and then adopt the finer step to give the prediction of u t as G θ k ( u T k , ∆ t ) . Due to the independent parameterization and stop-gradient operator, the proposed multi-scale framework can prevent the prediction at time t ′ from producing harmful effects on the former time t < t ′ in the optimization stage. Our experiments reveal that it can improve the performance on long-time simulation tasks where the PDE fields change dramatically over time (e.g., turbulent flow simulation).

4 Theoretical Results

In this section, we study the theoretical properties of MCNP Solver when simulating the convectiondiffusion equation, and the proof can be seen in Appendix B. In detail, we consider the periodical convection-diffusion equation defined as follows:

191

192

193

194

195

196

197

198

199

200

201

202

In the following main theorem, we consider the error of one-step rollout targets provided in PSM and MCMwhen training neural PDE solvers, respectively.

Theorem 4.1 Let u t ( x ) be solution of the convection-diffusion equation in the form of Eq. 13, and assume the exact solution at time t can be expressed by the Fourier basis, i.e., u t ( x ) = ∑ N n =1 a n sin( nx ) . Let G θ be the neural PDE solver, and its prediction on u t ( x ) can be written as G θ ( u 0 , t )( x ) = ∑ N n =1 ( a n + δ n ) sin( nx ) , where δ n denotes the residual of coefficient on each Fourier basis. Let H and M denote the gird size after Fourier Interpolation and sampling numbers in neural Monte Carlo loss. Let u PSM t +∆ t ( x ) and u MCM t +∆ t ( x ) be the one-step labels starting from G θ ( u 0 , t )( x ) , given by PSM and MCM, respectively. Assume ∆ t u and u t ( x ) are Lipschitz functions with respect to t and x , respectively, i.e.:

Then, we have

2) With probability at least 1 -(2 L x u ) 2 κ ∆ t Mϵ 2 , we have 203

In the PSM, error terms E PSM 1 and E PSM 2 arise from the temporal discretization and the perturbation of G θ ( u 0 , t ) , respectively. Additionally, the error term E PSM 2 increases with the rate of n 2 , where n 2 comes from the second order derivative of sin( nx ) . To mitigate the error induced by the PSM, one has to decrease ∆ t , which inevitably necessitates additional calls to classical or neural solvers. Conversely, for MCM, the error term E MCM 1 originates from the Fourier Interpolation trick, which can be controlled by increasing the interpolation rate. This operation does not consume much time because it does not require extra solver calls. Moreover, the error caused by the residual δ n ( E MCM 2 ) remains stable as n grows due to the derivative-free property of MCM. It is worth noting that while E MCM 3 can be controlled by the number of samples M , an excessive number of particles is not required in practice. Unlike deterministic biases introduced by other error terms, E MCM 3 stems from the variance of random processes and can be regarded as a type of stochastic label noise. Some studies [7, 10] have found that such stochastic label noise can aid generalization and even counteract inherent biases. Therefore, we assert that, compared to PSM, the neural Monte Carlo method can tolerate coarser time steps and spatial variations when solving convection-diffusion equations.

5 Experiments

In this section, we conduct numerical experiments to evaluate the proposed MCNP Solver on two tasks: 1D diffusion equations and 2D NSEs. Implementation details are introduced in Appendix E. We utilize the FNO [31] as the backbone network, with more detailed discussions in Appendix C. We evaluate the model performance for all tasks via the relative ℓ 2 error on 200 test PDE samples. We repeat each experiment with three random seeds in { 0 , 1 , 2 } and report the mean value and variance. All experiments are implemented on an NVIDIA A100 GPU.

5.1 1D Diffusion Equation

In this section, we conduct experiments on periodical 1D diffusion equation defined as follows:

The initial states u ( x, 0) are generated from the functional space F N ≜ { ∑ N n =1 a n sin(2 πnx ) : a n ∼ U (0 , 1) } , where U (0 , 1) denotes the uniform distribution over (0 , 1) , and N represents the maximum frequency of the functional space.

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

* Here we unitize NAN to represent the results whose relative error is larger than 200%.

Experimental Settings In this setting, κ represents the heat transfer rate, with larger κ values indicating faster temporal variation rates. N can be regarded as a measure of spatial complexity, where larger values correspond to a higher proportion of high-frequency signals. We select two different κ in { 0 . 01 , 0 . 02 } and N in { 6 , 12 } , respectively, to evaluate the performance of different methods in handling temporal-spatial variations. We divide the spatial domain [0 , 1] into 64 grid elements for all experiments.

Baselines We introduce the baselines conducted on 1D diffusion equations, including: i). PSM : A traditional numerical methods. We divide the time interval into 100 uniform lattices and utilize the 2nd Runge-Kutta method for temporal revolution. ii). PSM+ : PSM with a fine step size. We divide the time interval into 2000 uniform lattices. iii). MCM : a traditional numerical method based on the probabilistic representation of PDEs. We set the sampling numbers as 10 5 . iv). FNO : Training with 1000 pre-generated data, calculating from the analytic solution of Eq. 16. v). PINO [32]: An unsupervised neural operator based on PSM. We divide the time interval into 100 uniform lattices. vi). PI-DeepONet [59]: an unsupervised neural operator based on PINN loss and DeepONets. For MCNPSolver , we set the sampling numbers and the time step ∆ t as 64 and 0 . 2 , respectively. We interpolate the spatial domain into 1024 elements in the Fourier Interpolation trick.

Results Table 1 presents each method's performance and computational cost on the 1D diffusion equation. Among all unsupervised neural PDE solvers, including PI-DeepONet and PINO, the MCNP Solver performs best on all tasks, particularly for cases with large spatial or temporal variations. Despite PINO obtaining comparable results on the simplest tasks (i.e., κ = 0 . 01 and N = 6 ), its error rapidly increases on tasks with κ = 0 . 02 or N = 12 , which is consistent with our theoretical results. The results of PI-DeepONet indicate that the PINN loss cannot efficiently handle high-frequency components, which has also been observed in previous literature [25, 60]. Compared to the supervised method FNO, MCNP Solver obtains comparable results on the tasks when N = 6 while significantly outperforming it when N = 12 , which indicates that more data is required for FNO when handling complex spatial variants. As for classical solvers, PSM fails on all tasks because it requires a fine grid to prevent blowing up, which explains why MCNP Solver can beat PINO. Although PSM+ achieves spectral accuracy on the tasks with κ = 0 . 01 , it still fails to achieve meaningful results when κ = 0 . 02 . Moreover, it is more than 380 times slower than other neural solvers due to the refined step size, highlighting one of the main motivations for AI-based PDE studies. MCM's performance is limited by the variance inherent in Monte Carlo simulation, even sampling 10 5 particles. However, this stochastic label noise arising from the Monte Carlo simulation does not cause apparent harm to the MCNP Solver due to the involvement of neural networks, which is in line with the studies of label noise [7, 10]. In practice, the sampling numbers in MCNP Solver are only set as 64 per epoch, and the neural network can converge as expected with gradient descent during training.

5.2 2D Navier-Stokes Equation

In this experiment, we simulate the vorticity field for 2D incompressible flows in a periodic domain Ω = [0 , 1] × [0 , 1] , whose vortex equation is given as follows:

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

A

B

Initial

Ground-truth

Predict

Ground-truth

Predict

60

50

40

0.216

-0.144

0.072

-0.000

60

50

40

- 0.900

0.675

- 0.450

- 0.225

1.125

40

60

50

0.675

1125

0.900

0.450

0.225

60

50

40

- 1.08

0.72

0.36

1.44

60

50

40

- 1.08

0.72

0.36

1.44

30

20

0.072

0.216

0.144

OE

Z

- 0.000

0.225

0.450

30

20

0.000

0.225

0.450

OE

20

0.00

0.36

0.72

30

20

0.00

0.36

0.72

10

0.288

10

0.675

10

0.675

10

1.08

10

1.08

0

0

10

20

30

40

50

60

0.360

0

0

10

Z

30

40

50

60

0.900

0

0

10

20

30

40

50

60

0.900

0

0

10

20

30

40

50

60

1.44

0

0

10

20

30

40

50

60

1.44

where f ( x ) = 0 . 1 sin (2 π ( x 1 + x 2 )) + 0 . 1 cos (2 π ( x 1 + x 2 )) is the forcing function, and ν ∈ R + represents the viscosity term. The initial vorticity is generated from the Gaussian random field N ( 0 , 7 3 / 2 ( -∆+49 I ) -2 . 5 ) with periodic boundaries.

Experimental Setups The viscosity term ν can be regarded as a measure of the temporal-spatial complexity of NSE. As ν decreases, the nonlinear term ( u · ∇ ) ω gradually governs the motion of fluids, increasing the difficulty of simulation. To evaluate the performance of handling different degrees of turbulence, we conduct the experiments with ν in { 10 -3 , 10 -4 , 10 -5 } , respectively. We choose two different T in { 10 , 15 } to test the long-time simulation ability of each method. We divide the domain Ω into 64 × 64 grid elements.

Baselines We introduce the baselines conducted on 2D NSEs, including: 1 i). PSM : We divide the time interval into 100 (150) uniform lattices for T = 10 (15) and utilize the Crank-Nicolson scheme for temporal revolution. ii). PSM+ : We divide the time interval into 2000 (3000) uniform lattices for T = 10 (15) . iii). FNO : Training with 1000 pre-generated data, taking 0.624 hours for data generation. iv). PINO : We divide the time interval into 100 and 150 uniform lattices for T = 10 and 15 , respectively. For MCNP Solver , we set the sampling numbers and step size ∆ t to 16 and 0 . 1 , respectively. We interpolate the spatial domain into 256 × 256 elements in the Fourier Interpolation trick. The ∆ T in the multi-scale framework is set to 5 for all tasks.

Results Table 2 presents each method's performance and computational cost on the 2D NSEs. As the viscosity term ν decreases, simulating the flow becomes more challenging for all methods due to increased turbulence, as shown in Fig. 2. Compared to PINO, MCNP Solver achieves comparable results on ν = 10 -3 while outperforming it when ν = 10 -4 and 10 -5 , indicating that MCNP Solver is more accurate on turbulent flow simulation. Furthermore, MCNP Solver has advantages and disadvantages compared to the supervised baseline FNO. On the one hand, MCNP Solver can learn from more training samples due to its data-free regime. On the other hand, the FNO directly uses

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

the ground-truth data as training labels for all t ∈ [0 , T ] , thus avoiding accumulated errors arising from the calls of the solver during the training stage like other unsupervised methods. As a result, MCNP Solver and FNO achieve better results on most tasks when T = 10 and 15 , respectively. As for classical solvers, PSM only obtains meaningful results when ν = 10 -3 , confirming that both PSM and PINO are not robust to coarser time steps. PSM+ achieves the lowest error rate on most tasks but requires almost 180 ∼ 300 times more inference time than other neural solvers.

5.3 Ablation Study

We performed several ablation studies of MCNP Solver on NSE ( ν = 10 -5 , T = 15) to understand the contribution of each model component. MCNP-OR replaces the one-step rollout technique with two-step when simulating the SDEs. MCNP-FI and MCNP-MS represent the MCNP Solver without the Fourier Interpolation and multi-scale trick, respectively. MCNP-MC replaces the neural Monte Carlo loss with the PSM loss, which aligns with the loss function in PINO. Table 3 reports the results and training costs. MCNP-OR obtains comparable results with MCNP while spending 44% additional training time. Compared to MCNP with MCNP-FI, the Fourier Interpolation trick can significantly improve the accuracy of MCNP while introducing little extra computational cost. The reason is that the rate-determining step in the training stage is the optimization of neural solvers, and the Fourier Interpolation trick does not involve any calls of solvers. Compared to MCNP with MCNP-MS, we can see that the multi-scale framework plays a vital role in improving the long-time simulation ability of MCNP. Additionally, this architecture can reduce the training time because each sub-network is relatively lightweight. Finally, the gap between MCNP and MCNP-MC reveals the advantages of Monte Carlo loss compared to the PSM loss, which is more robust against spatial-temporal variations in turbulence simulation tasks.

5.4 Additional Numerical Results

We also conduct experiments to evaluate the MCNP Solver's ability to handle different boundary conditions, fractional Laplacian, and irregular grids, as detailed in Appendix C.

6 Conclusion and Discussion

Conclusion In this paper, we propose the MCNP Solver, which leverages the Feynman-Kac formula to train neural PDE solvers in an unsupervised manner. Theoretically, we characterize the approximation error and robustness of the MCNP Solver on convection-diffusion equations. Numerical analyses demonstrate the MCNP Solver's ability to adapt to complex spatiotemporal variations and long-time simulations on diffusion equations and NSEs.

Limitations This paper has several limitations: (1) The theoretical results are lacking when β is not constant, and the gradient flow of the MCNP Solver during the training stage requires further analysis. (2) Some PDEs are not suitable for the Feynman-Kac formula and therefore do not fall within the scope of the MCNP Solver, such as third or higher-order PDEs (involving high-order operators like u xxx ). (3) The accuracy of the MCNP Solver cannot outperform numerical solvers when disregarding inference time, which is also a major drawback for other existing neural solvers [55, 13]. As discussed in [55], AI-based methods lack precision compared to classical methods while achieving reasonable accuracy and offering great potential for efficient parameter studies.

Future Work In addition to addressing the limitations, we suggest several directions for future research: (1) Extend the proposed MCNP Solver to broader scenarios, such as high-dimensional PDEs and optimal control problems; (2) Utilize techniques from out-of-distribution generalization [53] to improve the generalization ability of MCNP Solver.

335

336

337

References

Juan A. Acebrón and Marco A. Ribeiro. A monte carlo method for solving the one-dimensional telegraph equations with boundary conditions. Journal of Computational Physics , 305:29-43, 2016.

W. F. Bauer. The monte carlo method. Journal of the Society for Industrial and Applied Mathematics , 338 6(4):438-451, 1958. 339

Julius Berner, Markus Dablander, and Philipp Grohs. Numerically solving parametric families of high340 dimensional kolmogorov partial differential equations via deep learning. Advances in Neural Information 341 Processing Systems , 33:16615-16627, 2020. 342

Deniz A Bezgin, Steffen J Schmidt, and Nikolaus A Adams. A data-driven physics-informed finite-volume 343 scheme for nonclassical undercompressive shocks. Journal of Computational Physics , 437:110324, 2021. 344

Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message passing neural PDE solvers. In 345 International Conference on Learning Representations , 2022. 346

Shuhao Cao. Choose a transformer: Fourier or galerkin. In A. Beygelzimer, Y. Dauphin, P. Liang, and 347 J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. 348

349

350

351

Pengfei Chen, Guangyong Chen, Junjie Ye, Jingwei Zhao, and Pheng-Ann Heng. Noise against noise: stochastic label noise helps combat inherent label noise. In International Conference on Learning Representations , 2021.

352

353

354

355

356

Zhao Chen, Yang Liu, and Hao Sun. Physics-informed learning of governing equations from scarce data. Nature communications , 12(1):1-13, 2021.

Robert Dalang, Carl Mueller, and Roger Tribe. A feynman-kac-type formula for the deterministic and stochastic wave equations and other pde's. Transactions of the American Mathematical Society , 360(9):4681-4703, 2008.

Alex Damian, Tengyu Ma, and Jason D Lee. Label noise sgd provably prefers flat global minimizers. 357 Advances in Neural Information Processing Systems , 34:27449-27461, 2021. 358

Francis X Giraldo and Beny Neta. A comparison of a family of eulerian and semi-lagrangian finite element 359 methods for the advection-diffusion equation. WIT Transactions on The Built Environment , 30, 1997. 360

Somdatta Goswami, Aniruddha Bora, Yue Yu, and George Em Karniadakis. Physics-informed neural 361 operators. arXiv preprint arXiv:2207.05748 , 2022. 362

Tamara G. Grossmann, Urszula Julia Komorowska, Jonas Latz, and Carola-Bibiane Schönlieb. Can 363 physics-informed neural networks beat the finite element method?, 2023. 364

365

366

367

Ling Guo, Hao Wu, Xiaochen Yu, and Tao Zhou. Monte carlo fpinns: Deep learning method for forward and inverse problems involving high dimensional fractional partial differential equations. Computer Methods in Applied Mechanics and Engineering , 400:115523, 2022.

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

Patrik Simon Hadorn. Shift-deeponet: Extending deep operator networks for discontinuous output functions. ETH Zurich, Seminar for Applied Mathematics, 2022.

Jiequn Han, Arnulf Jentzen, and Weinan E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences , 115(34):8505-8510, 2018.

Jihun Han, Mihai Nica, and Adam R Stinchcombe. A derivative-free method for solving elliptic partial differential equations with deep neural networks. Journal of Computational Physics , 419:109672, 2020.

Jan Hermann, Zeno Schätzle, and Frank Noé. Deep-neural-network solution of the electronic schrödinger equation. Nature Chemistry , 12(10):891-897, 2020.

Xiang Huang, Zhanhong Ye, Hongsheng Liu, Shi Bei Ji, Zidong Wang, Kang Yang, Yang Li, Min Wang, Haotian CHU, Fan Yu, Bei Hua, Lei Chen, and Bin Dong. Meta-auto-decoder for solving parametric partial differential equations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. Nsfnets (navier-stokes flow nets): Physicsinformed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics , 426:109951, 2021.

Matthias Karlbauer, Timothy Praditia, Sebastian Otte, Sergey Oladyshkin, Wolfgang Nowak, and Martin V Butz. Composing partial differential equations with physics-aware neural networks. In International Conference on Machine Learning , pages 10773-10801. PMLR, 2022.

George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. 386 Physics-informed machine learning. Nature Reviews Physics , 3(6):422-440, 2021. 387

Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew 388 Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications 389 to pdes. Journal of Machine Learning Research , 24(89):1-97, 2023. 390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

Tomasz J Kozubowski, Mark M Meerschaert, and Krzysztof Podgorski. Fractional laplace motion. Advances in applied probability , 38(2):451-464, 2006.

Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems , 34:26548-26560, 2021.

Jae Yong Lee, SungWoong CHO, and Hyung Ju Hwang. HyperdeepONet: learning operator with complex target function space using the limited resources via hypernetwork. In The Eleventh International Conference on Learning Representations , 2023.

Hong Li, Qilong Zhai, and Jeff ZY Chen. Neural-network-based multistate solver for a static schrödinger equation. Physical Review A , 103(3):032405, 2021.

Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning. arXiv preprint arXiv:2205.13671 , 2022.

Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. arXiv preprint arXiv:2207.05209 , 2022.

Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485 , 2020.

Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794 , 2021.

Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence , 3(3):218-229, 2021.

Sylvain Maire and Etienne Tanré. Monte carlo approximations of the neumann problem. In Monte Carlo Methods Appl. , 2012.

Xuerong Mao. Stochastic differential equations and applications . Elsevier, 2007.

Nils Margenberg, Dirk Hartmann, Christian Lessig, and Thomas Richter. A neural network multigrid solver for the navier-stokes equations. Journal of Computational Physics , 460:110983, 2022.

Guillermo Marshall. Monte carlo methods for the solution of nonlinear partial differential equations. Computer Physics Communications , 56(1):51-61, 1989.

Revanth Mattey and Susanta Ghosh. A novel sequential method to train physics informed neural networks for allen cahn and cahn hilliard equations. Computer Methods in Applied Mechanics and Engineering , 390:114474, 2022.

Chloé Mimeau and Iraj Mortazavi. A review of vortex methods and their applications: From creation to recent advances. Fluids , 6(2):68, 2021.

Sebastian K Mitusch, Simon W Funke, and Miroslav Kuchta. Hybrid fem-nn models: Combining artificial neural networks with the finite element method. Journal of Computational Physics , 446:110651, 2021.

Nikolas Nüsken and Lorenz Richter. Interpolating between bsdes and pinns-deep learning for elliptic and parabolic boundary value problems. arXiv preprint arXiv:2112.03749 , 2021.

Panos Pantidis and Mostafa E Mobasher. Integrated finite element neural network (i-fenn) for non-local continuum damage mechanics. Computer Methods in Applied Mechanics and Engineering , 404:115766, 2023.

Etienne Pardoux and Shige Peng. Backward stochastic differential equations and quasilinear parabolic partial differential equations. In Stochastic partial differential equations and their applications , pages 200-217. Springer, 1992.

Etienne Pardoux and Shanjian Tang. Forward-backward stochastic differential equations and quasilinear parabolic pdes. Probability Theory and Related Fields , 114(2):123-150, 1999.

Md Ashiqur Rahman, Zachary E Ross, and Kamyar Azizzadenesheli. U-NO: U-shaped neural operators. Transactions on Machine Learning Research , 2023.

Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics , 378:686-707, 2019.

Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity 447 and pressure fields from flow visualizations. Science , 367(6481):1026-1030, 2020. 448

Carl Remlinger, Joseph Mikael, and Romuald Elie. Robust Operator Learning to Solve PDE. working 449 paper or preprint, April 2022. 450

Lorenz Richter and Julius Berner. Robust sde-based variational formulations for solving linear pdes via 451 deep learning. In International Conference on Machine Learning , pages 18649-18666. PMLR, 2022. 452

Lorenz Richter, Leon Sallandt, and Nikolas Nüsken. Solving high-dimensional parabolic pdes using the 453 tensor train format. In International Conference on Machine Learning , pages 8998-9009. PMLR, 2021. 454

455

456

Rohan Sawhney, Dario Seyb, Wojciech Jarosz, and Keenan Crane. Grid-free monte carlo for pdes with spatially varying coefficients. ACM Transactions on Graphics (TOG) , 41(4):1-17, 2022.

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

Jacob H Seidman, Georgios Kissas, Paris Perdikaris, and George J. Pappas. NOMAD: Nonlinear manifold decoders for operator learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624 , 2021.

Wenlei Shi, Xinquan Huang, Xiaotian Gao, Xinran Wei, Jia Zhang, Jiang Bian, Mao Yang, and Tie-Yan Liu. Lordnet: Learning to solve parametric partial differential equations without simulated data. arXiv preprint arXiv:2206.09418 , 2022.

Derick Nganyu Tanyu, Jianfeng Ning, Tom Freudenberg, Nick Heilenkötter, Andreas Rademacher, Uwe Iben, and Peter Maass. Deep learning methods for partial differential equations and related parameter identification problems, 2022.

Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. In The Eleventh International Conference on Learning Representations , 2023.

Simone Venturi and Tiernan Casey. Svd perspectives for augmenting deeponet flexibility and interpretability. Computer Methods in Applied Mechanics and Engineering , 403:115718, 2023.

J. V om Scheidt. Kloeden, p. e.; platen, e., numerical solution of stochastic differential equations. berlin etc., springer-verlag 1992. xxxvi, 632 pp., 85 figs., dm 118,oo. isbn 3-540-54062-8 (applications of mathematics 23). ZAMM - Journal of Applied Mathematics and Mechanics / Zeitschrift für Angewandte Mathematik und Mechanik , 74(8):332-332, 1994.

Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science advances , 7(40):eabi8605, 2021.

Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics , 449:110768, 2022.

Yizheng Wang, Jia Sun, Wei Li, Zaiyuan Lu, and Yinghua Liu. Cenn: Conservative energy method based on neural networks with subdomains for solving variational problems involving heterogeneous and complex geometries. Computer Methods in Applied Mechanics and Engineering , 400:115491, 2022.

Li-Ming Yang. Kinetic theory of diffusion in gases and liquids. i. diffusion and the brownian motion. Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences , pages 94-116, 1949.

Rui Zhang, Peiyan Hu, Qi Meng, Yue Wang, Rongchan Zhu, Bingguang Chen, Zhi-Ming Ma, and Tie-Yan Liu. Drvn (deep random vortex network): A new physics-informed machine learning method for simulating and inferring incompressible fluid flows. Physics of Fluids , 34(10):107112, 2022.

Xicheng Zhang. Stochastic functional differential equations driven by lévy processes and quasi-linear partial integro-differential equations. The Annals of Applied Probability , 22(6):2505-2538, 2012.

Xicheng Zhang. Stochastic lagrangian particle approach to fractal navier-stokes equations. Communications in Mathematical Physics , 311(1):133-155, 2012.

Qingqing Zhao, David B. Lindell, and Gordon Wetzstein. Learning to solve pde-constrained inverse 493 problems with graph networks. In ICML , 2022. 494

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an unsupervised neural technique for solving PDEs based on the classical correspondence between (parabolic) partial differential equations (PDE) and stochastic differential equations (SDE) as given by the Feynman-Kac formula. Specifically, they propose minimizing the error between the neural approximation's deterministic prediction at timestep t+1 and the expected prediction of the neural approximation over particles that have evolved up to time t stochastically according to the Feynman-Kac SDE representation of the given PDE.\n\nSTRENGTHS:\nIncorporating Feynman-Kac into the PINN framework is an interesting idea.\n\nWEAKNESSES:\n# Poor numerics\nThe experiments do not justify the claims, e.g. that long rollouts are more stable using this method. To prove this, at the very least the authors need to present results where the trajectories actually exhibit turbulence. Then, an ablation with the multi-scale framework is required.\n\n# Poor presentation\nThe authors cannot expect the reader to be familiar with Feynman-Kac and need to give explanations in plain English of the significance of this result. Then, the equations in the main paper should aim to clarify this further, not give a comprehensive mathematical presentation. For example, the inclusion of the forcing function  distracts from the main result which is using time reversal to obtain an SDE that moves in the right time direction for equations that are most often solved using neural networks, e.g. ones with an initial condition, not a final condition.\n\nFurthermore, only the most experienced reads will walk away from this paper with a clear idea of how to implement the proposed algorithm. The emphasis in the paper should be to give the reader something to implement and try, not a theoretical proof.\n\nFinally, there are a number of tricks that are not included in the initial idea and are not sufficiently explained. For example, the Fourier interpolation. The multi-scale framework makes the model non-parametric which is a significant departure from previous work and from the presentation of this paper as a Monte-Carlo approximation to PDEs.\n\nQUESTIONS:\nPlease explain the Fourier interpolation, that would be very helpful.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present MCNP, a new unsupervised training loss for surrogate simulation networks. This loss is based on the link between stochastic processes and PDEs, sampling one-step Brownian motion to estimate the PDE solution. The learned network takes an initial state and the target simulation time to compute the state at that time in one pass. For longer periods, multiple NNs are trained, one for each sub-interval.\n\nSTRENGTHS:\nThe paper is generally well-written and relatively easy to understand. It includes a good overview over related work.\n\nThe paper includes both theoretical and numerical results. The method is derived using the Feynman-Kac formula and the authors show how the errors of PSM and MCM scale when given an incorrect input state, such as predicted by a neural network.\n\nA total of five numerical experiments are performed, covering a large range of simulation configurations. The paper contains an ablation study, giving some insight into the impacts of various parts of the MCNP method.\n\nWEAKNESSES:\nWhile the experiments are varied in the tested configurations, all but on experiment consider simple diffusion equations, some of which can be solved analytically.\n\nThe paper does not show any simulation trajectories and gives no insight into how the various tested methods behave in their experiments. Instead, only the final losses are reported. This makes it hard to determine the cause of improvement from the numerical results. I strongly recommend documenting your observations in the appendix.\n\nThe Navier-Stokes experiment seems to be mostly forcing-driven with all initial states ending in a similar configuration. A different forcing, such as Kolmogorov flow, would result in a much wider range of trajectories.\n\nThe paper does not give details as to how the numerical simulation (PSM) of the Navier-Stokes experiment was performed. Please describe the simulator in more detail if you implemented it yourself.\n\nThe source code is not part of the submission and the authors have not declared their intent to make it public. I strongly recommend doing so, especially if the employed CFD solver was implemented from scratch.\n\nMinor:\n\n* Eq. 3: Please indicate the x-dependence of xi\n* Fig 1 caption: The formulas are missing a factor of 1/M\n* L184: You mention that cutting the gradients prevents numerical instabilities but ignore the positive effects that gradient backpropagation can have.\n* You refer to Eq.13 as a convection-diffusion equation but it does not contain a convection term.\n* L199: The notation Δ ₜ u is confusing since Δ is already in use.\n* L216: The argument that label noise helps MCNP perform coarser time steps seems unfounded to me. This requires an explanation.\n* L237: By lattices, you probably mean frames or time steps?\n* L268 and Eq. 17: You can simplify the forcing to a single sine term.\n* Figure 2 is never referenced.\n* Figure 2 shows the vorticity, correct? Please specify in the caption.\n\nQUESTIONS:\nYou state that the random walk of particles should stop when they hit a Dirichlet boundary. Is this still true when performing multiple steps for the time integration of the stochastic process? I think this would over-value the boundary condition, as particles are denied the chance to re-enter the domain.\n\nUsing the Feynman-Kac formula, you could also derive a formulation where the stochastic process is simulated forward in time instead of backward. Have you tried this?\n\nIn your periodic diffusion experiments, PSM should be equal to a purely spectral method, right?\n\nWhy did you choose a vorticity formulation for the Navier-Stokes experiment instead of a velocity formulation?\n\nHow did you choose the temporal discretization (like 100 for PSM, 2000 for PSM+) for the tested methods?\n\nThe Fourier trick seems to emulate interpolation on the grid. Have you tried linear interpolation or Fourier-upsampling + linear interpolation?\n\nYou say that you run some baselines with PyTorch and others with Jax. Is JIT compilation enabled on all examples? How big do you think the performance difference would be when switching all to one library?\n\nIt looks like the PSM methods fail due to the time increment being too large. Have you looked at the CFL numbers? Would this problem be resolved by dynamic time steps?\n\nLIMITATIONS:\nThe limitations are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
The authors propose Monte Carlo Neural PDE Solver (MCNP Solver) which leverages the Feynman-Kac formula to train neural PDE solvers in an unsupervised manner.

I'm willing to revise my score based on the rebuttal from the authors to the questions that I raised below.

STRENGTHS:
* That paper addresses and interesting problem: learning the neural operator in an unsupervised way. 
* The authors propose practical enhancements to their method such as one-step rollout, Fourier Interpolation and the use of a multi-scale framework.

WEAKNESSES:
* Limited set of experiments, only two cases: 1d diffusion and 2d Navier-Stokes.

QUESTIONS:
* Why aren't some other equations such as the Poisson equation, Schrodinger equation, Allen-Cahn not ran?
* To my understanding, you use the multi-scale framework to achieve longer simulations. Do you have an experiment that shows the benefit of this? Especially an experiment that shows how alternative method break at certain points whereas MCNP lasts longer.

LIMITATIONS:
* The limitations that I see are encapsulated on the questions that I raised above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDesigning neural PDE sovler using deep neural networks is a challenging task for which several solutions have been proposed in the literature using for instance networks that encode the initial conditions or physics informed neural networks.\n\nThe authors propose to use Monte Carlo methods to train neural PDE solver for the solution of a general convection-diffusion equation.\nUsing Feynman-Kacformula, the authors derive a loss function that can be used to learn a mapping that can simulate the target fields using the input parameters and the initial condition. \n\nThey propose a theoretical guarantee on the solution provided by the Monte Carlo solver and the paper illustrates the performance of the proposed method with a  one dimensional differential equation and a 2-dimensional Navier-Stokes equation.\n\nSTRENGTHS:\nThe paper proposes a Monte Carlo based PDE solver strained via Monte Carlo approximation which can  handle  coarse time steps better than existing alternatives. \n\nThe proposed method does not require many particles in the numerical experiment and is computationally efficient in the settings explored.\n\nUnder some assumptions, the authors propose an upper bound on the error explicit in some hyperparameters of the approach in the case of a  convection diffusion equation.\n\nWEAKNESSES:\nTheorem 1 is obtained under several assumptions. \nThese assumptions should be discussed more, are they restrictive or common assumptions in the PDE literature ?\n\nThe claim that the approach is efficient even with few samples seems correct in the proposed experiments. However, these experiments are in dimension 1 and 2 and Monte Carlo methods can be cumbersome in high dimensional settings without tunning. \nThe authors could detail the explicit advice or theoretical guarantees we have for the error with respect to M.\n\nThe authors only explore one discretization scheme (Euler), the scheme used could have an impact on the performance of the method, can this be discussed ?\n\nQUESTIONS:\nThe algorithm relies on many hyperparameters which appear in the upper bound of Th.1. The dependency on these hyperparameters could be discussed more. For instance, the authors claim that the third term in (15) can be controlled by the number of samples M, and that an excessive number of particles is not required in practice. \nIs it possible to provide an explicit way to balance each term in the upper bound to guarantee a given precision ? How to choose M with respect to N or $\\Delta t$ for instance ?\n\nSection 4 focuses on a specific case where parameters are constant. Can the authors elaborate on the difficulties (practical and theoretical) when these parameters are not constant ?\n\nIn Section 3.3, Eq. (11), the authors propose to use an Euler Scheme to sample the stochastic process. Can the results be improved by choosing another discretization scheme ? Is this a sensitive step of the implementation ?\nIn comparison with PSM which requires to decrease $\\Delta t$ to improve the upper bound (which is costly) the proposed method seems less sensitive and increasing M is enough to reduce the impact of the additional term. In terms of computational complexity,  increasing   M is not too intensive ?\n\nThe simulation study provides application in dimensions 1 and 2, have you any insight on how the method scales with d ?\n\nLIMITATIONS:\nThe authors provide several research perspectives for this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new physics informed neural network based solver that utilizes the connection between PDEs and SPDEs. This is achieved through the Feynman-Kac formula and applies to a large class of PDEs. It comes with a bound on the error at each step in the rollout. The results are compared with multiple supervised and unsupervised PDE solvers on a number of 1D and 2D equations.\n\nSTRENGTHS:\nOriginality: To the best of my knowledge, the paper is original in combining the Feynman-Kac-based approach with a neural operator architecture. The connections to existing work that relies on the Feynman-Kac formula as well as other PINN/NO approaches are discussed in detail in the main paper and the appendix.\n\nQuality: The work is thorough in discussing existing literature and presenting the methodology. The theoretical result gives some intuition relating to how the proposed method scales as compared to the classical solver. \n\nSignificance: The methodology combines a number of existing techniques (Feynman-Kac formulation, neural operators, Fourier interpolation) to achieve some improvement in specific setups, e.g. where the solution of the PDE is rapidly varying in space and/or time.\n\nWEAKNESSES:\nClarity: The quality of the writing could be improved, particularly in the abstract/introduction. The rest of the paper is detailed enough in describing the experiments and discussing the results. \n\nSignificance: The proposed approach seems to give an advantage only in specific situations. While this is acknowledged in the paper, it could be beneficial to discuss specific applications where such oscillatory conditions occur.\n\nQUESTIONS:\n1. What are FDM and PSM as mentioned in the abstract?\n\n2. Is PSM a spectral solver? \n\n3. Line 119: Can you explain in more detail what is meant by inversion of $\\xi$?\n\n4. What is the number of dimensions in the latent space (i.e. the number of terms in the Fourier expansion) for the FNO method in your experiments? I assume increasing the number of terms in the expansion would improve the performance of FNO for high-frequency initial states and might not affect the computational cost hugely.\n\n5. To clarify, I assume the computational times given in Tables 1 and 2 do not include the generation of data for data-driven methods such as FNO?\n\nLIMITATIONS:\nThe limitations and extent to which this method gives an advantage over existing approaches are discussed in detail in the final section of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}