Physics-Informed Neural Operator for  Learning Partial Differential Equations
Authors: Zongyi Li
Hongkai Zheng
Nikola Borislavov Kovachki
David Jin
Haoxuan Chen
Burigede Liu
Andrew Stuart
Kamyar Azizzadenesheli
Anima Anandkumar
Venue: ICLR 2022 Submitted (2022)
Keywords: Partial Differential Equations
operator learning
physics-informed
PINN
inverse problem
Navier-Stokes Equation

Abstract: Machine learning methods have recently shown promise in solving partial differential equations (PDEs). They can be classified into two broad categories: solution function approximation, and operator learning. The Physics-Informed Neural Network (PINN) is an example of the former while the Fourier neural operator (FNO) is an example of the latter. Both these approaches have shortcomings. The optimization in PINN is challenging and prone to failure, especially on multi-scale dynamic systems. FNO does not suffer from this optimization issue since it carries out supervised learning on a given dataset, but obtaining such data may be too expensive or infeasible. In this work, we propose the physics-informed neural operator (PINO), where we combine the operating-learning and function-optimization frameworks, and this improves convergence rates and accuracy over both PINN and FNO models. In the operator-learning phase, PINO learns the solution operator over multiple instances of the parametric PDE family. In the test-time optimization phase, PINO optimizes the pre-trained operator ansatz for the querying instance of the PDE. Experiments show PINO outperforms previous ML methods on many popular PDE families while retaining the extraordinary speed-up of FNO compared to solvers. In particular, PINO accurately solves long temporal transient flows and chaotic Kolmogorov flows, while PINN and other methods fail to converge to a reasonable accuracy.  

Under review as a conference paper at ICLR 2022
PHYSICS-INFORMED NEURAL OPERATOR FOR LEARNING PARTIAL DIFFERENTIAL EQUATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning methods have recently shown promise in solving partial differen- tial equations (PDEs). They can be classified into two broad categories: solution function approximation and operator learning. The Physics- Informed Physics-Informed Neural Net- work (PINN) is an example of the former while the Fourier neural operator (FNO) is example of the latter. Both these approaches have shortcomings. The opti- mization in PINN is challenging and prone to failure, especially on multi-scale dynamic systems. FNO does not suffer from this optimization issue since it carries out supervised learning on a given dataset, but obtaining such data may be too expensive infeasible. In this work, we propose the physics-informed neural oper- ator (PINO), where we combine the frameworks, and this improves convergence rates and accuracy over both PINN and FNO models. In the operator-leaming phase, PINO learns the solution operator over multiple instances of the parametric PDE family. In the test-time optimization phase, PINO optimizes the pre-trained operator ansatz for the querying instance the PDE. Experiments show PINO outperforms previous ML methods on many popular PDE families while retaining the extraordinary speed-up of FNO compared to solvers. In particular, PINO accurately solves long temporal transient flows and Kolmogorov flows, while PINN and other methods fail to converge.
and function-opin cuactioonoptiizztiion
INTRODUCTION
Machine learning-based methods are starting to show promise scientific computing and especially in solving partial differential equations (PDEs). They have demonstrated advantages in both efficiency and accuracy compared to conventional solvers. They are even able to tackle previously intractable
problems such as sigher-dimensional multi-scale, high-contrast, and chaotic PDE systems (Um et al. 2020 Brunton et al. 22220] Fan et al., [2018] Long et al. 2018) Han et al. 2018) Bruno et al.) (021)). Broadly, MI--based approaches for PDHs can be divided into categories: optimizing to solve for a specific solution function of PDE vs. learning the solution operator over a family of PDEs.
Optimization of solution function and PINN. Most ML-based methods, well the conven- tional solvers, fall into this category. Conventional solvers such as FDM and FEM usually discretize the domain into grid and optimize/approximate the solution function the grid, which imposes a truncation error. The Physics-Inforinn overcome the discretization issue (Raissi et al. 2019). They use a neural network the ansatz of the solution function and take advantage of auto-differeniiation to compute the exact, mesh-free derivatives. Recently, researchers have developed numerous variations of PINN with promising
Neural Network (PINN)-type methods are proposed to
results on inverse problems and partially observed tasks (Lu et al., 2021a Zhu et al.) 2019 Smith al. 202T). However, compared conventional solvers, PINNs face several optimization issues: (1)
the challenging optimization landscape from soft physics or PDE constraints (Wang et al. (021a). (2)
the difficulty to propagate information from the initial or boundary conditions to unseen parts of the
interior or to future times (Dwivedi & Srinivasan, 2020), and (3) the sensitivity to hyper-parameters selection (Sun et al. 2020 As a result, PINNs are still unable to compete with conventional solvers in most cases, and they often fail to converge
high-frequene
multi-scale PDEs (Wang et al.)
20200b Fuks & Tchelepil 22000] Raissi et al.) 2020). In this work, we propose to overcome these optimization challenges by integrating operator learning with PINN.

---

Under review as a conference paper at ICLR 2022
(a) Operator learning
(b) Test-time optimization
Figure I: PINO: combine operator learning and PINN optimization.
(a) learn the solution operator from a family of equations. (b) use the learned operator as ansatz to solve for specific instance.
Operator learning and neural operators. A recent alternative approach is learn the solution operator of family of PDEs, defined by the map from the input-initial conditions and boundary conditions, to the output-solution functions. In this case, usually, dataset of imput-output pairs from an existing solver is given. There are two main aspects to consider (a) model: to design models for learning highly complicated PDE solution operators, and (b) data: be data-efficient and to improve generalization. Recent advances in operator learning replace traditional convolutional neural networks and U-Nets from computer vision with operator- model tailored to PDEs with greatly improved model expressiveness (Li et al., 2020c Lu et al. 2019, Patel et al.) 202 Wang al. 2020a Duvall et al. 202T). Specifically, the neural operator generalizes the neural network to the operator setting where the input and output spaces are
The framework has shown success in learning resolution -soluttioninvariint solution operators for highly non-linear problems such turbulence flow (Li et al., 2020b ab. However, the data challenges remain: (1) the need for training data, which assumes an existing solver or experimental setup, (2) the non-negligible generalization error, and (3) extrapolation unseen conditions. These issues can be addressed by adding physics or
PDE constraints to operator learning (Zhu et al. 2019 Wang et al. 2021b Zhang et al. 2021)
Our contributions. To overcome the shortcomings of both physics- infoomedd optimization and data-driven operator learning, we propose the physics-informed neural operator (PINO) that combines operator learning with equation solving (test-time optimization). It requires fewer no data points to learn the operator and generalizes better. PINO, we use the pre-trained operator as the ansatz to optimize for the solution function at test time, which reduces the generalization error. Compared to PINN, PINO has a much better optimization landscape and representation space, and hence, PINO converges faster and more accurately. Our contributions be summarized as follows:
We propose the physics-ifformed neural operator (PINO), combining the operator-learnng and physics-informed settings. We introduce the pre-training and test-time optimization schemes that utilize both the data and equation constraints (whichever are available). We develop an efficient method to compute the exact gradient for neural operators to incorporate the equation constraints.
By utilizing pre- pre-trained operator ansatz, PINO overcomes the challenge of propagating information from the initial condition to future time steps with (soft) physics constraints. It can solve the 2d transient flow over an extremely long time period, where PINN and DeepONet (Lu et al., 2019) fail. Even without any pre-training and using only PDE constraints for the given instance, PINO still outperforms PINN by 20x smaller error and 25x speedup on the chaotic Kolmogorov flow, demonstrating superior expressivity of the neural operator over standard neural networks.
training data and generalizes better compared to FNO (Li et al., 2020c). On average it has 7% smaller error on the transient and Kolmogorov flows, while matching the speedup of FNO (400x) compared the GPU-based pseudo-spectral solver (He & Sun 2007). matching FNO. Further, the pre- pre-trained PINO model on the Navier Stokes equation can be easily transferred different Reynolds numbers ranging from 100 to 500 using test-time optimization.
By utilizing the equation constraints, PINO requires fewer
We propose the forward and backward PINO models for inverse problems. Our approach accurately recovers the coefficient function in the Darcy flow which is 3000x faster than the conventional solvers using accelerated MCMC (Cotter et 2013).
Our major novelty and contributions are to use the pre-trained operator ansatz with instance-wise fine-tuning to overcome the optimization challenges in PINN and the generalization challenges
2

---

Under review as a conference paper at ICLR 2022
in operator learning. Previous works such as PINN-DeepONet (Wang et al. 2021b) and Physics- constrained modeling (Zhu et al. 2019) use the PDE constraints in operator learning. like we do during the pre-training phase in PINO. However, propose several methodological advances as well as extensive experiments to understand the optimization and generalization challenges. Our methodological advances include: (1) Instance-wise fine-tuning test-time to further improve the fidelity of the operator ansatz. (2) Efficient Fourier-space methods for computing derivatives present in the PDE loss. (3) Efficient learning through the design of data augmentation and loss functions. (4) Novel formulation for inverse problems that results in accurate recovery as well as good speedups.
PINN vs. PINO: pointwise vs. function-wist optimization. The neural operator ansatz in PINO has an easier optimization landscape and a more expressive representation space compared to the neural networks ansatz in PINN. The neural operator parameterizes the solution function as an aggregation of basis functions, and hence, the optimization is in the function space. This is casier than just optimizing single function in PINN. Further, can learn these basis functions in the pre-training phase which makes the test-time optimization on the querying instance even easier. In this case, PINO does not need to solve from scratch. It just fine-tunes the solution function parameterized by the solution operator. Thus, PINO is much faster and more accurate compared to PINN.
PRELIMINARIES AND PROBLEM SETTINGS
2.1
PROBLEM SETTINGS
We consider two natural class of PDEs. In the first, we consider the stationary system
P(u, a) = 0, in D C Rd
(I)
9.
aD
where D is a bounded domain, a E A C is a PDE coefficientpprrmmeter, u E ll is the unknown, and P 4 A F is a possibly non-linear partial differential operator with (21 V,F) triplet of Banach spaces. Usually the function is a fixed boundary condition (potentially can be entered as parameter). This formulation gives rise to the solution operator G A Li defined by + u. A prototypical example is the second-order elliptic equation P(u, a) = -V V (aVu) +
In the second setting, consider the dynamical system
du R(u), u 9. in ØD x (0, oo)
in D x (0)
u
in D {0}
where a =(0) E A is the initial condition, u(t) E U for t > 0 is the unknown, and R. is possibly non-linear partial differential operator with U4 and Banach spaces. As before, we take g to be a known boundary condition. We assume that exists and is bounded for all time and for every Uo E U. This formulation gives rise the solution operator Gt A - C((O, 24: defined by + Prototypical examples include the Burgers' equation and the Navier-Stokes equation.
2.2 SOLVING EQUATION USING THE PHYSICS- PHYSICS-INFORMED LOSS (PINN)
Given an instance and solution operator Gi defined by equations I or (2) we denote by ut = 9 (a) the unique ground truth. The equation solving task is to approximate ut. This setting consists of the ML-enhanced conventional solvers such as learned finite element, finite difference, and multigrid solvers (Kochkov et al. 2021 Pathak et al. 2021 Greenfeld et al., 2019). as well as purely Neural Networks (PINNs), Deep Galerkin Method, and Deep Ritz Method (Raissi et al.) 2019 Sirignano & Spiliopoulos 2018 Weinan & Yu 2018). Especially, these PINN-type methods use a neural network us with parameters & as the the ansatz approximate the solution function ut The parameters 0 are found by minimizing the physics-infe informed loss with exact derivatives computed using the stationary case, the physics-informed loss is defined by minimizing the l.h.s. of equation (1) in
neural network-basse solvers such as the Physics-Informa
(autograd), In
3

---

Under review as a conference paper at ICLR 2022
the squared norm of F. A typical choice is F (D), giving the loss function
Lpde (a, uo) = [a no 1(2(()) + alluolan - =lli>(aD)
J a())) + /
(3)
[u(()) 2x
aaD
In the case of a dynamical system, it minimizes the residual of equation (2) in some natural norm up to fixed final time T 0. typical choice the 1,3 (0, T); (D)) norm, yielding
due T S
dt (t,z) - ( -R(uu)(t,x) drdt + L/'' / (us(t,x) - g(t, x)) drdt
(a us) =
+ 3 (u0(0,)) a(x)/ddx
The PDE loss consists of the physics loss in the interior and the data loss on the boundary and initial conditions, with haper-parameters a, 3 >> O. It can be generalized to variational form as in (Weinan & Yu 2018).
Challenges of PINN PINNs take advantage of the universal approximability of neural networks, but, in return, suffer from the low oowffrequecyy induced bias. Empirically, PINNs often fail to solve challenging PDEs when the solution exhibits high-freeqence
multi-scale structure (Wang et al.) 2021a 2020b Fuks & Tchelepi 2020: Raissi et al. 2020). Further, as an iterative solver, PINNs have difficulty propagating information from the initial condition boundary condition unseen parts of the interior or to future times (wwivedi & Srinivasan, 2020). For example, in challenging problems such as turbulence, PINNs are only able to solve the PDE on relatively small domain (Jin et al. 2021. or otherwise, require extra observational data which is not always available in practice (Raissi et al. 2020 Cai al, 202T). In this work, we propose overcome the challenges posed by the optimization by integrating operator learning with PINNs.
2.3 LEARNING THE SOLUTION OPERATOR (NEURAL OPERATOR)
An alternative setting is to learn the solution operator G. Given a PDE as defined in T or 2 and the corresponding solution operator Gt, can neural operator Go with parameters surrogate model to approximate Gt. Usually we assume a dataset {aj. M is available, where (ay) "y and aj Jı are i.i.d. samples from some distribution jr supported on A. In this case, one can optimize the solution operator by minimizing the empirical data loss on a given data pair
Lama (u, (o(a) = -I Co(a) = J. |u( (o(a)(
(5)
where we assume the setting of ①I) for simplicity of the exposition. The operator data loss is defined as the average error across all possible inputs
Jana (Go) || Gollig(Amo = Earya EdrralLdatal (a, 0))
1
Σ
(e) - (6)
(1y(x)
N
Similarly, one can define the operator PDE loss as
[Go) Ba-uull Ga(a))
In general, is non-trivial compute the derivatives (G)(a) (a) dr and (G /t for model Gp. In the following section, we will discuss how to compute these derivatives for Fourier neural operator.
2.4 NEURAL OPERATORS
In this work, we will focus on the neural operator model designed for the operator learning problem. The neural operator, proposed in (Li al. 2020c), is formulated as generalization of standard deep neural networks to operator setting. Neural operator compose linear integral operator with pointwise non-linear activation function approximate highly non-linear operators.
4

---

Under review as a conference paper at ICLR 2022
Definition (Neural operator Go) Define the neural operator
Go Q (WL oWW
(8)
where P - IRd Q Rdd - Rdd are the pointwise neural networks that encode the lower
dimension function into higher dimensional space and vice versa. The model stack layers of oW + +() where Wi E Rd4+1xd xd, are pointwise linear operators (matrices), K₁ {D + R - {D + Rd:+1] } are integral kernel operators, and a are fixed activation functions. The parameters consists of all the parameters in P, 2, Wi,Ke.
Recently, Li et al. (2020a) proposes the Fourier neural operator (FNO) that deploys convolution operator for K. In this case, it can apply the Fast Fourier Transform (FFT) to efficiently compute XC
This leads fast architecture that obtains state-of- state-of-the-art results for PDE problems.
Definition (Fourier convolution operator KC) Define the Fourier convolution operator
(Kee) (x) = F (FoFF)
where part of the parameter to be learn.
Challenges of Operator learning. Operator learning similar to the supervised learning in com- puter vision and language where data play a very important role. One needs to assume the training points and testing points follow the same problem setting and the same distribution. Especially, the previous FNO model trained on one coefficient (e.g. Reynolds number) or one geometry cannot be easily generalized to another. Moreover, for more challenging PDEs where the solver is very slow or the solver is even not existent, it is hard to gather representative dataset. On the other hand, since FNO doesn't use any knowledge of the equation, it cannot get arbitrary close to the ground truth by using higher resolution as conventional solvers, leaving gap of generalization error. These challenges limit FNO's applications beyond accelerating the solver. the following section, we will introduce the PINO framework overcome these problems by using the equation constraints.
PHYSICS-INFORMER NEURAL OPERATOR (PINO)
We propose the PINO framework that uses one neural operator model Gy for solving both operator learning problems and equation solving problems. It consists of two phases
Pre-train the solution operator: learn neural operator Go to approximate Gl using either/both the data loss Jdata and/or the PDE loss Tpde which can be additional to the dataset.
Test-time optimization: use (a(a) as the ansatz to approximate ul with the pde loss Epde and/or an additional operator loss Cop obtained from the pre-train phase.
3.1 PRE-TRAIN: OPERATOR LEARNING WITH PDE LOSS
Given a fixed amount of data {aj. U, ) the data loss Jaata offers stronger constraint compared to the PDE loss Jytee However, the PDE loss does not require a pre-existing dataset. One can sample virtual PDE instances by drawing additional initial conditions or coefficient conditions aj μ for training, as shown Algorithm In this sense, we have access the unlimited dataset by sampling new aj in each iteration.
3.2 TEST-TIME OPTIMIZATION: SOLVING EQUATION WITH OPERATOR ANSATZ
Given learned operator Go. similar to PINNs where it computes the PDE loss Lnde on a, except that we propose to use a neural operator instead of a neural network. Since the PDE loss is soft constraints and challenging to optimize, we also add an optional operator loss Lop (anchor loss) to bound the further optimized model from the pre-trained model
Ga(a) as the ansatz to solve for wt The optimization procedure
Lop (Go, (a). Go, (a)) |19o, (a) Go, (a)
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Pre-training scheme for Physics-informet neural operator learning
Data: Data: input output function pairs (aj, و
Result: Neural operator A - U. for i 0, 1.... do
Compute Ldata and Ldde on (a,, u.). Update neural operator G; j to K do
Sample a' from distribution μi;
Compute Lyde on a'. Update neural operator 9;
end
end
where Go. (a) is the model at 7th training epoch. We update the operator Gg using the loss Cyde + &e It is possible to further apply optimization techniques fine-tune the last fewer layers of the neural operator and progressive training that gradually increase the grid resolution and use finer resolution in test time.
Optimization landscape. Using the operator as the ansatz has two major advantages: (1) PINN does point-wise optimization, while PINO does optimization in the space of functions. In the linear integral operation K., the operator parameterizes the solution function Optimization of the set of coefficients and basis is easier than just optimizing single function as in PINNs. (2) we can learn these basis functions in the operator learning phase which makes the later test-time optimization even easier. In PINO, we do not need to propagate the information from the initial condition and boundary condition to the interior. It just requires fine-tuning the solution function parameterized by the solution operator.
sum of the basis function.
Trade-off (1) complexity and accuracy: the test-time optimization is an option to spend more computation in exchange of better accuracy. When the speed is most desired, one should the pre-trained model to directly output the prediction. However, if one wants to solve for specific instance accurately, then the person can use test-time optimization to correct the generalization error posed in operator learning. (2) resolution effects optimization landscape and truncation error: using higher resolution and finer grid will reduce the truncation error. However, it may make the optimization instable. Using hard constraints such as the anchor loss Lop relieve such problem.
3.3 DERIVATIVE OF NEURAL OPERATORS
In order to use the equation loss Lyde- one of the major technical challenge is to efficiently compute the derivatives D(Gaa) = ((Gaa)(x)) (x) for neural operators, In this section, we discuss three efficient methods to compute the derivatives of neural operator Go defined in
Numerical differentiation. A simple but efficient approach is to use conventional numerical gradients such finite difference and Fourier gradient (Zhu al.) [219 Gao et al., (021). These numerical gradient are fast and memory-efficient: given re-oints grid, finite difference requires (n) and Fourier method require (n log However, they face the same challenges as the corresponding numerical solvers: finite difference methods require fine-resolution uniform grid; spectral methods require smoothness and uniform grids. Especially. These numerical errors on the gradient will be amplified on the output solution.
Autograd. Similar to PINN RRaisii et al., 2019), the most general way to compute the exact gradient to use the auto-differnntiation library neural networks (autograd). To apply autograd, one needs to use neural network to parameterize the solution function å + u(x). However, it is not straightforward to write out the solution function in the neural operator which directly outputs the grid, especially for FNO which uses FFT. To apply autograd,
numerical solution Go (a) design a query function à that input v and output u(x)). Recall Go == Q (W₂ + KL) ........ ( W K.)) and Gea Que Q(W, + Kr, .... Since @ is pointwise.
u(x) (x) 2 ((W2v)))) + (LVL-1(F))
(10)

---

Under review as a conference paper at ICLR 2022
10a1
100
relobive 12 error
10:1
relative
10/2
10
PINO
PINO
PINO (test-time opt)
PINO (test-time opt)
PINN
solver
10
solver
2
10
10 10
10
10
100
10a
10a
runtime
runtime
(a) long-temporal transient flow
(b) Kolmogorov flow
(a) the long-tempone transient flow with Re ~ 20, = 50. PINO output the full trajectory in one step, which leads to 400x speedup compared to the solver. PINN cannot converge to reasonable rate due the long time window. (b) the chaotic Kolmogorov flow with Re =~ 500, T 0.5. PINO retains 10x speed compared to the solver, while converges much faster compared to PINN.
Figure 2: The aceuracy-complexin trade-off on PINO, PINN, and the pseudo-spectral solver.
To query any location v, we need evaluate the Fourier convolution KL (x) and the bias part (Wzz)=1)(z) For the first part, we can directly query the Fourier coefficient R Fu, at spatial location r without doing the inverse FFT F. For the second part, we can define the query function as an interpolation or a low-rank integral operator as in (Kovachki et al. (021). Then we can use autograd to take the exact derivative for (TQ). The autograd method general and exact, however, less efficient. Since the number of parameters ( is usually much greater than the grid size the numerical methods are indeed significantly faster. Empirically, the autograd method is usually slower and memory-consuminn
Exact gradients. We develop efficient and exact gradient method based the architecture of the neural operator. The idea is similar to the autograd, but we explicitly write out the gradient on the Fourier space and apply the chain rule. Given the explicit form (TO). D(Kr, can be directly compute on the Fourier space. The linear part D(W can be interpolated using the Fourier method. Therefore to exactly compute the derivative of uz. one just needs run the numerical Fourier gradient. Then we just need to apply chain rule for Jacobian Ju J(Q o wr.) J J(Q(vr)) )...))))))) (wz) and Hessian Hu ( o v1.) = J(I.) o ((((v))) o ((r.) + (()())) o H (vz). Since Q is pointwise, this can be much simplified. In the experiments, mostly use the exact gradient and the numerical Fourier gradient.
4 EXPERIMENTS
In this section, we conduct empirical experiments to examine the efficacy the proposed PINO. In1 we show the physics constraints helps pperator-learning with fewer data and better generalization. Then in 4.2] we investigate how PINO uses operator ansatz. to solve harder equations with improved speed and accuracy. We study on three concrete cases of PDEs on Burgers' Equation, Darcy Equation, and Havier-Stoker equation.
Burgers' Equation. The 1-d Burgers' equation a non-linear PDE with periodic boundary condi- tions where up ((0, 1); R) is the initial condition and 0.01 is the viscosity coefficient. We aim to learn the operator mapping the initial condition to the solution, gt uo + u|[0.11)
a,l( 1) (u (x. f)/ u t),
E (0, 1). (0, = (0,
(11)
u(z,00 0) = (x).
Darey Flow. The 2-d steady-state Darey Flow equation the unit box which is the second order linear elliptic PDE with a Dirichlet boundary where E L (00, is a piecewise constant diffusion coefficient and f is fixed forcing function. We interested in learning the operator
7

---

Under review as a conference paper at ICLR 2022
mapping the diffusion coefficient to the solution, G ++ Note that although the PDE is linear, the operator G is not.
(a(x) = f(x) u(x) 0
x E (0, 1)2 I E a(0, 1)2
(12)
Navier-Stokes Equation. We consider the 2-d Wavier-Stokes equation for a viscous, incompress- ible fluid in vorticity form the unit torus, where E C(10, T); App ((0, ()); R²)) for any > 0 is the velocity field, x u is the vorticity, wo E par ((0, :)) the initial vorticity, E R+ the viscosity coefficient, and f Lper ((0, 1)2; R) is the forcing function. We want to learn the operator mapping the vorticity from the initial condition to the full solution G wo + w wlo.
O,w(a, t) + u(x,t) Vw(a, t) x) + f(=),
I E (0, (0,
V u(u 1) = O, w(x,0) = wo(x),
x E (0,1),,E [0,
(13)
x E (0,1)²
Specially, we consider two problem settings:
Long temporal transient flow:
we study the build-up of the flow from the initial condition to near-zero velocity to ur that reaches the ergodic state. We choose / E [0, 50), - 1, Re = 20 as in
Li al. 2020a). The main challenge is to predict the long time interval.
Chaotic Kolmogorov flow: In this case u lies in the attractor where arbitrary starting time lg. We choose t E [tu to + 0.5] or [to. to + 1]. = 1, Re = 500 similar to Li et al. (2021). The main challenge is capture the small details that evolve chaotically.
4.1 OPERATOR LEARNING WITH PDE LOSSES
We first show the effectiveness of applying the physics constraints Jpde to learn the solution operator.
Burgers equation and Darcy equation. PINO can learn the solution operator without any data on simpler problems such as Burgers and Darcy. Compared to other PDE-constrained operators, PINO is more expressive and thereby achieves better accuracy. On Burgers (TT). PINN- DeeN-eeepONet achieves 1.38% (Wang al. 2021b); PINO achieves 0.37%. Similarly, on Darcy flow (12). PINO outperforms FNO by utilizing physics constraints, as shown in Table [ For these simpler equation, test-time optimization may not be needed. The implementation detail and and the search space of parameters are included in the Appendix A. and A.2
o
4.2
SOLVE EQUATION USING OPERATOR ANSATZ
Long temporal transient flow. is extremely challenging to propagate the information from the initial condition to future time steps over such a long interval T [0, 50) just using the soft physics constraint. None of the PINN, DeepONet, and PINO (from scratch without pre-training) can handle this case (error 50%), no matter solving the full interval at once solving per smaller steps. However, when the pre- data is available for PINO, we can use the learned neural operator ansatz and the anchor loss Lop- The anchor loss is hard constraint that makes the optimization much easier. Providing N = 4800 training data, the PINO without test-time optimization achieves 2.87% % error, lower than FNO 3.04 % and retains 400x speedup compared to the GPU-based pseudo-spectral solver (He & Sun 2007), matching FNO. Further doing test time optimization with the anchor loss and PDE loss, PINO reduces the error 1.84%.
Chaotic Kolmogorov flow. We show an empirical study on how PINO can improve the generaliza- tion of neural operators by enforcing more physics. For this experiment, the training set consists of 4000 data points of initial condition and corresponding solution. Using Algorithm|] [ we can sample unlimited additional initial conditions from Gaussian random field at any resolution. Table 2 and compare the generalization error of neural operators trained by different schemes and different amounts of simulated data. The result shows that training neural operator with additional PDE instances consistently improves the generalization error all three resolutions we evaluating.
8

---

Under review as a conference paper at ICLR 2022
30-
20
20
50
20
20
0-
30
30-
30-
60
60
60
(4) ground truth
(b) forward PINO
(c) backward PINO
(d) solver MCMC
Figure 3: Darcy inverse problem
Based on the solution operators learned in the above operator-learning section, we continue to do test-time optimization. The results are shown in Figure and Table 3 Overall, PINO outperforms PINN by 20x smaller error and 25x speedup. Using pre-trained model make PINO converge faster. The implementation detail and the search space of parameters included in the Appendix B.2
Transfer Reynolds numbers. The extrapolation of different parameters and conditions is one of the biggest challenges for ML-based methods. By doing test-time optimization, the pre-trained PINO model on the Kolmogorov flow can easily transferred to different Reynolds numbers ranging from 100 to 500 in test-time optimization as shown in Table Such property envisions broad applications.
4.3 INVERSE PROBLEM
One of the major advantages of the physics-informed method to solve the inverse problem. this case, investigate PINO the inverse problem of the Darcy equation to recover the coefficient function at from the given solution function w². We assume a dataset {aj. uý} is available to pre-train the operator. We propose formulations of PINO for the inverse problem:
Forward model: Learn the forward operator Go a + u with data. Initialize to approximate at. Optimize à using
Iforward == Lpde (ā, ut) Lidat (Go(a) + R(ā).
with data. Use Fo()) approximate Backward model: Learn the backward operator Fo at. Optimize Fg using
Tbackward == Lpde (Fo(u))) + Lop (ul) Fo (w')) R(Fo(u))
Where R(a) is the regularization term. We the PDE loss Lpde deal with the small error in Go and the ill-defining issue of Fo.
We perform showcase example the Darcy equation, where
given the full-field observation of u (clean data) to recover the coefficient function a. The coefficient function a is piecewise constant (representing
types of media), the inverse problem can be viewed classification problem. We define R(a) as the total variance. As shown in Figure 3 the PINO backward model has the best performance: it has 2.29 % relative 12 eгтог on the output and 97.10% classification accuracy on the input the forward model has 6,43% еттоr on the output and 95 38% accuracy on the input. As reference, we compare the PINO inverse frameworks with PINN and the conventional solvers using the accelerated MCMC method with 500,000 steps (Cotter et al. (013). The posterior mean of the MCMC has 4.52% error and 90.30% respectively (Notice the Bayesian method outputs the posterior distribution, which is beyond obtaining maximum posteriori estimation) Meanwhile, PINO methods are 3000x faster compared to MCMC. PINN does not converge this case. The major advantage of the PINO backward model compared to the PINO forward model is that it uses neural operator F (²)) as the ansatz for the coefficient function. Similar to the forward problem, the operator ansatz has easier optimization landscape while being expressive.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Charles-Henri Bruneau and Mazen Saad. The 2d lid-driven cavity problem revisited. Computers & fluids, 3563)332--348, 2006.
Oscar P Bruno, Youngae Han, and Matthew M Pohlman. Accurate, high-order representation of complex three-dimensional surfaces via fourier continuation analysis. Journal of computational Physics, 227(2):094-11155 2007.
Oscar P Bruno, Jan S Hesthaven, and Daniel V Leibovici. Fe-based shock-dynamics solver with neural-network localized artificial-viscosity assignment. arXiv preprint vrXiv:2111.01311 2021.
Steven L Brunton, Bernd R Noack, and Petros Koumoutsakos, Machine learning for fluid mechanics. Annual Review of Fluid Mechanics, 52:477-508, 2020.
Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics- informed neural networks (pinns) for fluid mechanics: A review. arXiv preprint arXiv:2105.09506 2021.
Simon L Cotter, Gareth O Roberts, Andrew M Stuart, and David White. Mcme methods for functions: modifying old algorithms to make them faster. Statistical Science, 28(3):424-444 2013.
James Duvall, Karthik Duraisamy, and Shaowu Pan. Non-linear independent dual system (nids) for peteretization-ndeeeeneeee surrogate modeling over complex geometries, 2021.
Vikas Dwivedi and Balaji Srinivasan. Physics informed extreme learning machine (pielm)-a rapid method for the numerical solution of partial differential equations. Neurocomputing, 391:96-118 2020.
Yuwei Fan, Lin Lin. Lexing Ying, and Leonardo Zepeda-Nínez. A multiscale neural network based on hierarchical matrices. arXiv preprint arXiv:1807.01883, 2018.
Olga Fuks and Hamdi A Tchelepi. Limitations of physics informed machine learning for nonlinear two-phase transport in porous media. Journal of Machine Learning for Modeling and Computing. 1(1), 2020.
Han Gao, Luning Sun, and Jian-Xun Wang. Phygeonet: physics-informed adoaetry-adaptive convolutional neural networks for solving parameterized steady-state pdes on irregular domain. Journal of Computational Physics, 428: 10079, 2021.
Daniel Greenfeld, Meirav Galun, Ronen Basri, Irad Yavneh, and Ron Kimmel. Learning to optimize multigrid pde solvers. In International Conference on Machine Learning, pp. 2415-2422 PMLR, 2019.
Jiequn Han, Arnulf Jentzen, and E Weinan. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505-8510, 2018.
Yinnian He and Weiwei Sun. Stability and convergence of the cran--nickkko/adans--aaahfcc scheme for the time-dependent navier-stokes equations. SIAM Journal on Numerical Analysis, 45 (2):837-869, 2007.
Ameya D Jagtap, Kenji Kawaguchi, and George Em Karniadakis. Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks. Proceedings of the Royal Society A, 476(2239):20200334 2020.
Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426: 109951, 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 2014.
Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang. Michael P Brenner, and Stephan Hoyer. Machine learning accelerated computational fluid dynamics. arXiv preprint arXiv:2102.01000 2021.
10

---

Under review as a conference paper at ICLR 2022
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXh::2108.0844 2021.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations, 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations, 2020b.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03448 2020c.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Markov neural operators for learning chaotic systems. arXiv preprint arXiv:2106. 000..2800600888, 2021.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In International Conference on Machine Learning, pp. 3208-3216. PMLR, 2018.
Denghui Lu, Han Wang, Mohan Chen, Lin Lin, Roberto Car, E Weinan, Weile Jia, and Linfeng Zhang, 86 pflops deep potential molecular dynamics simulation of 100 million atoms with ab initio accuracy. Computer Physics Communications, 259:107624, 2021a.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv 2019.
preprint
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. arXiv preprint arXiv:2111.055112 2021b.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. DeepXDE: A deep learning library for solving differential equations. SIAM Review, 63(1):208-23) 228, 2021c. doi: 10.1137/ 19M1274067.
Levi McClenny and Ulisses Braga-Neto. Self-adaptive physics-informed neural networks using soft attention mechanism. arXiv preprint S.Kiv::0009005544 2020.
Ravi G Patel, Nathaniel A Trask, Mitchell A Wood, and Eric C Cyr. A physics-informed operator regression framework for extracting data-driven continuum models. Computer Methods Applied Mechanics and Engineering, 13500, 2021.
Jaideep Pathak, Mustafa Mustafa, Karthik Kashinath, Emmanuel Motheau, Thorsten Kurth, and Marcus Day. MI-pde: A framework for machine learning enhanced pde solver. Bulletin of the American Physical Sociery, 2021.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informe neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, 367(6481) 1026-113000 2020.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339-1164 2018.
Jonathan Smith, Zachary Ross, Kamyar Azizzadenesheli, and Jack Muir. Hyposvi: Hypocenter inversion with stein variational inference and physics informed neural networks. arXiv, 2021.

---

Under review as a conference paper at ICLR 2022
Luning Sun, Han Gao, Shaowu Pan, and Jian-Xun Wang. Surrogate modeling for fluid flows based deep learning without simulation data. Computer Methods in Applied Mechanics and Engineering, 361:112732, 2020.
on
Kiwon Um, Philipp Holl, Robert Brand, Nils Thuerey, al. Solver-in-the-loop: Learning from differentiable physics to interact with iterative pde-solvers. arXiv preprint arXiv:2007.000/1 2020.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1457-1466, 2020a.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. arXiv preprint arXiv:2007.14527, 2020b.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks, SIAM Journal on Scientific Computing, 4365533305 2021a.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-
physics-informed informed deeponets. arXiv preprint arXiv:2103. 109744 202
E Weinan and Bing Yu. The deep ritz method: deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1-12, 2018.
Lulu Zhang, Zhi-Qin John Xu, and Yaoyu Zhang. Data-informed deep optimization. arXiv preprint arXiv:2107.08166 2021.
Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelios Koutsourelakis, and Paris Perdikaris. Physics- constrained deep learning for high- high-dimensional surrogate modeling and uncertainty quantification without labeled data. Journal of Computational Physics, 394:56-81, 2019.
12

---

Under review as a conference paper at ICLR 2022
ground truth
prediction
(a) KF: vorticity
(b) Cavity: velocity in x,
Figure 4: PINO on Kolmogorov flow (left) and Lid-cavity flow (right)
A IMPLEMENTATION DETAILS
In this section, we list the detailed experiment setups and parameter searching for each experiment in Section 4 Without specification, we use Fourier neural operator backbone with width 64, mode 8, = 4 and GeLU activation. The numerical experiments are performed on Nvidia V100 GPUs
A. BURGERS EQUATION
We use the 1000 initial conditions ug μ where μ (0, 625(-2255-) + 2 2) to train the solution operator on PINO with width = 64, mode = 20, and GeLU activation. We use the numerical method to take the gradient. We use Adam optimizer with the learning rate 0.001 that decays by half every 100 epochs. 500 epochs in total. The total training time is about 1250s on a single Nvidia 3090 GPU. PINO achieves 0.37% relative 12 error averaged over 200 testing instances. PINN-DeepONet achieves 1.38 % which taken from Wang et al. (2021b) which uses the same problem setting.
A.2 DARCY FLOW
We use the 1000 coefficient conditions a to train the solution operator where a ~ iI where (0, (-A 9/) -?), (a(x)) 12 if a(x) 2 0; (<((2)) 3 if a(x) < 0. The zero boundary condition is enforced by multiplying a mollifier m(x)
(in()T) sin(Ty) for all methods. The parameter of PINO Darcy Flow is the same as in the Burgers equation above. Regarding the implementation detail of the baselines: as for FNO, we use the same hyperparameters as its paper did (Li et al. (200cc:: DeepONet (Lu al. 2019) did not study Darcy flow
grid search the hyperparameters of DeepONet: depth from 2 to 12, width from 50 to 100. The best result of DeepONet is achieved by depth 8, width 50. The results shown Table
Method
Solution 6.97 = 0.09%
Equation
DeepONet with data (Lu et al. 2019
FNO with data (Li et al. 2020c
1.98 + 0.05%
1.3645 + 0.014
PINO with data
1.22 = 0.03%
0.5740 + 0.008
PINO w/o data
1.50 = 0.03%
0.4868 + 0.007
Table 1: Operator learning on Darey Flow.
13

---

Under review as a conference paper at ICLR 2022
A.3 LONG TEMPORAL TRANSIENT FLOW.
We study the build-up of the flow from the initial condition uo near-zero velocity to ur that reaches the ergodic state. We choose T 50,
in Li et al. (2020a). We choose the weight parameters of error a = B 5. The initial condition wo( is generated according to wo ~ where = (0, (-A + 491) with periodic boundary conditions. The forcing is kept fixed f(x) (.1 ( (2] + x2)) + cos(2T((1) + x2))). We compare FNO, PINO (no test-time optimization), and PINO (with test-time optimization). They get 3.04%, 2.87% and 1.84% relative 12 error on the last time step u50) over 5 testing instance.
# data samples
# additional PDE instances
Solution error
Equation error
0
100k
74.36%
0.3741
0.4k
0
33.32%
1.8779
0.4k
40k
31.74%
1.8179
0.4k
160k
31.32%
1.7840
4k
0
25.15%
1.8223
4k
100k
24.15%
1.6112
4k
400k
24.22%
1.4596
Table 2: Operator-learning Kolmogorov flow Re = 500. Each relative La test error is averaged over 300 instances, which evaluated with resolution 128 128 x 65. Complete results other resolutions are reported in Table in appendix.
A.4 CHAOTIC KOLMOGOROV FLOW.
For this experiment, u lies in the attractor. We choose T = 0.5 or 1, and / = 1, similar to Li al. (202T). The training set consists of 4000 initial condition functions and corresponding solution functions with spatial resolution of 64 x 64 and a temporal resolution of 65. Extra initial conditions are generated from Gaussian random field / (o, (-/2 + 497)-5/2) We estimate the generalization error of the operator and reports the averaged relative L2 eπror. Each neural operator is trained with 4k data points plus number of extra sampled initial conditions. The Reynolds number in this problem is 500. The reported generalization етror is averaged over 300 instances.
test set that contains 300 instances of Kolmogorov flow
Comparison study. The baseline method PINN is implemented using library DeepXDE (Lu et al. 2021ch with TensorFlow as backend. We use the two-step optimization strategy (Adam (Kingma Ba 2014) and L-BFGS) following the same practice as NSFNet (Jin et al.) 2021). which applics PINNs to solving Navier Stokes equations. We grid search the hyperparameters: network depth from 4 to 6, width from 50 to 100, learning rate from 0.01 to 0.0001, the weight of boundary loss from 10 to 100 for all experiments of PINNs. Comparison between PINO and PINNs test-time optimization.
The results are averaged over 20 instances of the Navier Stokes equation with Reynolds number 500. The best result is obtained by PINO using pre-trained operator ansatz and virtual sampling. The neural operator ansatz used here is trained over set of 400 data points. The authors acknowledge that there could exist more sophisticated variants PINN that performs better in our test cases.
Method data samples
additional PDE instances
Solution error (w)
Time cost
PINNs
18.7%
4577s
PINO
0
0
0.9%
608s
PINO
4k
0.9%
536s
PINO
4k
160k
0.9%
473s
Table 3: Equation-solving on Kolmogorov flow Re 500.
A.5 TRANSFER LEARNING ACROSS REYNOLDS NUMBERS
We study the test-time optimization with different Reynolds numbers on the Is Kolmogorov flow. For the higher Reynolds number problem Re 500, 400, the pre-training operator shows better
14

---

Under review as a conference paper at ICLR 2022
data samples
# additional PDE instances
Resolution 128x
Solution error
Equation error
33.32%
1.8779
400
0
64 x 64 x 65
33.31%
1.8830
32 x 32 x 33
30.61% %
1.8421
128x128x65
31.74%
1.8179
400
40k
x 65
31.72%
1.8227
32 x 32 x 33
29.60%
1.8296
128x128x65
31.32% %
1.7840
400
160k
64 x 64 x 65
31.29 %
1.7864
32 x 32 x 33
29.28%
1.8524
128 128 65
0.2515
1.8223
4k
0
64 x 64 x 65
0.2516
1.8257
x
0.2141
1.8468
128 128 65
0.2415
1.6122
4k
100k
64 x 64 x 65
0.2411
1.6159
32 x 32 x 33
0.2085
1.8251
128 x 128 x 65
0.2422
1.4596
4k
400k
64 x 64 65
0.2395
1.4656
32 x 32 x 33
0.2010
1.9146
128 x 128 65
0.7436
0.3741
0
100k
64 x 64 x 65
O. 0.7438
0.3899
32 x 32 x 33
0.7414
0.5226
Table 4: Each neural operator is trained with 4k data points additionally sampled free initial conditions. The Reynolds number is 500. The reported generalization error is averaged over 300 instances. Training on additional initial conditions boosts the generalization ability the operator.
convergence accuracy. In all cases, the pre- vre-training operator shows better convergence speed as demonstrated in Figure 5 The results are shown in Table 5 where the error is averaged over 40 instances. Each row is testing case and each column a pre-trained operator.
Testing RE
From scratch
100
200
250
300
350
400
500
500
0,0493
0,0383
0.0393
0.0315
0.0477
0.0446
0.0434
0.0436
400
0.0296
0.0243
0.0245
0.0244
0,0300
0.0271
0.0273
0,0240
350
0.0192
0.0210
0.0211
0.0213
0.0233
0.0222
0.0222
0.0212
300
0.0168
0.0161
0.0164
0.0151
0.0177
0.0173
0.0170
0.0160
250
0.0151
0.0150
0.0153
0.0151
0.016
0.0156
0.0160
0.0151
200
0.00921
0.00913
0.00921
0.00915
0.00985
0.00945
0.00923
0.00892
100
0.00234
0.00235
0.00236
0.00235
0.00239
0.00239
0.00237
0.00237
Table 5: Reynolds number transfer learning. Each row is test set of PDEs with corresponding Reynolds number. Each column represents the operator ansatz we use as the starting point of test-time optimization. For example, column header '100' means the operator ansatz is trained over a set of PDEs with Reynolds number 100. The relative L2 errors is averaged over 40 instances of the corresponding test set.
B ADDITIONAL EXPERIMENTS
B. ADDITIONAL BASELINES
We add comparison experiment against the Locally adaptive activation functions for PINN (LAAF- PINN) (Jagtap et al. 2020) and Self-Adaptive PINN (SA-PINN) (McClenny & Braga Neto 2020). For the Kolmogorov flow problem, set Re=500, T=[0, 0.5]. We search among the following hyperparameters combinations: LAAF-PINN: n: 10. 100, learning rate: 0.1, 0.01, 0.001, depth 4, SA-PINNs: learning rate 0.001, 0.005, 0.01, 0.05, network width 50, 100, 200, depth 4, 6, 8.
15

---

Under review as a conference paper at ICLR 2022
0.30
from scratch
from
0.25
from 200
from 250
Relative L2 error
0.20
from 300
from 350
0.15
from 500
0.10
0.05
0.00
2000
3000
4000
5000
7000
8000
Steps
Figure 5: Plot of relative L2 error versus update step for the Kolmogorov flow with Re500, T=ls. We observe that ll the operator ansatzs pretrained over PDE instances with different Reynolds number can boost the test-time optimization accuracy and speed compared training from scratch.
As shown in Figure 6 both LAAF-PINN and SA-PINN converge much faster compared to the original PINN method, but there is still big gap with PINO. LAAF-PINN adds learnable parameters before the activation function; SA-PINN adds the weight parameter for each collocation point. These techniques help to alleviate the PINNs optimization problem significantly. However, they didn't alter the optimization landscape effectively in the authors' opinion. On the other hand, by using operator ansatz, PINO optimizes function-wise manner where the optimization fundamentally different.
Note that the contribution of PINO is orthogonal to the above methods. One can apply the adaptive activation functions or self-adaptive loss in the PINO framework too. All these techniques PINNs can be straightforwardly transferred to PINO. We believe it would be interesting future directions to study how all these methods work with each other in different problems.
B.2 LID CAVITY FLOW.
We demonstrate addition example using PINO solve for lid-cavity flow T [5, 10 with
Re = 500. In this case, we do not have the pre-train phase and directly solve the equation (test-time optimization). We PINO with the velocity-pressure formulation and resolution 65 65 50 plus the Fourier numerical gradient. It takes 2 minutes to achieve a relative error of 14.52%. Figure shows the ground truth and prediction of the velocity field predicts the ground truth.
10 where the PINO accurately
We assume no-slip boundary where u(x, (0, 0) left, bottom, and right walls and u(x: t) (1. 0) on top, similar to Bruneau & Saad (2006). We choose t E [5, 10), /= = 1. Re = 500. We use the velecity-pressure formulation as Jin et al. (2021) where the neural operator output the velocity field in æ, y. and the pressure field. We set widhh = 32, mode = 20 with learning rate 0.0005 which decreases by half every 5000 iterations, 5000 iterations in total. We use the Fourier method with Fourier continuation to compute the numerical gradient and minimize the residual error on the velocity, the divergence-free free condition, as well as the initial condition and boundary condition. The weight parameters (a, 35 between different error terms are all chosen 1. Figure shows the ground truth and prediction of the velocity field at = 10 where the PINO accurately predicts the ground truth.
16

---

Under review as a conference paper at ICLR 2022
0.8
Solver
PINO (pretrained)
0.7
PINO (test-time opt)
0.6
PINN
Relative L2 error
LAAF-PINN
0.5
SA-PINN
0.4
0.3
0.2
0.1
0.0
10
10
10'
102
Runtime(s)
Solver
PINO (pretrained)
0.5
PINO (test-time opt)
PINN
Relative L2 error
0.4
LAAF-PINN
SA-PINN
0.3
0.2
0.1
0.0
10
10
10' Runtime(s)
102
10
Figure 6: Plot of test relative L2 error versus runtime step for the Kolmogorov flow with Re500, T=0.5s. top: 64 x 64, bottom: 128 x 128. Averaged over 20 instances.
17

---

Under review as a conference paper at ICLR 2022
Input
padded input
(c) padded output
(d) target output
Figure 7: Fourier Continuation by padding zeros. The x-axis is spatial dimension; the y-axis the temporal dimension. FNO extends the output smoothly on the padded domain.
C FOURIER CONTINUATION
The Fourier neural operator can be applied to arbitrary geometry via Fourier continuations. Given any compact manifold M, we can always embed it into periodic cube (torus),
where we can do the regular FFT. Conventionally, people would define the embedding i as continuous extension by fitting polynomials (Bruno et al. (007). However, in Fourier neural operator, it can be simply done by padding zeros the imput. The loss is computed the original space during training. The Fourier neural operator will automatically generate smooth extension to do padded domain in the output, as shown Figure
This technique is first used in the original Fourier neural operator paper (Li et al. 2020a) to deal with the time dimension the Navier-Stokes equation. Similarly, [u et al. (202 apply FNO with extension and interpolation continuation widely for non-perinee non-periodic boundary conditions (Darcy, time dimension). We also added example of lid-cavity demonstrate that PINO can work with non-periodic boundary conditions.
diverse geometries the Darcy equation. In the work,
Fourier
Furthermore, this Fourier continuation technique helps to take the derivatives of the Fourier neural operator. Since the output of FNO is always periodic domain, the numerical Fourier gradient is usually efficient and accurate, except if there is shock (in this case, we will use the exact gradient method).
D CONCLUSION AND FUTURE WORKS
In this work, develop the physics-inforen informed neural operator (PINO) that bridges the gap between physics-informed optimization and data-driven neural operator learning. We introduce the pre-training and test-time optimization schemes for PINO to utilize both the data and physics constraints. In the pre-training phase, PINO learns an operator ansatz over multiple instances of a parametric PDE family. The test-time optimization scheme allows to take advantage of the learned neural operator ansatz and solve for the solution function on the querying instance faster and more accurately.
There many exciting future directions. PINO can be used enhance operator-learning with some constraints, for examples: (1) Control problem: one first learns a solution operator (pre-training) and does test-time optimization the control phase make sure satisfies desired constraints (Hwang et. al.). (2) Multiscale modeling: in multi-scale modeling, one learns fine-scale solution operator, which then regulated by a coarse-scale solver. Note that the fine-scale problems do not follow Gaussian distribution
in the training dataset (Liu et. al.). (3) Weather prediction: beyond the observation dataset, one can add physics constraints such as the conservation of mass (Jiang et. al.).
inverse problem shown in Section 4.3. PINO backward formulation is particularly effective for the inverse problem (Thuerey et. al.).
(4) Airfoil design: the design problem can be formulated
PINO is also designed to work most of the working scenarios of PINNs: if the optimization is benign, then pre-train is not needed. The major advantage compared to PINN-based methods is the optimization landscape. The operator ansatz, even randomly initialized, still has much better
18

---

Under review as a conference paper at ICLR 2022
optimization landscape compared to the PINN-based model. Most of the techniques and analysis of PINN can transferred to PINO.
It is interesting to ask how to overcome the hard trade-off of accuracy and complexity, and how the PINO model transfers across different geometries. Furthermore, we can develop software library of pre-trained models. PINO's excellent extrapolation property allows it to be applied broad set of conditions, as shown in the Transfer Reynold's 's number experiments.
19


Reviews:
[{"review_id": "qxvfg-djsQY", "reviewer": "Reviewer_oyPM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Given the claims of novelty is correct (that this paper is the first one proposing the combination of PINN and FNO) (I am not that familiar with the relevant latest literature there), I think this is a great paper with both technical contribution and strong emperical result.\n\nStrength: The idea to combine these two mainstream pde solver is, although simple, very effective as shown from the emperical results. The only technical difficulty of combining the two algorithm (test time PINN-based optimization for FNO) that I see is the derivative calculation for neural operator space, where the authors used exact gradient calculation coded by chainrule. The loss landscape \n\nWeakness/Questions: Some of the questions and rooms for potential improvement:\n1. Fig 2c the colors are very hard to recognize on the legend. \n2. Did the PINN benchmark in this work include the adaptive activation function? (\"Adaptive activation functions accelerate convergence in deep and physics-informed neural networks\" Jagtap & Karniadakis.) \n3. According to Table 2 for the Kolmogorov flow, the PINO have 0 solution error improvement with increase in both number of data samples and PDE instances. Although the test time optimization convergence is faster, didn't it take much longer time to pre-train in these two cases? \n4. The clarity for the section 4.3 inverse problem needs to be improved (maybe due to page limit?). For the forward model method, how many initializations of \"a\" were tried? What was the R(a) regularization term's underlying assumption there?", "summary_of_the_paper": "The authors presented a novel algorithm to solve PDEs by combining two mainstream PDE solving deep neural network genre, the PINN and the FNO method into PINO. Which is basically doing these two steps sequentially by first learning the neural operator and then do a optimization step using PINN method.", "summary_of_the_review": "I think the technical and algorithmic contribution is solid ,the emperical results are good, and (if the novelty is good) I think this is a publishable paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bDLdQE6huIz", "reviewer": "Reviewer_NBhE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " I have read the other reviews and responses from the authors. Two highly-related papers should be added. So the claims and experiments need major revisions. I lower my score to 5. \n===========================================\n\nThe paper is well-written and organized. The experiments are convincing. I have the following two main concerns, which prevents me from improving the score.\n1. For a PDE solver, the data is specialized for the PDE. So the whole solving process should include the pre-training stage.\n2. For vision tasks, using the pre-trained models to improve the accuracy and efficiency of the downstream tasks is a promising way. These vision tasks share the features from images. However, for a given PDE solving problem, when one can use the pre-trained model? Do we need to train a lot of pre-trained models?", "summary_of_the_paper": "The paper proposes the physics-informed neural operator (PINO). It combines the operating-learning and function-optimization\nframeworks, which improves convergence rates and accuracy over traditional  methods. Experiments test the advantage of PINO.", "summary_of_the_review": "The paper is clear and the proposed method is reasonable. It has the weaknesses on the use of pre-trained models for PDE solving. However, I tend to accept the paper for its effectiveness.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H8yUzxiUz3b", "reviewer": "Reviewer_Tav6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Update after author response: The author's responses do not address my concern on the novelty w.r.t. previous papers. Each of the 4 new aspects w.r.t. the previous method is too incremental given the standard knowledge in our communities. Each point is done in other papers without marking them as a contribution of the paper. Also, the comparison should be done with the comparable previous methods as well as better variants of PINNs, as I mentioned in my review. Accordingly, I cannot raise the score to be fair with other papers. I support this direction and I would personally like to recommend acceptance if there is no almost identical idea in the previous paper. However, there is an almost identical idea in the previous paper and I cannot recommend this for acceptance for that reason and additional reasons, including but not limited to the lack of fair comparisons with variants of PINNs. \n\nThe proposed method makes sense as a transfer training method for PINNs, with many source tasks and a single target task, although it is limited for the transfer within each specific parametric PDE family. Here, note that this can be viewed as a type of transfer “training” (which is to accelerate training), instead of transfer “learning” (which is designed to accelerate training and generalization). \n\nIn this regard, I like the paper and I think that the proposed method has a potential to be used as a transfer training method for PINNs.\n\nHowever, I have several doubts. First of all, this seems to be too incremental given the following previous work: \n\n[1] Sifan Wang, Hanwen Wang, Paris Perdikaris. 2021. Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets. \n\nThe authors should elaborate the difference more. If we replace DeepOnet by FNO in the previous method, does it result in something very similar to PINO?  \n\nMoreover, if we know the PDE constraints, I don’t understand the point of learning the solution operator with the PDE constraints in the first phase. If we already know the PDE constraints, we can solve for the solution, without learning the solution operator, for each instance of the PDE family. Thus, we are introducing additional, potentially unnecessary step, in most cases. The authors should rewrite the paper much more carefully to organize the problem setting with what is given and what is not. \n\nFrom the current presentation, I infer that mathematically, this problem setting is nothing different from that of PINNs for the forward problems, if we don’t have the additional data for the operator learning phase. In this case, from the abstract view point, the proposed method is equivalent to (1) change the architecture from PINNs to FNO for the solution operator, and (2) add additional the pre-training step to train the solution operator for the PDE family, instead of one instance from the PDE family. Thus, the experiments are not convincing, because the performance improvements can be coming from the change of the architecture and the pre-training, which is expected and trivial. The authors mention that PINO is more expressive and thereby achieves better accuracy. However, then, we can also increase the expressive power of PINNs by changing architectures of PINNs. Thus, the authors must compare the proposed method, PINO, against various architectures of PINNs with various expressive powers. \n\nSimilarly, to verify the claim of the paper, the paper must include variants of PINNs that have solved many drawbacks of PINNs, such as extended PINNs. At this point, I don’t see any fundamental reason why PINO against variants of PINNs with various architecture choices. \n\nAlthough the authors mention the limitation of optimization of PINNs, there are various methods proposed to mitigate such limitations, including self-adaptive PINNs (Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism), adaptive activation functions (Adaptive Activation Functions Accelerate Convergence in Deep and Physics-informed Neural Networks), and sf-PINN (Learning in Sinusoidal Spaces with Physics-Informed Neural Networks). Considering the purpose of the proposed method, the authors must compare PINO against those methods in the paper.\n\nMoreover, for the pre-training, the papers should measure the time complexity of pre-training too to compare it fairly.  My worry is that under this problem setting, we can more efficiently solve each problem instance if we count the extra time of pre-training as well, if I use appropriate architectures and hyper-parameters. Thus, a comprehensive investigation on this is required before publication.\n\nAnother question is whether the model pre-trained with one PDE family can generalize well to other different PDE families. This is not conclusive from this paper as well. If we cannot generalize in this way, the applicability of PINO is crucially limited, as we can always solve each instance within a PDE family with variants of PINNs or other PDE solvers in the problem setting of this paper.  \n\n\n\n", "summary_of_the_paper": "This paper proposes Physics-Informed Neural Operator (PINO) by combining two previous methods, Fourier neural operators (FNO) and physics-informed neural networks (PINNs). Similar to one use case of FNO, PINO learns the solution operator over multiple instances of a parametric PDE family in the first phase. In the second phase, given an instance from the parametric PDE family, PINO fine-tunes the learned solution operator as a pre-trained model to work well for the particular instance. The proposed method is evaluated by using numerical experiments.   ", "summary_of_the_review": "This seems to be too incremental given the previous work and the experimental setups are not sufficient to support the claims of the paper. There is no theory provided.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "EnlcWJghjmV", "reviewer": "Reviewer_WVp6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and highlights the current challenges in PDE learning before introducing the approach. The following paragraphs detail the strengths and weaknesses of the paper.\n\nStrengths\n---\n- The paper is well-written and structured. The authors gives a good introduction of the field with its challenges and describe their contributions clearly. The next sections describe the PINN and FNO approaches before introducing the authors novelty method called PINO. The authors conclude by presenting a series of experiments.\n- The authors introduce a method that combines the strengths of two popular approaches in the field of physics-informed machine learning: a stable and fast method for learning solution operators based on a large number of input-output training pairs (FNO) and a deep learning technique for solving PDEs using PDE constraints. The main contribution consists of applying the physics (PDE) constraints onto a trained neural operator. At the heart of this technique lies an efficient procedure for computing derivatives of Fourier neural operator described in section 3.3 of the paper.\n- The authors show experiments on challenging (nonlinear, two-dimensional, chaotic, time-dependent) examples and achieve better results than standard deep learning methods on the three setups that they consider: learning solution operators, solving PDEs, solving inverse problems.\n\nWeaknesses\n---\n- My main concern about the method described is that it requires the knowledge of the PDE expression and therefore the title is a bit misleading since the PDE is already known and hard-coded as a constraint in the optimization. Therefore, it also combines the two weaknesses of FNO and PINNS: the need for training input-output data and the knowledge of the PDE, this should be explicitly written in the conclusions of the paper as a limitation.\n- I have some concerns about the choice of the training and testing data and optimization parameters. The training data used by the authors seem to be optimized for the problem considered, which questions the stability of the method. Is it able to tackle a new application without a lot of fine-tuning ? As an example, the Gaussian kernel used to sample the initial conditions vary across all the examples considered following a logic that I don't understand (see the experiments details described in the Appendix). This is a large limitation of the proposed method since it's heavily affects its flexibility for new applications. Similarly, the authors give little details about the training test used in the different applications. I am then wondering whether the good performances shown by the authors is due to the fact that the testing and training sets for the forcing terms (or initial conditions) are drawn from the same random distribution. In this case, the method could only be learning the solution operator on a specific vector space of functions and not in general (i.e. transfer learning would not work on a completely different distribution).\n- I don't think that the authors report an example with noisy data for learning and solving PDEs, which is essential to check the robustness of the method.\n- To my understanding, the Fourier neural operator method has severe limitations on the type of domains and boundary conditions, unlike PINNs (Raissi et al., Science, 2020), as it requires simple geometries and periodic boundary conditions to apply the Fourier transform layers. These limitations carries over in the present paper and the method can only be applied on simple domains with homogeneous/periodic boundary conditions. This questions the practical utility of the method for e.g. solving PDEs with respect to traditional numerical solvers such as FEM/spectral methods/finite differences. Can the authors comment on that, point out the limitations in the conclusions of the papers and discuss potential ways to overcome them ?\n- It would be great if the authors could add a paragraph to give a hint on potential practical concrete applications of the present methods, where one would have access and take advantage of both training data (sampled from a carefully selected Gaussian distribution) and the expression of the PDE. Right now, I can hardly see any concrete applications beyond fluid dynamics problems.\n\nMinor comments\n---\nCan the authors explain what they mean by \"additional PDE instances\" in Table 1 ? I couldn't find it in the Appendix and this should be described explicitly in the main text.\n\n### Post-rebuttal update\nI updated my score following the rebuttal.", "summary_of_the_paper": "The authors introduce physics-informed neural operator for learning partial differential equations (PDEs) from data with deep learning. This approach combines two recent and popular methods in the field of machine learning and PDEs, the physics-informed neural network (PINNs) for solving PDEs and Fourier neural operator (FNO) for learning solution operators. The paper takes advantages of both approaches to overcome convergence issues arising in PINNs by incorporating PDE constraints into FNO. Then, the authors apply their method to a number of challenging test cases and obtain better results than PINNs on PDE solving problems and FNO for learning solution operators.", "summary_of_the_review": "I find that the paper is well-written and introduces a method combining two known techniques for learning and solving PDEs from data. The authors present numerical results that outperform PDE benchmark used in the literature, taking advantage of the PDE constraint from the PINN approach or the neural operator learning from the FNO technique. Yet, there are some weaknesses in the study that I discuss in the main review. However, most weaknesses are inherent from the current challenges faced by the field of PDE discovery (such as choice of training data, practical applications, limitations on the domain geometry, boundary conditions).\n\nTherefore, I believe that the paper could motivate further explorations and research between deep learning PDE solvers and PDE learning methods and recommend acceptance of the paper to ICLR provided that the authors address the points raised in the review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes the physics-informed neural operator. It combines the operating-learning and function-optimization frameworks, which improves convergence rates and accuracy over traditional methods. While the paper was well written, several reviewers raised their concerns on the novelty of the paper, especially regarding the difference from PINN-DeepONet (Wang et. al.). Following this, there have been a long discussion between the authors and the reviewers, as well as among the reviewers. As a consequence, we think the authors somehow overclaimed their contributions on combining PINN and operator learning, and there are some important references missing and baselines not compared empirically. With this, the conclusion is that we cannot accept this paper in its current form, and we hope that authors can take all the review feedback into consideration and better position the novelty and impact of their work in the future submissions.
Decision: Reject