Semi-supervised learning of partial differential operators and dynamical flows
Authors: Michael Rotman
Amit Dekel
Ran Ilan Ber
Lior Wolf
Yaron Oz
Venue: ICLR 2022 Submitted (2022)
Keywords: Hypernetworks
Partial Differential Equations
Fluid Dynamics

Abstract: The evolution of dynamical systems is generically governed by nonlinear partial differential equations (PDEs), whose solution, in a simulation framework, requires vast amounts of computational resources. For a growing number of specific cases, neural network-based solvers have been shown to provide comparable results to other numerical methods while utilizing fewer resources.
In this work, we present a novel method that combines a hyper-network solver with a Fourier Neural Operator architecture. Our method treats time and space separately. As a result, it successfully propagates initial conditions in discrete time steps by employing the general composition properties of the partial differential operators. Following previous work, supervision is provided at a specific time point. We test our method on various time evolution PDEs, including nonlinear fluid flows in one, two, and three spatial dimensions. The results show that the new method improves the learning accuracy at the time point of supervision point, and is also able to interpolate and extrapolate the solutions to arbitrary times.

Under review as a conference paper at ICLR 2022
SEMI-SUPERVISER LEARNING OF PARTIAL DIFFEREN- TIAL OPERATORS AND DYNAMICAL FLOWS
Anonymous authors Paper under double-blind review
ABSTRACT
The evolution of dynamical systems is generically governed by nonlinear partial differential equations (PDEs), whose solution, in a simulation framework, requires vast amounts of computational resources. For growing number of specific cases, neural network-based solvers have been shown to provide comparable results to other numerical methods while utilizing fewer resources. In this work, we present a novel method that combines hyper-network solver with a Fourier Neural Op- erator architecture. Our method treats time and space separately. As a result, it successfully propagates initial conditions in discrete time steps by employing the general composition properties of the partial differential operators. Following pre- vious work, supervision is provided at specific time point. We test method on various time evolution PDEs, including nonlinear fluid flows in one, two, and three spatial dimensions. The results show that the method improves the learning accuracy at the time point of supervision point, and is also able to interpolate and extrapolate the solutions arbitrary times.
INTRODUCTION
The evolution of classical and quantum physical dynamical systems in space and time is generically modeled by non-linear partial differential equations. Such are, for instance, Einstein equations of General Relativity, Maxwell equations of Electromagnetism, Schrödinger equation of Quantum Me- chanics and Navier-Stokes (NS) equations of fluid flows. These equations, together with appropriate initial and boundary conditions, provide complete quantitative description of the physical world within their regime of validity. Since these dynamic evolution settings are governed by partial dif- ferential operators that are often highly non-linear, it rare to have analytical solutions for dynamic systems. This is especially true when the system contains a large number of interacting degrees of freedom in the non-linear regime.
Consider,
example, the NS equations, which describe the motion of viscous fluids. In the regime of high Reynolds numbers of the order of one thousand, one observes turbulences, in which all symmetries broken and all analytical techniques fail. The solution these (deterministic) equations seems almost random and is very sensitive to the initial conditions. Many numerical tech- niques have been developed for constructing and analysing the solutions fluid dynamics systems. However, the complexity of these solvers grows quickly as the spacing in the grid that is used for approximating the solution is reduced and the degrees of freedom of the interacting fluid increases.
Given the theoretical and practical importance of constructing solutions to these equations, it is nat- ural ask whether neural networks can learn such evolution equations and construct new solutions. The two fundamental questions are: (i) The ability to generalize to initial conditions that are dif- ferent from those presented in the training set, and (ii) The ability to generalize to new time points beyond the fixed grid points provided during training. The reason to hope that such tasks can be performed by machine learning is that despite the seemingly random behaviour of, e.g. fluid flows in the turbulent regime, there is an underlying low-entropy structure that be learnt. Indeed, in diverse cases, neural network-based solvers have been shown to provide comparable results to other numerical methods, while utilizing fewer resources.
Our Contributions Operator architecture.
We present a hyper-network based solver combined with a Fourier Neural

---

Under review as a conference paper at ICLR 2022
Our hyper-network architecture treats time and space separately. Utilizing data set of initial conditions and the corresponding solutions at labeled fixed time, the network learns large class of time evolution PDEs.
Our network successfully propagates initial conditions in discrete time steps by implement- ing the general composition properties of the partial differential operators.
Our solutions improve the learning accuracy at the supervision time-points.
Our solutions are able interpolate and extrapolate to arbitrary (unlabelled) times.
We thoroughly test our method on various time evolution PDEs, including nonlinear fluid flows in one, two and three spatial dimensions.
2 RELATED WORK
Hypernetworks. While conventional networks employ fixed set of pre-determined parameters, which is independent of the input, the hypernetwork scheme, invented multiple times, and coined by Ha et al. (2116), allows the parameters of neural network to explicitly rely the input by combining two neural networks. The first neural network, called the hypernetwork, processes the input or part of it and outputs the weights of second neural network. The second network, called the primary network, has a fixed architecture and weights that vary based on the input. It returns, given its input, the final output.
This framework was used successfully in variety of tasks, ranging from computer vision (Littwin & Wolf 2219)), continual learning (von Oswald et al. 22019), and language modeling (Suarez 2017). While it natural to learn functions with hypernetworks, since the primary network can be seen as dynamic, input-dependent function, we are not aware of any previous work that applies this scheme for recovering physical operators.
Neural network-based PDE solvers. Due to the well-known limitations of traditional PDE solvers on one hand, and in light of new advances made in the field of neural networks on the other, lately we have witnessed very significant progress in the field of neural network-based PDE solvers (Karniadakis et al. (021).
Neural network-based PDE solvers can be roughly divided into two groups according to the resource they utilize for learning: data-driven and mosel- Data-driven solvers use a large dataset con- taining initial conditions and final states, all at the same timepoint T train (Zhu & Zabaras 2018). These solvers predict the solution for new initial conditions at the same T. but cannot be interpolated extrapolated to times that not increments of T. However, the PDE is not required for such solvers, they can be used in cases where the underlying PDE is unknown.
Data-driven approaches often reduce the problem setting of PDE solvers to the well-known problem setting of image-to-image mapping in computer vision. Following this approach, it is customary to use networks such U-Net (Ronneberger al. 2015).
Model-based solvers, known as Physics Informed Neural Networks (PINNs) Raissi et al.) 2019), harness the differential operator itself for supervision. This done by defining a loss, the residual of the PDE. These solvers do not require training dataset and can provide solutions for arbitrary times.
Both approaches learn solutions on specific discretized grid, which poses a limitation for any practical applications. Recently, a
Bata-driven direction has been proposed (Lu al.)
2019 BBhatacharya et al.) 2020 Nelsen & Stuart, 2021 Li et al.) 2020b) Patel et al., (021). The mesh invariance is obtained by learning operators rather than mappings between initial and final
states.
Li et al. (020cc have advanced the mesh-invariant line of work by introducing Fourier Neural Op- erators (FNO). FNOs utilize both a convolution layer in real space and Fourier integral layer in the Fourier domain. It has been shown that the FNO solver outperforms previous solvers in a num- ber of important PDEs. FNO has major limitation: the method (in the framework of learning from pairs of initial states and final solutions) cannot used to provide solutions for interpolation and arbitrary extrapolations. Knowing the solutions along the complete time evolution trajectory is
2

---

Under review as a conference paper at ICLR 2022
highly valuable for theoretical and practical reasons and provides means for learning dynamical principles.
PARTIAL DIFFERENTIAL EQUATIONS
Consider d-dimensional vector field (ax, 1) T X - R2, where a: = (#11 Fa) are periodic spatial coordinates 2; 2, + 2nT the d-dimensional torus, T and t is the time coordinate. The
vector field evolves dynamically from the initial condition v(a, = 0) T + Rd according to non-linear PDE of the form,
0, t) = Lv(x, t),
(1)
where a differential operator that does not depend explicitly on time, and has an expansion in and its spatial derivatives. We assume that given regular bounded initial vector field there is unique regular solution to equation]] [
Since L does not depend explicitly on time, we can formally
write the solution of such equations
v(x, = vlx, 0) (, 0)
Because of dissipation terms, such as the viscosity term the NS equations, solutions to equation generically break time reversal invariance (1 - -1, v - -v). Furthermore, the total energy of the system is non- con-increasing as a function of time since we are not injecting energy at * 0,
0, 1'= v 00. 2
(3)
Equation serves as consistency on the network solutions.
In this work, we consider the following non-linear PDEs,
(i) Burgers equation describing one-dimensional compressible fluid flows with scalar velocity field v(z,t). t).
0e + =
(4)
where is the kinematic viscosity, V the gradient and A the Laplacian. iii Generalized onc-dimensional Burgers equation with parameter q 2, 3, 4,
0,0 + Vv = vAv.
(5)
(iii) One-dimensional Chafee- equation with a constant À parameter that models reaction- diffusion dynamics,
0₂u X(x2 v) v) Av.
(6)
(iv) Two-dimensional Burgers equation for wwodiimensional vector field, v(x,t) = (v', v²)), that describes compressible fluid flows in two space dimensions:
0, (v V)u = vAv.
(v) Two and three-dimensional incompressible NS equations.
0,u (v V)r Vp vāe, V-v=0, where p is the fluid's pressure, v(x,t) = (v' ...., vd) and v is the kinematic viscosity.
(8)
4 METHOD
Typically, data-driven PDE solver, such a neural network, evolves unseen velocity fields, v(x, 0) fixed time T.
(9) by learning from a set of i = 1... N initial conditions sampled at t = 0, v; (xx t = 0), and their corresponding time-evolved solutions of (x, = T) of equation
Dru (z, t = 0) = v (x,T),
We generalize 4r, to propagate solutions at intermediate times, 0 < t T, by elevating a standard
neural architecture to hyper-network one. A hyper-network architecture is composition of two
3

---

Under review as a conference paper at ICLR 2022
neural networks, primary network fw (2) and hypernetwork go(t) with learned parameters 8, such that the parameters w of f are given as the output of g. Unlike common neural architectures, where single imput is mapped to single output, in this construction, the input is mapped to function, (go(t), which maps x to its output. Applying this to our case we have,
2,0(2,0) 0) = (t) (v(xz 0)) = v(x,t).
(10)
This architecture may be used learn not only the time-evolved solutions at = T, but also inter- mediate solutions, at 0 < T, without explicitly providing the network with any intermediate time solution.
This task may be accomplished by utilizing general consistency conditions, which apply to any equation of the form equation
112
11 t1 = +1
(11)
Notice that &o- lla follows from equation 11
4.1 LOSS FUNCTIONS
In this section we introduce the loss functions used to train our models. In order to simplify our notation, we will suppress the explicit space dependence in our equations, so v( f 0) will be denoted by v0). We denote by N the size of the training set of solutions. Our loss function consists of several terms, supervised and unsupervised.
The information of the specific PDE is given to our model via supervised term,
N
Cinal
Σ
Err (ir (0), vz (T))
(12)
which is responsible for propagating the initial conditions to their final state T. The recon- struction error function Err is defined as,
>n (an, )
Err (a, b)
(13)
Σ b
where the 9 indices run over the grid coordinates.
To impose the consistency condition t = Id. we use the loss function term,
Cinitial
Σ Err (Roz; (0), v (0))
(14)
The composition law in equation II is implemented by the loss function term,
Linter
Σ Err (014 (0), 01 0 (0))
(15)
1=1
where ī t) + th is drawn uniformly from U|0, 7 and t is drawn uniformly from U(o, TJ. the sum of 4 + 15 does not exceed T.
The last term concerns the composition of tb maps to generate oy by p-sub-intervals of the interval [o, T]
p
p
II
21
Et t --=T-
400,
(16)
and it reads,
Comp
ΣΣ Em
I j=1
(0) 8. (
(17)
where P is the maximal number of intervals used. The time intervals are sampled using ty U [ 1T and the last interval is dictated by the constraint in equation 16 The P term is
omitted, since it corresponds to Linat- The total loss function reads,
c(P) Linal Linitial + Linter f.
(18)
4

---

Under review as a conference paper at ICLR 2022
5 THEORETICAL CLAIM
Consider the differential equation 1 Suppose we construct map Φ such that Tr (æ, t = 0) = by (a; t = T", where V; (ax f = 0) are all initial conditions (or form an appropriate complete set that spans all possible initial conditions) for equation
vs (ær, = T)) are the corresponding solutions of the equation at time T. We would like to know whether P, (æ, 0) = v, (x, t) for all times t 2 O, Le. that the map 4, evolves all the initial condition correctly the solutions of equation for any time.
Lemma 1. A map 4 that propagates a complete set of initial conditions at 1 = to the correspond- ing solutions of equation 7 fixed time TT and satisfies the composition law in equation 77 propagates the initial conditions to the corresponding solutions for any time t 2 0.
Proof. Denote by yt the map that propagates correctly at any time the initial conditions to solutions of equation[] The space of initial conditions v v(z, 0) T + R is complete basis of functions and the same holds for v(æ, 7) by the uniqueness of the solutions to equation [ Since yr agrees with T on this set of initial condition it follows that 4T ΦT. Consider next partition of the
interval [0, T to p identical parts, [0, F), 17. 2I I.... (p-1)T TJ. We have the composition of maps:
& $
r ΦT -T= PpPE
PT
(19)
from which we conclude that ør Φτ (there could have been an overall phase in the relation between them, but since our PDEs and maps are real we can neglect it). We can perform this division for any p and conclude using the composition rule in equation II that D, sp for any
EXPERIMENTS
We present battery of experiments in 1D, 2D and 3D. The main baseline we use for comparison is FNO (Li et al. (2020c), which is the current state of the art and shares the same architecture as our primary network f. For the ID Burgers equations, we also compare with the results of Multipole Graph Neural Operator (MGNO) (Li et al. 2020a).. We do not have the results of this method for other datasets, due to its high runtime and memory complexity.
6.1 EXPERIMENTAL SETUP
Our neural operator is represented by a hyper-network composition of two networks. For the primary network f. we use the recent FNO architecture. This architecture consists of four Fourier integral operators followed by a ReLU activation function. The width of the Fourier integral operators is 64, 32 and 20, and the operators integrate up to cutoff of 16, 12, modes Fourier space, for one, two, and three dimensions, respectively.
For the hypernetwork y we use fully-connected network with three layers, each with a width of 32 units, and ReLU activation function. y generates the weights of network for given time point t, thus mapping the scalar t to the dimensionality of the parameter space of f. A tanh activation function is applied to the time dimension, t, before it is fed g. Together, f and g constitute the operator 4 as appears in equation [0
with with learning rate of 0.001, a weight decay of 0.0001, and a batch size of 20. For MGNO, we used learning rate of 0.00001, weight decay of 0.0005 and batch size of 1, as in its official github repository. For our method, at each iteration and for each sample, we sample different time points for obtaining the time intervals of ClPPp and Limer detailed in Sec. 4.1
All experiments trained for 500 epochs,
ADAM Kingma & Ba (2014) optimizer and
For the one-dimensional PDEs, we randomly shift both the input and the output to leverage their periodic structure. For one- and two-dimensional problems, the four loss terms employed un- weighted. For the dimenssonl NS equation, the Cinter term was multiplied by a factor of 0.1, the reason being that increase the number of space dimensions encounter complexity
5

---

Under review as a conference paper at ICLR 2022
due to the growing number of degrees of freedom (i.e. the minimum number of grid points per inte- gral scale) required to accurately describe a fluid flow. Standard Kolmogorov's 's type scaling (Frisch) 1995) implies that the number of degrees of freedom needed scales as Na ~ R S where R is the Reynolds number. This complexity is seen in numerical simulations and we observe it in our framework of learning, as well.
6.2 DATA PREPARATION
Our experiments require a set of initial conditions at = 0 and time-evolved solutions at time t = T for our PDEs. For the simplicity of notations we will set T 1 in the following description. In all cases, the boundary conditions as well as the time-evolved solutions are periodic in all spatial dimensions. The creation of these datasets consists of two steps. The first step is to sample the initial conditions from some distribution. For all of our one-dimensional experiments we use the same initial conditions from the dataset of Li et al. (2020c), which was originally used for Burgers equations and sampled using Gaussian random fields. For Burgers equation, we used solutions sampled at 512, 1024, 2048, 4096, 8192, while for the other PDEs we used grid size of 1024. All one-dimensional PDEs assume a viscosity, V = 0.1.
For the two-dimensional Burgers equation our data is sampled using a Gaussian process with a periodic kernel,
k(2, g. 3y) = 02e (1=xr1
2xin ilg-v 12)
(20)
with 1 and / 0.6, and with grid size 64 64 and viscosity of 0.001. For the two and three-dimensional NS equation we use the ΦFlww package (Holl et al. 2020 to generate samples on 128 x 128 and 128 x 128 128 grids, later resampled to 64 x 64 and 64 64 x 64, respectively. For the two-dimensional NS, we use a viscosity of v = 0.01, and for three-dimensional NS, we use viscosity of v 0.001. In all cases the training and test sets consist of 1000 and 100 samples, respectively.
The second step is to compute the final solution at For the equations in one dimension and the
two-dimensional Burgers equations we use the py-pde python package (Zwicker (020) to evolve the initial state for the final solution. To evaluate the interpolation and extrapolation capabilities of our method, we also compute all intermediate solutions from / = 0 to = 3 with A = 0.01 time- step increments. For the wwo-dimensional Burgers equations we do this up to t = 1.15, since at this regime the solutions develop shock waves due to low viscosity and we learn regular solutions. For the two-dimensional and dimee-iinnsional NS equations we use @How package (Holl et al. 2020) to increment the solutions with At = 0.1 increments.
6.3 RESULTS
We compare the performance of our method on the one dimensional PDEs with FNO (Li et al. 2020ch and MGNO (Li et al. 2020a). In this scenario, we use loss term with two intervals, P 2. Since MGNO may only use a batch size of 1 and requires a long time to execute, we present it only for the one-dimensional Burgers equation. As seen in Table in all PDEs, our method outperforms FNO by at least a factor of 3. We further compare the one- -ieeennniional Burgers equation for various grid resolutions at = 1 in Figure D As previously noted by Li et al. (2020c), the grid resolution does not influence the reconstruction error for any of the methods.
To assess the contribution of the Limer term and to account for the influence of the number of intervals induced by Clopp. we evaluate the reconstruction error for the task of interpolating in / E [0, 1] in comparison the ground truth solutions in Figure 2 As baseline for the interpolation capabilities of our network, we further apply a linear interpolation between the initial and final solution, utt) = (1-t)e(0) + (1) (even though one uses the unfair advantage at f = 1). The horizontal dashed cyan line marks the reconstruction error of the the FNO method at 1, thus the interpolation our approach provides exceeds the prediction ability of FNO at discrete time steps. For extrapolation, > 1, we divide the time into unit intervals plus residual interval (e.g. for = 2.3 we have the three intervals 1, 1, 0.3). and we let the network evolve the initial condition to the solution at t = 1, and then repeatedly use the solution initial condition to estimate the solution at later times. As can be seen, the addition of the Limter term dramatically improves the reconstruction error. Furthermore, increasing the number of intervals P induced by
solution explicitly for that which gives clear and

---

Under review as a conference paper at ICLR 2022
Reconstruction Error
PNO
MGNO
Ours
10-2
10
512
1,024
2,048
4,096
8, 8,192
Grid Size
Figure 1: The reconstruction on the one-dimensional Burgers equation for different grid reso- lutions.
Table 1:
The reconstruction eπror on the one-dimensional PDEs at t = 1, grid size = 1024. B refers Burgers equations, GB to generalized Burgers equations, and CI Chafee-Infante equation.
Method B
CI
GB q=2
GB q=3
GB q=4
MGNO
0,0552
FNO
0.0096
0.0028
0.0090
0.0095
0.0098 0.0030
Ours
0.0022
0.0008
0.0031
0.0030
(P)
Coomp reduces the reconstruction error, but with smaller effect. The reconstruction error in the extrapolation region is low mainly because of the high energy dissipation. Note that for all of the presented variations, the reconstruction error of our method at is lower than FNO.
In Table 2 we compare the performance of our methods to FNO on the two dimensional PDEs, Burgers and NS equations. In 2D the advantage of our method over the baseline at 1 relatively modest. However, a much larger advantage is revealed on intermediate time, can be seen in FaguresBaals and 3bb] In both PDEs is apparent that Linter an essential ingredient for providing satisfactory interpolation, and that the number of intervals, P, is not crucial in higher dimensions.
Table Soole3summe summarizes the reconstruction error on the three-dimensional NS equation at 1. this case, we applied our method with P = 1 for computational reasons, larger number of intervals required significantly more memory (during training only; during inference requires a memory size similar that of FNO). As can be seen, method outperforms FNO and performs the best with the Lintrr term in this benchmark as well.
DISCUSSION AND OPEN QUESTIONS
Lemma 1 points to practical sources of possible errors in learning the map Φ.. The first source of error concerned with the training data set. Quantifying the set of initial conditions and solutions at time T, from which we can learn the map Φ with required accuracy, depends the complexity of the PDE. Burgers equation is integrable (for regular solutions) and can analytically solved. In contrast, NS equations Thus, for a given learning accuracy, we expect the data set needed for Burgers equation to be much smaller than the needed for NS equations. Indeed,
not integrable dynamical flows and cannot be solved analytically.
numerically the difference in the

---

Under review as a conference paper at ICLR 2022
0.12
Baseline
P
0.1
Reconstruction Error
P=2
P-3
8-
2 w/o Linter
Interpolation
Bitrrapolation
6-10-2
Heggion
Etagton
4-10--
2-10
0.5
1.5
2
2.5
3
the one-dimensional Burgers equation for intermediate time predictions and the py-pde solver solutions for different numbers of intervals P in the composition (P) loss term, Comp. We show the median well the 10th, 25th, 75th and 90th percentiles. The statistics is based on 100 samples from the test set. The dashed horizontal cyan line marks the reconstruction error of FNO 1 for comparison.
Figure 2: The reconstruction
0.25
P= w/o Cinter Center
1 w/o Cinter w/o Cinter
Iaserslaaton
Rocnssrrelan Error
Begico
P
Reconstnetion Ereor
0.2
Begiss
=2
P 2
0.15
0.15
0.1
0.1
5.
5-10-²
0.2
0..4
0.6
00.8
0.5
1.5
(b)
Figure 3: The reconstruction error for intermediate time predictions and ground truth solver solu- ((P)
tions for different number of intervals P in the composition loss term, Ccomp- We show the median as well as the 10th, 25th, 75th and 90th percentiles. The statistics is based on 100 samples on the test set. The dashed horizontal cyan line marks the reconstruction error of FNO f for comparison. (a) two-dimensional Burgers equation. (b) two-dimensional NS equation.
learning accuracy of these PDEs. Second, there is error in the learning of the consistency conditions in equation II] Third, we impose only limited set of partitions equation [6] which leads to an error that scales as J. where p the partition number of the time interval. We decreased this error by applying various such partitions. Note, that the non-convexity of the loss minimization problem implies, as we observe in the network performance, that the second source of error tends to increase when decreasing the third one and vice versa.
8

---

Under review as a conference paper at ICLR 2022
Table The reconstruction error on the two-dimensional PDEs 1, Burgers and NS equations.
Method 2D Burgers
2D NS
FNO
0.0336
0.0467
Ours
0.0335
0.0442
Table 3: The reconstruction error the three-dimensional NS equation 1.
Method
3D NS
FNO
0.2778
Ours
0.2504
Ours w/o Linter
0.2675
Our work opens up several directions for further interesting studies. First, it is straightforward to generalize our framework to other PDEs, such as having higher-order time derivatives, as in relativistic systems where the time derivative is of order two.
Second, we only considered regular solutions our work. is known that singular solutions such shock waves in compressible fluids play important role in the system's dynamics. The con- tinuation of the solutions passed the singularity is known to be unique in one-dimensional Burgers system. This is not the in general as, for instance, for inviseid fluid flows modelled by the incompressible Euler equations. would be interest to study how well our network learns such solutions and extends them beyond the singularity.
Third, we learnt the dynamical equations at a fixed grid size the spatial coordinates. It would be highly valuable if the network could learn interpolate smaller grid sizes. FNO allows super- resolution and hence an interpolation to smaller grid size. It does so by employing the same physics of the larger scale. Learning the renormalization group itself can open up a window learning new physics that appears at different scales.
Fourth, studying the space of solutions and its statistics (in contrast individual solutions) for PDEs such as NS equations is expected to reveal insights about the universal structure of the physical system. For example, it would be valuable to automatically gain insights into the major unsolved problem of anomalous scaling of turbulence.
In our work we considered a decaying turbulence that follows equation 3] In order to reach steady- state turbulence and study the statistics of the fluid velocity distribution one needs to introduce in the network a random force pumping energy to the system same rate as that of dissipation.
8 CONCLUSIONS
Learning the dynamical evolution of non-linear physical systems governed by PDEs is of much importance, both theoretically and practically. In this work we presented network scheme that learns the map from initial conditions to solutions of PDEs. The scheme bridges an important gap between data- and model-driven approaches, by allowing data-driven models provide results for arbitrary times.
Our hyper-network based solver, combined with a Fourier Neural Operator architecture, propagates the initial conditions, while ensuring the general composition properties the partial differential operators. It improves the learning accuracy the supervision time-points and interpolates and extrapolates the solutions to arbitrary (unlabelled) times. We tested our scheme successfully on non-linear PDEs in one, and three dimensions.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
All the code for reproducing all experiments presented in this work available in the supplementary material.
REFERENCES
Kaushik Bhattacharya, Bamdad Hosseini, Nikola Kovachki, and Andrew M Stuart. Model reduc- tion and neural networks for parametric pdes. arXiv preprint arXiv:2005.03//00 2020.
Uriel Frisch. Turbulence: the legacy of AN Kolmogorov. Cambridge university press, 1995.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint 1rXi: 2016.
Philipp Holl, Vladlen Koltun, Kiwon Um, Telecom Paris LTCI, IP Paris, and Nils Thuerey. phiflow: A differentiable pde solving framework for deep learning via physical simulations. In NeurlPS Workshop, 2020.
George Em Karniadakis, Ioannis Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physies-informed machine learning. Nature Reviews Physics, 3(6):422 440, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint orXiv:1412.6980 2014.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. arXiv preprint wrxiv:2000.09535. 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differ- ential equations. arXiv preprint arXiv:2003. 00:.:000000085,,.. 2020b.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar, et al. Fourier neural operator for parametric partial differential equa- tions. In International Conference on Learning Representations, 2020c.
Gidi Littwin and Lior Wolf. Deep meta functionals for shape representation. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 1824-1833, 2019.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for iden- tifying differential equations based on the universal approximation theorem of operators. arXiv preprint 19X0:19900.3: 2019.
Nicholas H Nelsen and Andrew M Stuart. The random feature model for input-output maps between banach spaces. SIAM Journal Scientific Computing, (000))3222-334444 2021.
Ravi G Patel. Nathaniel Trask, Mitchell Wood, and Eric C Cyr. physics-informed operator re- gression framework for extracting data-driven continuum models. Computer Methods in Applied Mechanics and Engineering, 373:1 13500, 2021.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informi neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Joseph Suarez. Language modeling with recurrent highway hypernetworks. In Advances in neural information processing systems, pp. 3267-3276, 2017.
Johannes von Oswald, Christian Henning. João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. arXiv preprint arXiv: (eee::90000999 2019.
10

---

Under review as a conference paper at ICLR 2022
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional Secoder-decoder networks for surrgaate modeling and uncertainty quantification. Journal of Computationnll Physics, 366:415- 447, 2018.
David Zwicker. py-pde: python package for solving partial differential equations. Journal of Open Source Software, 5(48):2158, 2020. doi: 10.21 105/1000/ssss11.. 02158. URL https://doi.
org/10. 2I1/10.21105joss.00200
A ONE-SHOT EXTRAPOLATION
In order assess the ability the hyper-netword scheme extrapolate solutions beyond simulated an additional 20 solutions ranging from f = to f = 20, in increments of = 0.01 for the 2D Navier-Stoke equation. Fig. shows that our method is able to extrapolate solutions in one-shot up to ~ 2 while doubling the error. This linear structure of error accumulation is unlikely to hold at long times due to the general chaotic structure of the solution. On general ground, one expects that for chaotic systems nearby trajectories would diverge exponentially:
we
(e Au(0)e
(21)
When At << we have Av(()-Ar()) At and we can estimate from Fig. b)) in the time window Ar(O) [0, 1] that x 10-2, With such a value we can estimate,
1 Av(t) - An(0)
~~ 20
(22)
Ar(O)
Thus, 10 we expect error about 65 percent. The flattening of the error curve Fig can be traced to the effect of the viscosity in the system.
0.8
Reconstruction Error
0.6
0.4
0.2
0.1
0
10
15
20
Figure 4: Reconstruction error for the 2D Navier-Stokes equations for one-shot time extrapolation.
B VISUALIZATIONS
Fig. shows the time evolution of the states of 2D Burgers equations. As can be seen, shocks are being developed the vicinity of 1.15.
11

---

Under review as a conference paper at ICLR 2022
Ground
Prodlctioo
Groumd
Vortinity
0.4
Figure 5: The velocity vector fields and the corresponding vorticity of the 2D Burgers equation. Shocks are being formed the final state.
12


Reviews:
[{"review_id": "iG7K6Rs4E0Y", "reviewer": "Reviewer_3Uae", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is in a popular direction, and has all the pros and cons of such kind of papers. The idea of using hypernetworks to model the transition operator is reasonable and of practical interest.  The drawback is that the computed solutions are approximate and do not show (at least, this not considered) systematic convergence to the true solution of the PDEs. Let me explain in more details. A classical FEM/FDM solver typically converges as the number of parameters (mesh points) increase. For neural network based method, a similar phenomenon should be observed when the number of parameters in a neural network representation increases. There is a bunch of work (C. Schwab, G. Kutyonok, D. Yarotsky) that show theoretically for some classes of problems that it is true. But in practice, such results are never reported. Moreover, this paper uses only one fixed architecture and tests it for different grid sizes. So, what is the actual error of the computed solution?\n\nA second question is the methodology of learning the map. As initial condition, a sequence of Gaussian function is used. However, difficult examples for the Burgers equation are \"kinks\", and there is rich literature on constructing efficient numerical schemes for such kind of tasks. Can you provide any guarantee that the model learned for a small set of function will be able to generalize to other functions? \n\nComments&Questions:\n\n- The Theorem one seems more like a Lemma or Statement, since it is rather straightforward.\n- What norms is used to measure the reconstruction error? In PDEs, the errors are measure in the relative norms, which are suitable for the corresponding PDE. \n- Computational speed vs accuracy: the error does not depend on the mesh size (Figure 1) . For finite-element, finite volume method there is a systematic convergence of the error with respect to the number of parameters. What can be said about the convergence of the solution with increasing number of parameters? What is the complexity of solving a PDE compared to the classical solver of the same accuracy? \n- Probably, the most interesting part of the model is the potential ability to predict the solution for long time in \"one shot\".  Can you clarify, who long you can do it? What is the largest time the model can predict?  If smaller steps are needed, then it is more close to the classical time-stepping (also interesting, but not so, since again, a classical solver, maybe with higher-order elements, will outperform the Neural Solver quite significantly).\n\n", "summary_of_the_paper": "The paper presents a model and a loss function for approximation of the solution map of certain partial differential equations. \nThe idea is to use HyperNetworks: the mapping of the initial condition to time t done by model f($\\theta$, x), and the parameters are predicted by another neural network. The models utilize the recently proposed Fourier Neural Operator (FNO) for the mapping model, and a fully-connected network for the hypernetwork. Several losses are proposed to ensure consistency of the proposed models, including the composition and reconstruction. The tests are done on a battery of examples, and compared to other neural network baselines the error is smaller. A theorem is proved, but is rather straightforward.", "summary_of_the_review": "The paper is in a popular direction, proposes a reasonable improvement, but the presentation and methodology of the study could be significantly improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WwuwEHfbUuR", "reviewer": "Reviewer_j3Ma", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well-written and easy to follow. I quite like the idea of using a hyper-network\nto make the time domain continuous instead of having to add a new dimension to the FNO (the paper does lack\na short review of the FNO which I think is needed as it is not a standard architecture quite yet). The \nidea of learning Markovian systems by exploiting the compositional structure of the flow map\n(again using FNO) was earlier proposed in (https://arxiv.org/abs/2106.06898) where the approach is \nto learn a FNO which emulates \\Phi_{h} for some fixed time step h > 0 and predict forward by simply\nhaving the FNO act on its own output repeatedly. While this paper considers ergodic systems and tries\nto capture the statistics of the invariant measure by emulating very long-time dynamics, I suspect a \nsimilar approach can be used in the present setting by using a small time-step h along with the loss\nfunction proposed by the authors. A comparison of the two methods would be quite helpful. \n\nI am slightly worried about the quoted results for the 1-d Burgers' equation where the authors' method\noutperforms the standard FNO by almost an order of magnitude. The paper (https://arxiv.org/abs/2108.08481 - Table 2)\nshows standard FNO results on 1-d Burgers' that are better than those of the presented method and can be \nimproved even more by using a GeLU activation. This casts doubt as to whether the author's hypernetwork\napproach is giving the improved results or simply not tuning the FNO. The numerical section is, in my\nopinion, the weakest part of the paper as it is not convincing that this method of training which is certainly\nmore computationally expensive offers a large advantage over standard training. A clear advantage is the \nability to extrapolate the solution outside the training time window and the authors should make this more \nof a focus, perhaps on problems where shocks don't develop, and show for how many Lyapunov times the \nextrapolated solution can keep up with the true trajectory. Furthermore considering much longer time domains \nwould be of great interest as simply taking the solution to t=1 is not enough. Even the original FNO\npaper considers NS up time time 50.  Performing such experiments is crucial if the work is to be published.", "summary_of_the_paper": "The paper combines the FNO architecture with hypernetworks to learn the solution operator (flow map)\nof Markovian PDE systems. A hypernetwork whose input is the time domain (\\R_+) is trained to output \nthe weights of a FNO which then acts on an initial condition to produce the PDE solution at the time \ninput to the hypernetwork. Training is done by uniformly sampling random times on a bounded input domain\nand exploiting the Markov property of the flow map, allowing to use as data only the initial condition \nand the solution at a final time without a need for the entire time series. Numerical experiments showing \na modest improvement over the standard FNO are performed on the 1-d Burgers' equation, generalized Burgers'\nequation, and the Chafee–Infante equation as well as on the 2-d Burgers' equation, and the Navier-Stokes\nequation.", "summary_of_the_review": "Nice way of dealing with the time domain by using a hypernetwork, but experiments are not so convincing that\nthe approach is better than the standard.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-3vr_u_dnkO", "reviewer": "Reviewer_5dc6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is quite interesting to me. It provides a continuous time representation of FNO using hyper-networks.\n\nComments:\n1. In the original FNO paper, there are two formulations of time: end-to-end spatial mapping G:v(0) -> v(t) and space-time mapping G: v(0) -> (v(0), v(t1), v(t2), ...., v(t)). It seems this paper mainly addresses the end-to-end mapping, but it's indeed more similar to the space-time FNO.\n\n2. The model learns the map G:(v(0), t) -> v(t). In some sense, if once input a full time squeeze (v(0), [0,t1,t2,t3,...,1]), the model will output the full time trajectory (v(0), v(t1), ...., v(1)). In this sense, both this hyper-network model and the original space-time FNO learns one set of parameters for the full trajectory. The major contribution of this work, I think, is to continuously evaluate at any time t, which is quite significant.\n\n3. Same as the space-time FNO, when working on a chaotic system such as NS, the problem will be very challenging if the time interval t is too large. It will be interesting to study the limit of t for the hyper-networks.\n\n4. If remove the batchnorm in the original FNO, its error on 1d Burgers will drop from ~0.01 to ~0.0025, which matches the performance of this hyper-networks model, and then the performance becomes consistent among all other 2d 3d PDEs in the paper (check https://github.com/zongyi-li/fourier_neural_operator/issues/30). In my opinion, it is totally expected that the proposed FNO model does NOT outperform the original FNO at discrete time steps. In general, discrete models get better errors at discrete time steps compared to continuous models (well, in this case, the continuous model is trained using extra information between steps, so it may not be fair). The main advantage of a continuous model is to do continuous evaluations. For a fair and consistent comparison, I suggest the author use the updated FNO code on 1d Burgers equation.\n\n5. the model enforces the composition law (Equation 11) by adding a loss, which seems sub-optimal to me. Ideally one should develop a model that satisfies the composition law intrinsically (like in https://arxiv.org/abs/2106.06898), which is of course a non-trivial problem.\n\n6. some personal tastes:\n(a) why call it \"semi-supervised\"?\n(b) It will be great to include some virtualizations of the 2d burgers and 2d/3d NS equation, at least in the appendix.\n(c) It is better to call this theorem a remark.", "summary_of_the_paper": "In this paper, the authors propose to use hyper-network to learn a continuous time-dependent evolution operator based on Fourier neural operator.", "summary_of_the_review": "I think the problem is significant and the model is novel. While the paper needs some revision and polish, I would recommend borderline acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach to learn the solution operator of Markovian partial differential equations (PDEs) by combining the Fourier Neural Operator (FNO) with a hyper-network.  In short, the hyper-network g_\theta(t) is trained to output the weights of a FNO f_w(x), which (given an initial condition) outputs the PDE solution at the time given to the hyper-network. The main claimed contributions of the proposed approach (as compared to, e.g., the original FNO architecture) are that the obtained solutions improve the learning accuracy at the supervision time points and that the solutions are able to interpolate and extrapolate to arbitrary times.  

The reviewers seemed to like the idea of using hyper-networks for modelling continuous-time FNOs. Several issues were raised by the reviewers, e.g., with respect to related work by Li et al. (2021, https://arxiv.org/abs/2106.06898), which I believe were addressed by the authors satisfactorily. However, it is still concerning that the reported performance for FNO in, e.g., 1d-Burgers does not quite match those recently published  (Kovachki et al, 2021, https://arxiv.org/abs/2108.08481, Table 2).  Although the authors did report additional results using GeLU, these results are still very different to those in  Kovachki et al ( 2021) and it is unclear whether the improved performance is due to a lack of tuning the baseline FNO. I commend the authors for, as suggested by the reviewers, running more extrapolation experiments. However, I believe the reviewers also made a point about considering (training) times much longer than 1, as even the original FNO paper did this for Navier Stokes (NS) with T=50. 

Overall, the paper provides modest improvement wrt FNOs, although it does extend its capabilities to interpolation and extrapolation. The paper will also benefit from providing a brief overview of FNOs.
Decision: Reject