Semi-supervised learning of partial differential operators and dynamical flows

1 INTRODUCTION . The evolution of classical and quantum physical dynamical systems in space and time is generically modeled by non-linear partial differential equations . Such are , for instance , Einstein equations of General Relativity , Maxwell equations of Electromagnetism , Schrödinger equation of Quantum Mechanics and Navier-Stokes ( NS ) equations of fluid flows . These equations , together with appropriate initial and boundary conditions , provide a complete quantitative description of the physical world within their regime of validity . Since these dynamic evolution settings are governed by partial differential operators that are often highly non-linear , it is rare to have analytical solutions for dynamic systems . This is especially true when the system contains a large number of interacting degrees of freedom in the non-linear regime . Consider , as an example , the NS equations , which describe the motion of viscous fluids . In the regime of high Reynolds numbers of the order of one thousand , one observes turbulences , in which all symmetries are broken and all analytical techniques fail . The solution to these ( deterministic ) equations seems almost random and is very sensitive to the initial conditions . Many numerical techniques have been developed for constructing and analysing the solutions to fluid dynamics systems . However , the complexity of these solvers grows quickly as the spacing in the grid that is used for approximating the solution is reduced and the degrees of freedom of the interacting fluid increases . Given the theoretical and practical importance of constructing solutions to these equations , it is natural to ask whether neural networks can learn such evolution equations and construct new solutions . The two fundamental questions are : ( i ) The ability to generalize to initial conditions that are different from those presented in the training set , and ( ii ) The ability to generalize to new time points beyond the fixed grid points provided during training . The reason to hope that such tasks can be performed by machine learning is that despite the seemingly random behaviour of , e.g . fluid flows in the turbulent regime , there is an underlying low-entropy structure that can be learnt . Indeed , in diverse cases , neural network-based solvers have been shown to provide comparable results to other numerical methods , while utilizing fewer resources . Our Contributions We present a hyper-network based solver combined with a Fourier Neural Operator architecture . • Our hyper-network architecture treats time and space separately . Utilizing a data set of initial conditions and the corresponding solutions at a labeled fixed time , the network learns a large class of time evolution PDEs . • Our network successfully propagates initial conditions in discrete time steps by implementing the general composition properties of the partial differential operators . • Our solutions improve the learning accuracy at the supervision time-points . • Our solutions are able to interpolate and extrapolate to arbitrary ( unlabelled ) times . • We thoroughly test our method on various time evolution PDEs , including nonlinear fluid flows in one , two and three spatial dimensions . 2 RELATED WORK . Hypernetworks . While conventional networks employ a fixed set of pre-determined parameters , which is independent of the input , the hypernetwork scheme , invented multiple times , and coined by Ha et al . ( 2016 ) , allows the parameters of a neural network to explicitly rely on the input by combining two neural networks . The first neural network , called the hypernetwork , processes the input or part of it and outputs the weights of a second neural network . The second network , called the primary network , has a fixed architecture and weights that vary based on the input . It returns , given its input , the final output . This framework was used successfully in a variety of tasks , ranging from computer vision ( Littwin & Wolf , 2019 ) , continual learning ( von Oswald et al. , 2019 ) , and language modeling ( Suarez , 2017 ) . While it is natural to learn functions with hypernetworks , since the primary network can be seen as a dynamic , input-dependent function , we are not aware of any previous work that applies this scheme for recovering physical operators . Neural network-based PDE solvers . Due to the well-known limitations of traditional PDE solvers on one hand , and in light of new advances made in the field of neural networks on the other , lately we have witnessed very significant progress in the field of neural network-based PDE solvers ( Karniadakis et al. , 2021 ) . Neural network-based PDE solvers can be roughly divided into two groups according to the resource they utilize for learning : data-driven and model-based . Data-driven solvers use a large dataset containing initial conditions and final states , all at the same timepoint T to train ( Zhu & Zabaras , 2018 ) . These solvers predict the solution for new initial conditions at the same T , but can not be interpolated or extrapolated to times that are not increments of T . However , as the PDE is not required for such solvers , they can be used in cases where the underlying PDE is unknown . Data-driven approaches often reduce the problem setting of PDE solvers to the well-known problem setting of image-to-image mapping in computer vision . Following this approach , it is customary to use networks such U-Net ( Ronneberger et al. , 2015 ) . Model-based solvers , known as Physics Informed Neural Networks ( PINNs ) ( Raissi et al. , 2019 ) , harness the differential operator itself for supervision . This is done by defining a loss , the residual of the PDE . These solvers do not require a training dataset and can provide solutions for arbitrary times . Both approaches learn solutions on a specific discretized grid , which poses a limitation for any practical applications . Recently , a mesh-invariant data-driven direction has been proposed ( Lu et al. , 2019 ; Bhattacharya et al. , 2020 ; Nelsen & Stuart , 2021 ; Li et al. , 2020b ; Patel et al. , 2021 ) . The mesh invariance is obtained by learning operators rather than mappings between initial and final states . Li et al . ( 2020c ) have advanced the mesh-invariant line of work by introducing Fourier Neural Operators ( FNO ) . FNOs utilize both a convolution layer in real space and a Fourier integral layer in the Fourier domain . It has been shown that the FNO solver outperforms previous solvers in a number of important PDEs . FNO has a major limitation : the method ( in the framework of learning from pairs of initial states and final solutions ) can not be used to provide solutions for interpolation and arbitrary extrapolations . Knowing the solutions along the complete time evolution trajectory is highly valuable for theoretical and practical reasons and provides means for learning new dynamical principles . 3 PARTIAL DIFFERENTIAL EQUATIONS . Consider a d-dimensional vector field v ( x , t ) : Td×R→ Rd , where x = ( x1 , . . . , xd ) are periodic spatial coordinates xi ' xi + 2π on the d-dimensional torus , Td , and t is the time coordinate . The vector field evolves dynamically from the initial condition v ( x , t = 0 ) : Td → Rd according to a non-linear PDE of the form , ∂tv ( x , t ) = Lv ( x , t ) , ( 1 ) where L is a differential operator that does not depend explicitly on time , and has an expansion in v and its spatial derivatives . We assume that given a regular bounded initial vector field there is a unique regular solution to equation 1 . Since L does not depend explicitly on time , we can formally write the solution of such equations as v ( x , t ) = etLv ( x , 0 ) ≡ Φtv ( x , 0 ) . ( 2 ) Because of dissipation terms , such as the viscosity term in the NS equations , solutions to equation 1 generically break time reversal invariance ( t→ −t , v → −v ) . Furthermore , the total energy of the system is non-increasing as a function of time since we are not injecting energy at t 6= 0 , ∂t ∫ ddx v · v 2 ≤ 0 . ( 3 ) Equation 3 serves as a consistency on the network solutions . In this work , we consider the following non-linear PDEs , ( i ) Burgers equation describing one-dimensional compressible fluid flows with scalar velocity field v ( x , t ) , ∂tv + v∇v = ν∆v , ( 4 ) where ν is the kinematic viscosity , ∇ the gradient and ∆ is the Laplacian . ( ii ) Generalized one-dimensional Burgers equation with a parameter q = 2 , 3 , 4 , ∂tv + v q∇v = ν∆v . ( 5 ) ( iii ) One-dimensional Chafee–Infante equation with a constant λ parameter that models reactiondiffusion dynamics , ∂tv + λ ( v 3 − v ) = ∆v . ( 6 ) ( iv ) Two-dimensional Burgers equation for a two-dimensional vector field , v ( x , t ) = ( v1 , v2 ) , that describes compressible fluid flows in two space dimensions : ∂tv + ( v · ∇ ) v = ν∆v . ( 7 ) ( v ) Two and three-dimensional incompressible NS equations , ∂tv + ( v · ∇ ) v = −∇p+ ν∆v , ∇ · v = 0 , ( 8 ) where p is the fluid ’ s pressure , v ( x , t ) = ( v1 , . . . , vd ) and ν is the kinematic viscosity . 4 METHOD . Typically , a data-driven PDE solver , such as a neural network , evolves unseen velocity fields , v ( x , t = 0 ) to a fixed time T , ΦTv ( x , t = 0 ) = v ( x , T ) , ( 9 ) by learning from a set of i = 1 . . . N initial conditions sampled at t = 0 , vi ( x , t = 0 ) , and their corresponding time-evolved solutions of vi ( x , t = T ) of equation 1 . We generalize ΦT , to propagate solutions at intermediate times , 0 ≤ t ≤ T , by elevating a standard neural architecture to a hyper-network one . A hyper-network architecture is a composition of two neural networks , a primary network fw ( x ) and a hypernetwork gθ ( t ) with learned parameters θ , such that the parameters w of f are given as the output of g. Unlike common neural architectures , where a single input is mapped to a single output , in this construction , the input t is mapped to a function , fgθ ( t ) , which maps x to its output . Applying this to our case we have , Φtv ( x , 0 ) = fgθ ( t ) ( v ( x , 0 ) ) = v ( x , t ) . ( 10 ) This architecture may be used to learn not only the time-evolved solutions at t = T , but also intermediate solutions , at 0 < t < T , without explicitly providing the network with any intermediate time solution . This task may be accomplished by utilizing general consistency conditions , which apply to any equation of the form equation 1 , Φt1Φt2 = Φt2Φt1 = Φt1+t2 . ( 11 ) Notice that Φ0 = Id follows from equation 11 .

Summary: The paper presents a model and a loss function for approximation of the solution map of certain partial differential equations.  The idea is to use HyperNetworks: the mapping of the initial condition to time t done by model f($\theta$, x), and the parameters are predicted by another neural network. The models utilize the recently proposed Fourier Neural Operator (FNO) for the mapping model, and a fully-connected network for the hypernetwork. Several losses are proposed to ensure consistency of the proposed models, including the composition and reconstruction. The tests are done on a battery of examples, and compared to other neural network baselines the error is smaller. A theorem is proved, but is rather straightforward.