Enforcing physics-based algebraic constraints for inference of PDE models on unstructured grids

1 INTRODUCTION . Multiple works have shown the capability of neural networks to solve complex physical problems and learn the behavior of physical systems from data . Examples include learning and solving ordinary differential equations ( ODEs ) [ 6 ] , partial differential equations ( PDEs ) [ 28 ; 20 ] and rigid body dynamics [ 31 ; 5 ] . Purely data-driven models are typically not forced to satisfy physical constraints of the system that generated the data . This might lead to unrealistic predictions that violate some known properties of the underlying physical system . Incorporation of relevant constraints allows to make a better use of the available data and makes predictions more physically plausible . The field dealing with physics-constrained learning is diverse and offers many approaches to adding constraints to models . We refer the reader to many reviews for details [ 30 ; 3 ; 36 ; 19 ] . The approach we consider in this work is based on forcing a model to satisfy algebraic constraints represented by a set of equalities and inequalities . This is the most commonly used approach which allows to represent a wide range of constraints and has been shown to work well in many cases [ 18 ; 17 ; 25 ] . However , while many constraints can be represented algebraically , it is not always clear how to evaluate and enforce them . Currently available approaches to enforcing algebraic constraints are limited to uniform grids and have a very narrow range of constraints they can enforce ( e.g . only pointwise , or specific differential constraints ) , see Section 5 for details of related work . Such approaches can be readily applied to models based on convolutional neural networks ( CNNs ) but can not be extended to recently developed models based on graph neural networks ( GNNs ) [ 33 ; 27 ; 15 ] and other models working on unstructured grids . We propose a much more general method which allows to enforce pointwise , differential and integral constraints on unstructured spatial grids and demonstrate its applicability in learning of PDE-driven dynamical systems and distributions of physical fields . The method is based on using a models ’ s output at the nodes of a grid to construct an interpolant and applying constraints directly to that interpolant ( Section 3 ) . Code and data will be made publicly available . 2 BACKGROUND . PDE-driven dynamical systems . Many physical systems can be described in terms of PDEs . Such systems are defined on a bounded domain on which they evolve over time . We consider continuous dynamical systems with state u ( t , x ) ∈ Rp that evolves over time t ∈ R≥0 and spatial locations x ∈ Ω ⊂ RD . For physical systems , D is typically limited to { 1 , 2 , 3 } although our method will work with any value of D. We assume the system is governed by an unknown PDE ∂u ( t , x ) ∂t = F ( x , u ( t , x ) , ∇xu ( t , x ) , ∇2xu ( t , x ) , ... ) ( 1 ) which describes the temporal evolution of the system in terms of the locations x , state u and its first and higher-order partial derivatives w.r.t . x . The goal of a data-driven PDE model is to learn the dynamics F from data . Data for learning F is collected by measuring the state of the system at observation locations ( x1 , . . . , xN ) over increasing time points ( t0 , . . . , tM ) . This results in a dataset ( y ( t0 ) , . . . , y ( tM ) ) , where y ( ti ) = ( u ( ti , x1 ) , . . . , u ( ti , xN ) ) is a collection of observations . The dataset is used to train the model to predict ( y ( t1 ) , . . . , y ( tM ) ) starting from the initial state y ( t0 ) . Training is typically done by minimizing an average loss between the model ’ s predictions u ( t ) and the data y ( t ) . PDE models differ in restrictions they impose on time points ( temporal grid ) and observation locations ( spatial grid ) . Some models require both grids to be uniform [ 23 ] , other models relax these requirements and allow arbitrary spatial [ 27 ] and spatio-temporal grids [ 15 ] . We build our algebraic constraints method using the model from [ 15 ] as the most general one . The model is based on application of the method of lines [ 32 ] to Equation 1 which results into a system of ODEs u̇ ( t ) : = du ( t , x1 ) dt ... du ( t , xN ) dt ≈ Fθ ( x1 , xN ( 1 ) , u1 , uN ( 1 ) ) ... Fθ ( xN , xN ( N ) , uN , uN ( N ) ) ( 2 ) which approximates the solution of Equation 1 at the observation locations xi using their neighboring points N ( i ) , where xN ( i ) and uN ( i ) are the neighbors ’ positions and states respectively , and ui is u ( t , xi ) . The approximate solution converges to the true solution as N increases . The true dynamics F is approximated by a parametric model Fθ whose parameters θ are learned by minimizing the difference between the model ’ s predictions u ( t ) = u ( 0 ) + ∫ t 0 u̇ ( τ ) dτ ( 3 ) and the data y ( t ) . The integral in Equation 3 is solved using a numerical ODE solver . In [ 15 ] , the function Fθ was represented by a graph neural network ( GNN ) which takes states and locations at an observation point i and its neighboring points N ( i ) . The observation points are connected into a grid using Delaunay triangulation which allows to naturally defineN ( i ) as a set of points connected to the point i . However , Fθ can be represented by other models and a different neighbor selection criterion can be used . The model parameters θ are learned by minimizing the MSE between y ( t ) and u ( t ) Ldata = 1 M M∑ i=1 ‖u ( ti ) − y ( ti ) ‖22 . ( 4 ) The gradient of Ldata w.r.t . θ is evaluated using the adjoint method as shown in [ 7 ] . Generative Adversarial Networks One of the tasks that we consider is learning distributions of physical fields . For that purpose we utilize generative adversarial networks ( GANs ) . A GAN is a generative model consisting of a generator and a discriminator [ 12 ] . The generator , G , learns to transform a random variable Z ∼ pZ over a latent space Z to the data space Y in such a way that the discriminator , D , can not tell the difference between samples generated by G and samples from the data distribution pdata . Both , G and D are learned by solving the following minimax problem min G max D V ( G , D ) = EY∼pdata [ logD ( Y ) ] + EZ∼pZ [ log ( 1−D ( G ( Z ) ) ) ] . ( 5 ) Solution of this problem exists and is unique with the optimal generator perfectly mimicking the data distribution [ 12 ] . 3 METHODS . In this section we presents an approach to evaluating pointwise , differential and integral constraints on unstructured grids . Then , we demonstrate how this approach can be used to enforce arbitrary soft and linear hard constraints . 3.1 EVALUATING CONSTRAINTS ON UNSTRUCTURED GRIDS . We assume the data y ( t ) is available at observation points ( x1 , . . . , xN ) and time points ( t1 , . . . , tM ) and that a model makes predictions u ( t ) at these points . We assume the predictions to be evaluations of an unknown underlying function . Since the underlying function is unknown , we can not impose constraints on it directly . Instead , we approximate it by an interpolant uf ( t , x ) and impose constraints on uf ( t , x ) ( Figure 1 ) . The approximation is constructed from u ( t ) by placing a basis function at each xi and representing uf ( t , x ) as uf ( t , x ) = N∑ j=1 αj ( t ) φj ( x ) , ( 6 ) where φj is a scalar basis function at xj and αj ∈ Rp . The coefficients αj ( t ) are obtained from u ( t ) ( see Section 3.4 ) . Next , we show how to evaluate constraints on uf ( t , x ) using basic building blocks . To avoid cluttered notation , we consider equality constraints and assume u ( t , x ) , x ∈ R. Generalization to inequality constraints , vector fields and higher spatial dimensions is straightforward . Pointwise constraints . Consider points z = ( z1 , . . . , zK ) in Ω on which a pointwise constraint h ( uf ( t , zi ) ) = 0 should be evaluated . Assume the function h : R→ R is representable in terms of a finite number of functions γm ( uf ( t , zi ) ) : R → R indexed by m. For example , should the constraint be h ( uf ) = 3uf + u2f = 0 , then we would define γ1 ( uf ) = uf , γ2 ( uf ) = u2f and h ( uf ) = 3 · γ1 ( uf ) + γ2 ( uf ) = 0 . Then , h can be evaluated by evaluating each γm as γm ( uf ( t , zi ) ) = γm N∑ j=1 αj ( t ) φj ( zi ) = γm ( Φi , ·α ( t ) ) , ( 7 ) where α ( t ) = ( α1 ( t ) , . . . , αN ( t ) ) T , Φ is K-by-N matrix with elements Φi , j = φj ( zi ) , and Φi , · is the i ’ th row of Φ . Differential constraints . Consider the same setup as before but now h ( uf ( t , zi ) ) = 0 consists of differential operators and is representable in terms of a finite number of functions ∂ qγm ( uf ( t , zi ) ) ∂zqi : R→ R indexed bym , where the derivative order q could be different for eachm . For example , should the constraint be h ( uf ) = 3uf + uf · ∂u2f ∂x = 0 , then we would define γ1 ( uf ) = uf , γ2 ( uf ) = u 2 f and h ( uf ) = 3 · γ1 ( uf ) + γ1 ( uf ) · ∂γ2 ( uf ) ∂z = 0 . Then , h can be evaluated by evaluating each ∂qγm ( uf ( t , zi ) ) ∂zqi using the generalization of the chain rule ( Appendix A ) which contains only two types of terms . The first type of terms dγmduf , . . . , dqγm duqf can be evaluated using automatic differentiation while the second type of terms ∂uf∂zi , . . . , ∂quf ∂zqi can be evaluated as ∂quf ∂zqi = N∑ j=1 αj ( t ) ∂qφj ( zi ) ∂zqi = Φ ( q ) i , · α ( t ) , ( 8 ) where Φ ( q ) i , j = ∂qφj ( zi ) ∂zqi . Mixed partial derivatives can be handled in a similar way ( Appendix A ) . Integral constraints . Consider the same setup as before but with h ( uf ( t , x ) ) =∫ Ω τ ( uf ( t , x ) ) dx = 0 , where the function τ : R → R is representable in terms of functions γm ( uf ( t , zi ) ) : R→ R similarly to the pointwise constraints . Then , ∫ Ω τ ( uf ( t , x ) ) dx can be evalu- ated using a numerical integration technique , e.g . midpoint rule , Gaussian quadrature or Monte-Carlo integration , as ∫ Ω τ ( uf ( t , x ) ) dx ≈ K∑ i=1 τ ( uf ( t , zi ) ) µi , ( 9 ) where K is the number of integration points , µi are integration coefficients which depend on the grid and integration method , and τ ( uf ( t , zi ) ) is evaluated as in Equation 7 .

Summary: The authors present a method to incorporate constraints into the output of learnable PDE models. They cover point-wise, differential, and integral constraints. They achieve this by representing the PDE solution in a basis, as is common for the variational method (e.g. pseudo-spectral method, and finite element method). The neural network outputs interpolation coefficients for each time step into the future. To implement the constraints, they have two methods: ‘soft constraints’, whereby a constraint-breaking penalty is applied to the neural network output and ‘hard constraints’, whereby the output of the network is projected on to constraint-satisfying solutions. This latter method is achieved by solving a convex programme.  The authors test on a variety of benchmarks, demonstrating that their method works; although, it is hard to parse the experimental section for whether these results are significant.