Enforcing Physical Constraints in Neural Neural Networks through Differentiable PDE Layer

1 INTRODUCTION . Convolutional Neural Network ( CNN ) based deep learning architectures have achieved huge success in many tasks across computer vision , but their use in the physical sciences have only recently been explored . Many parallels exist between physical science problems and those in computer vision . For instance , grid-based simulations generate a physical scalar or vector field which can been compared to multidimensional arrays in computer vision . However , unlike computer vision problems , physical fields are often constrained by PDEs that arise from the governing equations of the physical system . For example , the Poisson equation of the form∇2φ = f is often encountered in heat diffusion problems , whereas the divergence-free ( also known as solenoidal ) conditions in the form of ∇ · φ = 0 is fundamental to magnetic fields , as well as incompressible fluid velocity fields to ensure conservation of mass . For meaningful application of deep learning to a range of important physical problems it is essential to enforce such spatial PDE constraints to guarantee physical consistency and reliability of the model output for scientific applications . Yet , general means of enforcing these constraints do not exist and the existing methods do not scale well with high dimensional , high resolution outputs . In this paper , we address this issue by proposing a novel differentiable PDE layer ( PDEL ) that efficiently enforces spatial PDE constraints for neural networks , at costs on par with a single CNN layer . We use spectral methods , which leverages the highly efficient Fast Fourier Transform ( FFT ) algorithm for enforcing such constraints . Using this formulation , we are able to exploit the structures of the spectral matrices corresponding to these differential operators that renders the entire layer O ( n log n ) for processing a 3 dimensional field of size n. The method is general for enforcing arbitrary linear combinations of differential operators on these fields , which encompasses physical constraints from a broad range of important scientific and engineering systems . We apply this hard constraining layer to the problem of turbulence superresolution , where we show that training with the hard constraining layer in-the-loop not only guarantees that the imposed constraint is strictly satisfied , but also generates solutions that are more accurate measured via a variety of fluid flow metrics . In summary , the main contributions of this paper are : • We propose the highly efficient differentiable spatial linear PDE layer ( PDEL ) , which strictly enforces linear spatial PDEs constraints . • We apply the PDE layer towards the superresolution task for turbulent flows , showing that training with hard constraints in-the-loop results in solutions that not only strictly satisfy the imposed constraint but also produce flow fields with more accurate fluid flow statistics . 2 BACKGROUND AND RELATED WORK . 2.1 CONSTRAINTS IN NEURAL NETWORKS . Many studies in machine learning have considered imposing some form of constraints for their respective applications. ? proposed an approach for constraining the prediction of a discriminative predictor and showed that a Gaussian Process can be forced to satisfy linear and quadratic constraints. ? proposed training a kernalized latent variable model that imposes equality and inequality constraints. ? proposed a constrained CNN , which phrases the training objective as a biconvex optimization for linear models , which is then relaxed to nonlinear deep networks for any set of linear constraints on the output space. ? proposed an alternative approach by randomly subsampling a set of constraints at each optimization step and projecting the gradients onto the feasible solution space . OptNet ( ? ) solves a generic quadratic programming problem differentiably within the neural network , but its cubed complexity does not handle high dimensional output. ? proposed parameterizing the feasible solution space for imposing inequality constraints . However , methods to impose physical constraints into machine learning and deep learning models remains largely unexplored . 2.2 APPLICATIONS OF MACHINE LEARNING FOR PDE AND TURBULENCE . More recently there have been some work in the literature to apply machine learning to PDE and physical problems . ? ? ? developed physics informed deep learning frameworks for assimilating observational data . In the context of applying machine learning methodologies to turbulence problems , earlier works using data-driven modeling approaches have used optimization and Bayesian inference approaches to calibrate existing turbulence models ( ? ? ? ? ? ) . With the advancement of efficient and accurate modeling tools in machine learning , recent studies have looked at data driven approaches for turbulence modeling ( ? ) , including the direct use of random forests for modeling Reynolds-Averaged Navier Stokes ( RANS ) model errors ( ? ) , the use of multilayer perceptrons for modeling Reynolds stress anisotropy tensor from simulation data ( ? ) , and using random forests to predict mean velocities of turbulent flow fields. ? use a random forest to compute particle location and ? use a CNN to approximate part of a numerical PDE scheme . A recent application of deep learning for generating realistic fluid flow fields is TempoGAN ( ? ) that uses a specialized discriminator network for temporal coherence . None of these methods , however , enforce constraints that are necessary for physically consistent fluid flow fields. ? addressed this with a customized loss function for the divergence-free constraint of fluid flow , but since it is a loss-based soft constraint , the conditions are ultimately not exactly satisfied . 3 PROBLEM STATEMENT . The main focus of this paper is to introduce a novel and efficient method for imposing spatial linear PDE constraints to the outputs of Convolutional Neural Networks ( CNNs ) . This is discussed in the context of underdetermined systems , since solutions do not exist for overdetermined systems , while solutions for determined systems do not fit in the context of constraining the outputs . More specifically , given the output of the network to be a discretization of a 3D vector field f : R3 7→ R3 and a linear spatial PDE operator A ( ( ∂∂xj ) 0 , ( ∂∂xj ) 1 , · · · ) that maps vector fields to scalar fields Af : R3 7→ R , we seek a means of efficiently imposing the spatial linear PDE constraints within CNNs , i.e. , Af = b ( 1 ) Note that this form encompasses a wide range of physically relevant constraints . In particular , all spatial PDE constraints composed of divergence , curl , Laplacian and other higher order partial differential terms in linear combination may be expressed in this form . Depending on the domain of application , this includes mass conservation for incompressible fluid flows , the heat equation , the wave equation , Laplace ’ s equation , Helmholtz equation , Klein-Gordon equation , and Poisson ’ s equation . For the important constraint of mass conservation in incompressible flows , we investigate the divergence-free ( solenoidal ) constraint of : ∇ · f = ∑ j ∂ ∂xj fj = 0 ( 2 ) 4 METHODS . Before presenting our proposed method for enforcing the solenoidal condition on CNN outputs , we present an overview of two commonly utilized strategies for enforcing general linear constraints , which we will compare and benchmark against in the experiments section ( Sec . 5 ) . We first discuss enforcing linear constraints on the outputs of the neural network , where we have a neural network that learns the function mapping f : Rt 7→ Rm , where the function f ( x ; θ ) is parameterized by learnable parameters θ , and is subject to the linear constraint Af ( x ; θ ) = b , where A ∈ Rn×m , b ∈ Rn . For this to be an underconstrained system , we have n < m . 4.1 GENERALIZED LINEAR CONSTRAINTS . Two forms of constraints are possible for explicitly enforcing a certain set of constraints for neural network outputs : soft constraints and hard constraints . Soft constraints are easy to implement , by adding a differentiable residual loss for penalizing the network during training time for violating the explicit constraints . For simplicity , let y : = f ( x ; θ ) . In the conventional unconstrained case , assume the neural network is trained under the differentiable loss function L ( f ( x ; θ ) ) , in the constrained case , the loss function can be augmented by an additional residual loss term defined by : Lc ( θ ) = L ( θ ) + α ( ( Ay − b ) T ( Ay − b ) ) ( 3 ) where α is a hyper-parameter weighing the two loss functions that can be difficult to determine and vary between applications . Although easy to implement , soft constraints provide no guarantees on the solutions satisfying the imposed constraint . Hard linear constraints can be enforced by posing the problem as a constrained optimization problem for seeking the closest point in the solution space subject to the constraints , which can be solved by satisfying the Karush-Kuhn-Tucker ( KKT ) condition . The result of the projection step can be written as the stationary point of the Lagrangian : min ŷ max λ L ( ŷ , λ ) ( 4 ) where we have the Lagrangian as : L ( ŷ , λ ; y ) = 1 2 ( y − ŷ ) T ( y − ŷ ) + λT ( Ay − b ) ( 5 ) ∂L ∂ŷ = y − ŷ + λTA ( 6 ) The KKT condition leads to the following linear system , the solution of which involves solving a linear system of dimensions ( m + n ) × ( m + n ) . Given that the linear system is symmetric and positive definite , the solution can be sought by inverting the system : [ I AT A 0 ] [ ŷ λ ] = [ y b ] ⇒ [ ŷ λ ] = [ I AT A 0 ] −1 [ y b ] ( 7 ) While this approach is general for enforcing arbitrary linear constraints on arbitrary network outputs , it is difficult to scale it to higher dimensions , and particularly difficult for 2-dimensional and 3- dimensional outputs , by direct matrix inversion followed by matrix multiplication . 4.2 SPECTRAL METHODS . First , we introduce and review the spectral methods ( ? ) for discretizing the spatial PDE operators . Spectral methods are a class of numerical methods that computes the spatial partial derivatives of a field based on the spectral decomposition of the signal . By decomposing the original signal into a linear combination with respect to trigonometric basis functions of varying wavenumbers ( or frequencies ) , the spatial derivatives with respect to the trigonometric basis functions can be easily and efficiently computed . The Fast Fourier Transform ( FFT ) is a well known algorithm for efficiently computing the Direct Fourier Transform ( DFT ) of uniform discrete signals . The multidimensional FFT and inverse FFT respectively compute the following : F ( k ) = N−1∑ n=0 f ( n ) e−i2πk· ( n/N ) ; f ( n ) = 1 N1N2N3 N−1∑ k=0 F ( k ) ei2πk· ( n/N ) ( 8 ) where F ( k ) = FFT ( f ( n ) ) , f ( n ) = IFFT ( F ( k ) ) , spatial indices n = ( n1 , n2 , n3 ) , nj ∈ { 0 , 1 , · · · , Nj − 1 } and spectral indices k = ( k1 , k2 , k3 ) , kj ∈ { 0 , 1 , · · · , Nj − 1 } . The spatial derivative with respect to xj can be computed by : ∂ ∂xj f ( n ) = IFFT ( ikjF ( k ) ) ( 9 ) In matrix form , for the t-th component of a 3 dimensional vector field : Ft , taking its flattened vector form , and taking the flattened vector form of the wavenumber kj corresponding to the dimension xj , the spatial derivative with respect to xj can be computed using matrix multiplication : ∂ ∂xj Ft = diag ( ikj ) Ft ( 10 ) where diag ( ) converts a vector into a corresponding diagonal matrix . In general , arbitrary linear combination of spatial derivatives of varying orders can be computed using a single diagonal matrix multiplication : ( ∑ j ∑ r cjr ( ∂ ∂xj ) r ) Ft = diag ( ∑ j ∑ r cjr ( ikj ) r ) Ft : = AtFt ( 11 ) where At is a diagonal matrix for the spatial derivatives corresponding to the t-th component of the vector field , that is a polynomial of ikj , and A = [ A1 , A2 , A3 ] .

Summary: The paper describes a way to efficiently enforce physical constraints expressed by linear PDEs on the output of a neural network. The idea is to have, as a last layer of the network, a projection onto the constrained solution space, and to back-propagate through it. That projection layer is made efficient for high-dimensional outputs via the fast Fourier transform (FFT), exploiting a well-known numerical trick. Importantly, the proposed strategy is very general, and can indeed be used with any PDE constraint that is a linear combination of differential operators.