Enforcing Physical Constraints in Neural Neural Networks through Differentiable PDE Layer

1 INTRODUCTION . Convolutional Neural Network ( CNN ) based deep learning architectures have achieved huge success in many tasks across computer vision , but their use in the physical sciences have only recently been explored . Many parallels exist between physical science problems and those in computer vision . For instance , grid-based simulations generate a physical scalar or vector field which can been compared to multidimensional arrays in computer vision . However , unlike computer vision problems , physical fields are often constrained by PDEs that arise from the governing equations of the physical system . For example , the Poisson equation of the form∇2φ = f is often encountered in heat diffusion problems , whereas the divergence-free ( also known as solenoidal ) conditions in the form of ∇ · φ = 0 is fundamental to magnetic fields , as well as incompressible fluid velocity fields to ensure conservation of mass . For meaningful application of deep learning to a range of important physical problems it is essential to enforce such spatial PDE constraints to guarantee physical consistency and reliability of the model output for scientific applications . Yet , general means of enforcing these constraints do not exist and the existing methods do not scale well with high dimensional , high resolution outputs . In this paper , we address this issue by proposing a novel differentiable PDE layer ( PDEL ) that efficiently enforces spatial PDE constraints for neural networks , at costs on par with a single CNN layer . We use spectral methods , which leverages the highly efficient Fast Fourier Transform ( FFT ) algorithm for enforcing such constraints . Using this formulation , we are able to exploit the structures of the spectral matrices corresponding to these differential operators that renders the entire layer O ( n log n ) for processing a 3 dimensional field of size n. The method is general for enforcing arbitrary linear combinations of differential operators on these fields , which encompasses physical constraints from a broad range of important scientific and engineering systems . We apply this hard constraining layer to the problem of turbulence superresolution , where we show that training with the hard constraining layer in-the-loop not only guarantees that the imposed constraint is strictly satisfied , but also generates solutions that are more accurate measured via a variety of fluid flow metrics . In summary , the main contributions of this paper are : • We propose the highly efficient differentiable spatial linear PDE layer ( PDEL ) , which strictly enforces linear spatial PDEs constraints . • We apply the PDE layer towards the superresolution task for turbulent flows , showing that training with hard constraints in-the-loop results in solutions that not only strictly satisfy the imposed constraint but also produce flow fields with more accurate fluid flow statistics . 2 BACKGROUND AND RELATED WORK . 2.1 CONSTRAINTS IN NEURAL NETWORKS . Many studies in machine learning have considered imposing some form of constraints for their respective applications. ? proposed an approach for constraining the prediction of a discriminative predictor and showed that a Gaussian Process can be forced to satisfy linear and quadratic constraints. ? proposed training a kernalized latent variable model that imposes equality and inequality constraints. ? proposed a constrained CNN , which phrases the training objective as a biconvex optimization for linear models , which is then relaxed to nonlinear deep networks for any set of linear constraints on the output space. ? proposed an alternative approach by randomly subsampling a set of constraints at each optimization step and projecting the gradients onto the feasible solution space . OptNet ( ? ) solves a generic quadratic programming problem differentiably within the neural network , but its cubed complexity does not handle high dimensional output. ? proposed parameterizing the feasible solution space for imposing inequality constraints . However , methods to impose physical constraints into machine learning and deep learning models remains largely unexplored . 2.2 APPLICATIONS OF MACHINE LEARNING FOR PDE AND TURBULENCE . More recently there have been some work in the literature to apply machine learning to PDE and physical problems . ? ? ? developed physics informed deep learning frameworks for assimilating observational data . In the context of applying machine learning methodologies to turbulence problems , earlier works using data-driven modeling approaches have used optimization and Bayesian inference approaches to calibrate existing turbulence models ( ? ? ? ? ? ) . With the advancement of efficient and accurate modeling tools in machine learning , recent studies have looked at data driven approaches for turbulence modeling ( ? ) , including the direct use of random forests for modeling Reynolds-Averaged Navier Stokes ( RANS ) model errors ( ? ) , the use of multilayer perceptrons for modeling Reynolds stress anisotropy tensor from simulation data ( ? ) , and using random forests to predict mean velocities of turbulent flow fields. ? use a random forest to compute particle location and ? use a CNN to approximate part of a numerical PDE scheme . A recent application of deep learning for generating realistic fluid flow fields is TempoGAN ( ? ) that uses a specialized discriminator network for temporal coherence . None of these methods , however , enforce constraints that are necessary for physically consistent fluid flow fields. ? addressed this with a customized loss function for the divergence-free constraint of fluid flow , but since it is a loss-based soft constraint , the conditions are ultimately not exactly satisfied . 3 PROBLEM STATEMENT . The main focus of this paper is to introduce a novel and efficient method for imposing spatial linear PDE constraints to the outputs of Convolutional Neural Networks ( CNNs ) . This is discussed in the context of underdetermined systems , since solutions do not exist for overdetermined systems , while solutions for determined systems do not fit in the context of constraining the outputs . More specifically , given the output of the network to be a discretization of a 3D vector field f : R3 7→ R3 and a linear spatial PDE operator A ( ( ∂∂xj ) 0 , ( ∂∂xj ) 1 , · · · ) that maps vector fields to scalar fields Af : R3 7→ R , we seek a means of efficiently imposing the spatial linear PDE constraints within CNNs , i.e. , Af = b ( 1 ) Note that this form encompasses a wide range of physically relevant constraints . In particular , all spatial PDE constraints composed of divergence , curl , Laplacian and other higher order partial differential terms in linear combination may be expressed in this form . Depending on the domain of application , this includes mass conservation for incompressible fluid flows , the heat equation , the wave equation , Laplace ’ s equation , Helmholtz equation , Klein-Gordon equation , and Poisson ’ s equation . For the important constraint of mass conservation in incompressible flows , we investigate the divergence-free ( solenoidal ) constraint of : ∇ · f = ∑ j ∂ ∂xj fj = 0 ( 2 ) 4 METHODS . Before presenting our proposed method for enforcing the solenoidal condition on CNN outputs , we present an overview of two commonly utilized strategies for enforcing general linear constraints , which we will compare and benchmark against in the experiments section ( Sec . 5 ) . We first discuss enforcing linear constraints on the outputs of the neural network , where we have a neural network that learns the function mapping f : Rt 7→ Rm , where the function f ( x ; θ ) is parameterized by learnable parameters θ , and is subject to the linear constraint Af ( x ; θ ) = b , where A ∈ Rn×m , b ∈ Rn . For this to be an underconstrained system , we have n < m . 4.1 GENERALIZED LINEAR CONSTRAINTS . Two forms of constraints are possible for explicitly enforcing a certain set of constraints for neural network outputs : soft constraints and hard constraints . Soft constraints are easy to implement , by adding a differentiable residual loss for penalizing the network during training time for violating the explicit constraints . For simplicity , let y : = f ( x ; θ ) . In the conventional unconstrained case , assume the neural network is trained under the differentiable loss function L ( f ( x ; θ ) ) , in the constrained case , the loss function can be augmented by an additional residual loss term defined by : Lc ( θ ) = L ( θ ) + α ( ( Ay − b ) T ( Ay − b ) ) ( 3 ) where α is a hyper-parameter weighing the two loss functions that can be difficult to determine and vary between applications . Although easy to implement , soft constraints provide no guarantees on the solutions satisfying the imposed constraint . Hard linear constraints can be enforced by posing the problem as a constrained optimization problem for seeking the closest point in the solution space subject to the constraints , which can be solved by satisfying the Karush-Kuhn-Tucker ( KKT ) condition . The result of the projection step can be written as the stationary point of the Lagrangian : min ŷ max λ L ( ŷ , λ ) ( 4 ) where we have the Lagrangian as : L ( ŷ , λ ; y ) = 1 2 ( y − ŷ ) T ( y − ŷ ) + λT ( Ay − b ) ( 5 ) ∂L ∂ŷ = y − ŷ + λTA ( 6 ) The KKT condition leads to the following linear system , the solution of which involves solving a linear system of dimensions ( m + n ) × ( m + n ) . Given that the linear system is symmetric and positive definite , the solution can be sought by inverting the system : [ I AT A 0 ] [ ŷ λ ] = [ y b ] ⇒ [ ŷ λ ] = [ I AT A 0 ] −1 [ y b ] ( 7 ) While this approach is general for enforcing arbitrary linear constraints on arbitrary network outputs , it is difficult to scale it to higher dimensions , and particularly difficult for 2-dimensional and 3- dimensional outputs , by direct matrix inversion followed by matrix multiplication . 4.2 SPECTRAL METHODS . First , we introduce and review the spectral methods ( ? ) for discretizing the spatial PDE operators . Spectral methods are a class of numerical methods that computes the spatial partial derivatives of a field based on the spectral decomposition of the signal . By decomposing the original signal into a linear combination with respect to trigonometric basis functions of varying wavenumbers ( or frequencies ) , the spatial derivatives with respect to the trigonometric basis functions can be easily and efficiently computed . The Fast Fourier Transform ( FFT ) is a well known algorithm for efficiently computing the Direct Fourier Transform ( DFT ) of uniform discrete signals . The multidimensional FFT and inverse FFT respectively compute the following : F ( k ) = N−1∑ n=0 f ( n ) e−i2πk· ( n/N ) ; f ( n ) = 1 N1N2N3 N−1∑ k=0 F ( k ) ei2πk· ( n/N ) ( 8 ) where F ( k ) = FFT ( f ( n ) ) , f ( n ) = IFFT ( F ( k ) ) , spatial indices n = ( n1 , n2 , n3 ) , nj ∈ { 0 , 1 , · · · , Nj − 1 } and spectral indices k = ( k1 , k2 , k3 ) , kj ∈ { 0 , 1 , · · · , Nj − 1 } . The spatial derivative with respect to xj can be computed by : ∂ ∂xj f ( n ) = IFFT ( ikjF ( k ) ) ( 9 ) In matrix form , for the t-th component of a 3 dimensional vector field : Ft , taking its flattened vector form , and taking the flattened vector form of the wavenumber kj corresponding to the dimension xj , the spatial derivative with respect to xj can be computed using matrix multiplication : ∂ ∂xj Ft = diag ( ikj ) Ft ( 10 ) where diag ( ) converts a vector into a corresponding diagonal matrix . In general , arbitrary linear combination of spatial derivatives of varying orders can be computed using a single diagonal matrix multiplication : ( ∑ j ∑ r cjr ( ∂ ∂xj ) r ) Ft = diag ( ∑ j ∑ r cjr ( ikj ) r ) Ft : = AtFt ( 11 ) where At is a diagonal matrix for the spatial derivatives corresponding to the t-th component of the vector field , that is a polynomial of ikj , and A = [ A1 , A2 , A3 ] .

Summary: This work develops a differentiable spectral projection layer to enforce spatial PDE constraints using spectral methods, to achieve the introduction of the physical constraints in the end-to-end network without damaging the intrinsic property of the network. Analysis of computational cost shows the proposed layer is cheaper than the convolutional layer. The experimental comparison demonstrates the superiority of the proposed method. In my viewpoint, the novelty of this paper is somewhat novel.