Enforcing Physical Constraints in Neural Neural Networks through Differentiable PDE Layer
Authors: Chiyu "Max" Jiang
Karthik Kashinath
Prabhat
Philip Marcus
Venue:  (2020)
Keywords: PDE
Hard Constraints
Turbulence
Super-Resolution
Spectral Methods

Abstract: Recent studies at the intersection of physics and deep learning have illustrated successes in the application of deep neural networks to partially or fully replace costly physics simulations. Enforcing physical constraints to solutions generated
by neural networks remains a challenge, yet it is essential to the accuracy and trustworthiness of such model predictions. Many systems in the physical sciences are governed by Partial Differential Equations (PDEs). Enforcing these as hard
constraints, we show, are inefficient in conventional frameworks due to the high dimensionality of the generated fields. To this end, we propose the use of a novel differentiable spectral projection layer for neural networks that efficiently enforces
spatial PDE constraints using spectral methods, yet is fully differentiable, allowing for its use as a layer in neural networks that supports end-to-end training. We show that its computational cost is cheaper than a regular convolution layer. We apply it to
an important class of physical systems – incompressible turbulent flows, where the divergence-free PDE constraint is required. We train a 3D Conditional Generative Adversarial Network (CGAN) for turbulent flow super-resolution efficiently, whilst
guaranteeing the spatial PDE constraint of zero divergence. Furthermore, our empirical results show that the model produces realistic flow fields with more accurate flow statistics when trained with hard constraints imposed via the proposed
novel differentiable spectral projection layer, as compared to soft constrained and unconstrained counterparts.

Under review as a conference paper at ICLR 2020
ENFORCING PHYSICAL CONSTRAINTS IN NEURAL NETWORKS THROUGH DIFFERENTIABLE PDE LAYER
Anonymous authors Paper under double-blind review
ABSTRACT
Recent studies the intersection of physics and deep learning have illustrated successes in the application of deep neural networks to partially or fully replace costly physics simulations. Enforcing physical constraints solutions generated by neural networks remains a challenge, yet it is essential to the accuracy and trustworthiness of such model predictions. Many systems in the physical sciences are governed by Partial Differential Equations (PDEs). Enforcing these as hard constraints, we show, are inefficient in conventional frameworks due to the high dimensionality of the generated fields. To this end, propose the use of novel differentiable spectral projection layer for neural networks that efficiently enforces spatial PDE constraints using spectral methods, yet is fully differentiable, allowing for its use as layer in neural networks that supports end-to-end training. We show that its computational cost cheaper than regular convolution layer. We apply it important class of physical systems incompressible turbulent flows, where the divergence-free PDE constraint is required. We train 3D Conditional Generative Adversarial Network (CGAN) for turbulent flow superresolution efficiently, whilst guaranteeing the spatial PDE constraint of zero divergence. Furthermore, our empirical results show that the model produces realistic flow fields with more accurate flow statistics when trained with hard constraints imposed via the proposed novel differentiable spectral projection layer, as compared to soft constrained and unconstrained counterparts.
INTRODUCTION
Convolutional Neural Network (CNN) based deep learning architectures have achieved huge success in many tasks across computer vision, but their use in the physical sciences have only recently been explored. Many parallels exist between physical science problems and those computer vision. For instance, grid-based simulations generate a physical scalar vector field which can been compared to multidimensional arrays in computer vision. However, unlike computer vision problems, physical fields are often constrained by PDEs that arise from the governing equations of the physical system. For example, the Poisson equation the form V = f is often encountered in heat diffusion problems, whereas the divergence-free (also known as solenoidal) conditions in the form of V = is fundamental to magnetic fields, as well as incompressible fluid velocity fields to ensure conservation of mass. For meaningful application of deep learning range of important physical problems it is essential to enforce such spatial PDE constraints to guarantee physical consistency and reliability of the model output for scientific applications. Yet, general means of enforcing these constraints do not exist and the existing methods do not scale well with high dimensional, high resolution outputs.
In this paper, we address this issue by proposing a novel differentiable PDE layer (PDEL) that efficiently enforces spatial PDE constraints for neural networks, at costs on par with single CNN layer. We use spectral methods, which leverages the highly efficient Fast Fourier Transform (FFT) algorithm for enforcing such constraints. Using this formulation, we are able to exploit the structures of the spectral matrices corresponding to these differential operators that renders the entire layer On logn) for processing a 3 dimensional field of size n. The method is general for enforcing arbitrary linear combinations of differential operators these fields, which encompasses physical constraints from a broad range of important scientific and engineering systems. We apply this hard constraining layer to the problem of turbulence superresolution, where we show that training with the

---

Under review as a conference paper at ICLR 2020
hard constraining layer in-the-loop not only guarantees that the imposed constraint is strictly satisfied, but also generates solutions that are more accurate measured via a variety of fluid flow metrics. In summary, the main contributions of this paper are:
We propose the highly efficient differentiable spatial linear PDE layer (PDEL), which strietly enforces linear spatial PDEs constraints.
We apply the PDE layer towards the superresolution task for turbulent flows, showing that training with hard constraints in in-the-loop results in solutions that not only strictly satisfy the imposed constraint but also produce flow fields with more accurate fluid flow statistics.
BACKGROUND AND RELATED WORK
2.1 CONSTRAINTS IN NEURAL NETWORKS
Many studies in machine learning have considered imposing some form of constraints for their respective applications. ? proposed approach for constraining the prediction of discriminative predictor and showed that a Gaussian Process can be forced to satisfy linear and quadratic constraints. proposed training kernalized latent variable model that imposes equality and inequality constraints. ? proposed constrained CNN, which phrases the training objective as a biconvex optimization for linear models, which then relaxed nonlinear deep networks for any set of linear constraints on the output space. ? proposed an alternative approach by randomly subsampling a set of constraints each optimization step and projecting the gradients onto the feasible solution space. OptNet (?) solves generic quadratic programming problem differentiably within the neural network, but its cubed complexity does not handle high dimensional output. ? proposed parameterizing the feasible solution space for imposing inequality constraints. However, methods to impose physical constraints into machine learning and deep learning models remains largely unexplored.
2.2 APPLICATIONS OF MACHINE LEARNING FOR PDE AND TURBULENCE
More recently there have been some work in the literature to apply machine learning to PDE and physical problems. ??? developed physics informed deep learning frameworks for assimilating observational data. In the context of applying machine learning methodologies turbulence problems, earlier works using data-driven modeling approaches have used optimization and Bayesian inference approaches to calibrate existing turbulence models (?????). With the advancement of efficient and accurate modeling tools in machine learning, recent studies have looked at data driven approaches for turbulence modeling (?), including the direct use of random forests for modeling Seyyoods-Averageee Navier Stokes (RANS) model errors (?), the of multilayer perceptrons for modeling Reynolds stress anisotropy tensor from simulation data (?), and using random forests to predict mean velocities of turbulent flow fields. ? use a random forest compute particle location and ? use CNN to approximate part of numerical PDE scheme. A recent application of deep learning for generating realistic fluid flow fields is TempoGAN (?) that uses a specialized discriminator network for temporal coherence. None of these methods, however, enforce constraints that are necessary for physically consistent fluid flow fields. ? addressed this with a customized loss function for the divergence constraint of fluid flow, but since it is loss-based soft constraint, the conditions are ultimately not exactly satisfied.
PROBLEM STATEMENT
The main focus of this paper is to introduce a novel and efficient method for imposing spatial linear PDE constraints to the outputs of Convolutional Neural Networks (CNNs). This is discussed in the context of underdetermined systems, since solutions do not exist for overdetermined systems, while solutions for determined systems do not fit in the context of constraining the outputs. More specifically, given the output of the network to be discretization of 3D vector field f R3 - R3 and a linear spatial PDE operator A(( 8 yo, (ōzz) ",,),... ..).... that maps vector fields to scalar fields dzy)
Af 3 R, we seek means of efficiently imposing the spatial linear PDE constraints within
2

---

Under review as a conference paper at ICLR 2020
CNNs, i.e,,
(1) Note that this form encompasses a wide range of physically relevant constraints. In particular, all spatial PDE constraints composed of divergence, curl, Laplacian and other higher order partial differential terms in linear combination may be expressed in this form. Depending on the domain of application, this includes mass conservation for incompressible fluid flows, the heat equation, the wave equation, Laplace's equation, Helmholtz equation, Klein-Gordon equation, and Poisson's 's equation. For the important constraint mass conservation in incompressible flows, we investigate the divergence-free (solenoidal) constraint of:
-
à Oag
V f
f,
(2)
4 METHODS
Before presenting our proposed method for enforcing the solenoidal condition on CNN outputs, we present an overview of two commonly utilized strategies for enforcing general linear constraints, which we will compare and benchmark against in the experiments section (Sec. 5)
We first discuss enforcing linear constraints the outputs of the neural network, where have a neural network that learns the function mapping f R + RT, where the function f(æ; 0) is parameterized by learnable parameters 0, and is subject to the linear constraint (f 0) b, where A E IR x E R". For this to be an underconstrained system, we have n < m.
4.1 GENERALIZED LINEAR CONSTRAINTS
Two forms of constraints possible for explicitly enforcing certain set of constraints for neural network outputs: soft constraints and hard constraints.
Soft constraints easy implement, by adding differentiable residual loss for penalizing the network during training time for violating the explicit constraints. For simplicity, let y := f(;; 0). In the conventional unconstrained case, assume the neural network is trained under the differentiable loss function L(f(x 0)), in the constrained case, the loss function can be augmented by an additional residual loss term defined by:
(+) = L()) +
- b))
(3)
where a is aaper-parameter weighing the two loss functions that can be difficult to determine and vary between applications. Although easy to implement, soft constraints provide no guarantees on the solutions satisfying the imposed constraint.
Hard linear constraints can be enforced by posing the problem a constrained optimization problem for seeking the closest point in the solution space subject to the constraints, which can be (KKT) condition. The result of the projection step can
solved by satisfying the Marush-Kuhn-Tuce be written as the stationary point of the Lagrangian:
min max L(D.A)
(4)
X
where we have the Lagrangian as:
C(ú. X; 2 y T) D) + AA
(5)
dc
9 A
(6)
The KKT condition leads to the following linear system, the solution of which involves solving linear system of dimensions (m + n) x (m + n). Given that the linear system is symmetric and positive definite, the solution can be sought by inverting the system:
T A
A
T
A
y
A
0
b
3

---

Under review as a conference paper at ICLR 2020
While this approach is general for enforcing arbitrary linear constraints arbitrary network outputs, is difficult to scale it to higher dimensions, and particularly difficult for 2-dimensional and dimensional outputs, by direct matrix inversion followed by matrix multiplication.
4.2 SPECTRAL METHODS
First, introduce and review the spectral methods (?) for discretizing the spatial PDE operators. Spectral methods are class of numerical methods that computes the spatial partial derivatives of field based the spectral decomposition of the signal. By decomposing the original signal into a linear combination with respect to trigonometric basis functions of varying wavenumbers (or frequencies), the spatial derivatives with respect the trigonometric basis functions be easily and efficiently computed. The Fast Fourier Transform computing the Direct Fourier Transform (DFT) of uniform discrete signals. The multidimensional FFT and inverse FFT respectively compute the following:
(FFT) is well known algorithm for efficiently
N-1 Σ ==0
N-1 Flk) (2kk/N2)))
F(k)
- p(n)
(8)
NIN2N
where ((k) FFT ( (m)),, f(m) (()), spatial indices (n1, n2, na): ny
{0. 10.1, N₂ }} and spectral indices & = (ki, k2, ka), kj E {0, 10,1,... ... ,,
Nj - 1}. The spatial
derivative with respect 2, can be computed by:
0
((m) IFFT ((k) 0x,
In matrix form, for the (-th component of a 3 dimensional vector field: F,, taking its flattened vector form, and taking the flattened vector form of the wavenumber k, corresponding to the dimension zj. the spatial derivative with respect to z can be computed using matrix multiplication:
i
F. - diag(ik;) dr,
(10)
where diag() converts vector into a corresponding diagonal matrix. In general, arbitrary linear combination of spatial derivatives of varying orders can be computed using single diagonal matrix multiplication:
0 ):.: ))))= diog( E Gjr (ikk)) = A,F, ΣΣ Ojr dm,)
(11)
where A, is diagonal matrix for the spatial derivatives corresponding the t-th component of the vector field, that is a polynomial of ikj, and A [A1, A2, Ag].
4.3 SPECTRAL PROJECTION LAYER
For brevity, we present our main results for computing the spectral projection operator that efficiently enforces spatial linear PDE constraints using spectral methods. We defer readers to Equs (14 (77 of Appendix A for detailed derivation of these results. In spectral space, the projection of the original vector field F into solution space: F, can be computed by:
QQB
(12)
where FFT(f), FFT(b), and
A
A
A As
A
1
1
P 1
A2
A
A2A3
:Q
Ag
(13)
Σ Σi=o= A3
A
A3
4
Ag
4

---

Under review as a conference paper at ICLR 2020
Generator
PDEL
low
high
FFT
spectral
IFFT
projection
Interpolate
Constraint
Conditioning
Conv(512)
Conv3d(3)
(SesB1k(512)
SResBlk(256) Up(2)
(128
(Re)Bik(61) Up(2)
SResBik(16)
Up(2)
Up(2)
Up(2)
Discriminator (single)
low res
Conv(16)
Conv(64)
Conv(128)
Conv(256)
LReLU
LReLU
LReLU
LReLU
IN
LReLU
Concut
high res
Conv(1)
logits
Figure 1: Schematic for the network architecture for the turbulent flow superresolution task. The network is a modified GauGAN (?) architecture for 3D fields that utilizes spatially-adattive nor- malization for conditioning with the input, and residual blocks for facilitating gradient flows. The network inputs low resolution flow field in 32x32xx32 and outputs output field of R 128x1122 128x128xx18 128 with an upscaling factor in each dimension.
EXPERIMENTS AND RESULTS
5.1 COMPUTATIONAL COMPLEXITY AND COST
We first show that although the classic Lagrangian based hard constraining method in Eqn. 7 is general and able to enforce hard linear constraints, solving it by direct inversion leads to poor computational efficiency, especially with high high-resolution 3-dimensional data outputs from CNNs.
Given Eqn. 7 for enforcing hard linear constraints using Lagrangian multipliers, we estimate the computational complexity regarding enforcing solenoidal conditions as follows. Without loss of generality, assume that the vector field on which we enforce the solenoidal constraints is dimensional of resolution N in each spatial dimension, with a total of n N nodes. The overall degrees of freedom in the system is 3n, and enforcing solenoidal constraint for each voxel results n linear constraints, hence the resulting linear system Eqn. 7 is of dimensions 4n x 4n. Though the matrix inversion is shared, hence reusable by caching, each projection involves matrix multiplication of ((3n)²) ~ (9n2)) operations. In comparison, the spectral projection method only involves element- wise operations, resulting in overall complexity of O(n) operations for enforcing constraints, and (n log(n)) for the FFT and IFFT operations. Results of an empirical analysis for computational time and memory usage is shown in Fig.
5.2 TURBULENCE SUPERRESOLUTION WITH CONDITIONAL GAN
Conditional Generative Adversarial Networks While Generative Adversarial Networks (GANs) have been effective at generating 2D (???) and 3D (?) images, the unconditional generative modeling scenario of generating entire fields from random latent vectors is hardly useful in scientific settings. more desirable model is one that is conditioned upon set of inputs from either partial observations, or low resolution simulations due to computational limitations Conditional GANs, which have been widely used in various image-to tange-to--mage translation problems (?????), and recent extension of conditional GANs, the GauGAN (?), utilizes a novel ppatiall--ddaptivn normalization layer that better preserves semantic information in the conditional input and produces improved texture outputs.
5

---

Under review as a conference paper at ICLR 2020
direct solve
direct salve
conv3d
memory usage (MB)
conv3d
compute time (5)
109
PDEL
10
PDEL
10-1
10
10-3
10a
10
10
10a
10a
10
102
10
102
(a) Compute time vs system size
(b) Peak memory usage system size
Figure 2: Comparison of computational performance for direct solve using Lagrangian multiplier method and our PDE Layer (PDEL). Computational performance for single Conv3d layer of kemell size 3 is also included for comparison purposes. Our PDEL fairs well with respect to size of the linear system, even compared with the highly optimized 3D convolution layers, allowing for its direct integration into CNN architectures. Direct solve leads to memory overflow at very small output resolutions (24%). Above computation benchmarks performed 2.4 GHz CPU chip.
In this paper we use the GauGAN architecture for the task superresolution of turbulent fluid flow fields (See Fig. T
Problem setup The main target application of this study is the super-resolution of turbulent flow fields. Fully resolving turbulence requires direct numerical simulation (DNS) that can resolve the smallest scales the flow (Komogorov scale), which is prohibitively expensive. Therefore, motivation of this study is to produce flow fields and flow statistics comparable to DNS at the cost of now-resolution proxy, whilst strictly enforcing PDE constraints. To this end, we leverage high- resolution DNS data to train a deep neural network to learn the mapping between the low-resolution flow and its high-rese high-resolution counterpart. We compare between several algorithms for the task: the conventional trilinear interpolation which is not learning based, and various deep learning methods leveraging the GauGAN architecture for conditional generative modeling. We benchmark our PDEL in-the-loop hard constraining method against unconstrained training (denoted by "none" in figures and tables) and soft constrained training (denoted by "soft"), with and without hard constraining spectral projection at test time. The goal is satisfy the imposed constraint and evaluate the accuracy of the predicted flow fields using key Nomain-specinf metrics.
Dataset Description We the Forced Isotropic Turbulence dataset from the Johns Hopkins Turbulence Database for this experiment? ?. The dataset consists of DNS at 1024 resolution performed by solving the Navier-Stoke equations using the pscudo-specteo method. The dataset consists of 5028 frames (time steps) of data, each with 3 velocity components, For this experiment, we use all simulation frames starting from the 16th time step, since the initial frames consist of underdeveloped flow. Furthermore, since 10243 resolution is practically impossible to fit into modern GPUs, we use subsampled version the data 1283 as high-resolution targets, and further subsampled 323 fields low-resolution inputs. Subsampling for the high-resolution field is performed by uniformly sampling the original flow field intervals of each dimension. For test and training splits, we take random subset of 70/ 10/20% 20% of the original data training, validation and test splits.
Evaluation metrics Since the superresolution task constraining on the fow-resolution input is mathematically underdetermined, it is not possible to recover the exact velocity field. Further, given the chaotic nature of turbulence, a single low-resolution flow field corresponds to various different realizations of high-resolution flows. Therefore, we refrain from directly comparing the norm of the difference in velocity fields. Instead, we compare the distributions of various key flow statistics, as outlined by ?, which are more informative from turbulence modeling standpoint. In Tab. [ we report the Molmoogrov-Smmine and the distributions generated by various models conditioning We also report the mean difference between the distributions, i.e., difference between the mean of the modelled and ground truth distributions in units of ground truth distribution standard deviation.
(KS) statistic between the ground truth test set distributions
the low-resolution test inputs.

---

Under review as a conference paper at ICLR 2020
none
0.08
soft
soft
PDEL
0.08
PDEL
data
data
probabilit
0.06
probability
0.06
0.04
0.04
0.02
0.02
0.00
0.00
0.66
0.68
0.70
0.72
0.74
0.0400
0.0425 0.0450
0.0500 0.0525 0.0550 value
0.0475
value
(a) Turbulent Kinetic Energy
(b) Dissipation
0.12
none
none
0.10
soft
10"
soft
PDEL
0.08
data
probabiitty
-5/3 power law
energy
0.06
10
0.04
0.02
10-3
0.00
1.35
1.40
1.45
1.50
1.55
10"
10
wavenumber k
(c) Large Eddy Turnover Time
(d) Spectrum Plot
Figure Probability Density Function plots of samples generated by various models well as ground truth distributions (highlighted in red). The quality of the distributions compared against the ground truth distributions are captured well by the KS statistic and mean difference measurements in Tab. [ (d) Shows the spectrum, where all methods trail the data distribution well in the low wavenumber regime and deviate at higher wavenumbers.
The flow statistics in Tab. is defined as below. For simplicity, we denote the different velocity components using Einstein notation, and use angle brackets <> to denote spatial averaging.
Total kinetic energy, Fist
Dissipation, 20 < oijoiz >, where dij == { constant for fluid viscosity.
Ous Jee
Dug Jz. and 0.000185 is
+
Large eddy turnover time: Tr L/u', where
o E(k) dk and u' (2/3)
Results The main quantitative results for this experiment are presented in Tab. whereas a visual- ization of the distributions regarding various key flow statistics are presented in Fig. 3 We observe from empirical evaluations that training with the hard constraining layer in-the-loop effectively imposes the solenoidal constraints (zero residue), and enforcing the hard constraint at training time achieves more accurate flow field distributions measured by various key flow statistics. We note that although this method is not the most accurate for the dissipation statistic, presumably because of discrepancies in the high wavenumber regime (where dissipation occurs), the overall mean statistics and individual statistics for the other metrics superior compared all the other methods.
7

---

Under review as a conference paper at ICLR 2020
Constraint Type
No PDE Layer
PDEL at test time
In-the-loop
Trilinear
None
Soft
Trilinear
None
Soft
PDEL
Residue(())
3.597
19.763
0.150
0.000
0.000
0.000
0.000
(kenergy
1.000
0.308
0.712
1.000
0.216
0.632
0.163
KS
dissipation
1.000
0.283
0.549
1.000
1.000
0.332
0.422
Stats
eddytime
1.000
0.388
0.599
1.000
0.229
0.487
0.276
(1)
mean
1.000
0.326
0.620
1.000
0,482
0.484
0.287
tkenergy
6.227
0.745
2.192
6.593
0.396
1.845
0.106
Mean
dissipation
16.245
0.016
1.301
16.732
2.690
0.731
0.804
Diff
eddytime
9.343
0.878
1.516
10.037
0.436
1.125
0.591
(L)
mean
10.605
0.546
1.670
11.121
1.174
1.234
0.500
Table 1: Comparison of generated distributions test set against ground truth distribution on test set. Smaller values indicate better match between distributions. Results indicate that while soft constraining can encourage the network to adhere to constraints, its residue is nonzero, implying that the imposed constraint is not strictly satisfied. Spectral based projection method can effectively eliminate residue. Training with the PDE layer in-the-loop eliminates residue and achieves greater accuracy on key statistical quantities, as compared to unconstrained and soft constrained cases, even when hard constraints enforced at test time.
CONCLUSIONS AND FUTURE WORK
Enforcing hard physical constraints to solutions generated using neural networks are essential for their application to important scientific problems. In this paper, propose novel differentiable spectral projection layer for neural networks that efficiently enforces spatial PDE constraints using spectral methods, yet is fully differentiable, allowing for its supports end-to-end training. Further, we show that its computational cost is cheaper than a regular convolution layer. We demonstrate its use in important class of physies problems incompressible turbulent flows, where the divergence-free PDE constraint required. We are able to train a 3D) Conditional Generative Adversarial Network (CGAN) for turbulent flow superresolution efficiently, whilst guaranteeing the spatial PDE constraint of zero divergence. Further, our results show that the model produces more accurate flow statistics when trained with hard constraints imposed via the proposed novel differentiable spectral projection layer, as compared to soft constrained and unconstrained counterparts.
a layer in neural networks that
Some key limitations of this work are: (i) the method is applicable in its current form only to flows with periodic boundary conditions; (ii) we only develop method for linear spatial constraints and (iii) we only consider statistically steady flows. In future will address all the above limitations to extend our work to more general sets nonlinear unsteady constraints with arbitrary boundary conditions.
8

---

Under review as a conference paper at ICLR 2020
APPENDIX
A MATHEMATICAL DERIVATION FOR SPECTRAL PROJECTION
The solution to the Lagrangian multiplier method for enforcing the solenoidal conditions involves inverting the left-hand-sii hand-side matrix in Eqn. 7 Since I, A, O are block matrices, the inverse can be represened by
A
-
A 0
1 (AAA)) (-AA -(AA7)-
(14)
(AA")
Hence the projected vector in spectral space can be computed as:
= Q
(15)
The second term in the equation above drops out since b 0 for solenoidal constraints. More specifically for spectral methods, the matrix A can be represented as three diagonal matrices for the wavenumbers in the three dimensions multiplied by the imaginary number i:
A
A
A2
A3
(16)
[Ai
A2
Aal
(17)
The only matrix inverse associated is AAT, whose value can be computed by block matrix multipli- cation:
AAT = 4 + A3 43
(18)
Given that A1, A3 are diagonal matrices (eliminating the terms regarding the [0, 0J mode), its inverse can be computed by directly inverting the diagonal terms:
(AAT) A + A3 + A3
(19)
Hence the linear projection matrix can be written as:
A
A1A2
A1A3
T- AAAAAA))-A== A + A3
A1Ag
A
AgA3
(20)
A
A
A2A3
A3
A1
AT
A2
(21)
Σラ=o
A3
Aaa
Recovering the same solution in Eqn. 13 More specifically for the divergence-fiee condition,
have:
A; = diag( ik,)
(22)
B = 0
(23)
Hence the spectral projection step can be further simplified as:
F k F
F
k
(24)
k
is easy to show that the result is divergence-free since:
ikF - - + ikF -
(25)
It is also easy to show that the projection is orthogonal to the solution space, since the dot product
between the F - F and F is zero:
k
F
F F)
F).
k) (F
k)
(26)
k.k
k
k
0
(27)

---

Under review as a conference paper at ICLR 2020
Notation
Description
Conv(n) 3D convolution block with n output layers
SResBlock
SPADE Residue Block consisting of SPADE ReLU + Conv + SPADE ReLU Conv
Up(n)
Upsampling layer with nearest neighbor interpolation with scaling factor of n
IN
3D Instance Normalization Layer
Table 2: Description of module notations in Fig.
B MODEL AND TRAINING DETAILS
We the GauGAN architecture (with schematics shown in Fig. for the conditional flow field generation task. The abbreviated names for the various modules are given Tab. [ Our model differs from the original GauGAN model in two distinct aspects. First, our architecture utilizes dimensional convolutions instead of the 2 dimensional counterparts in the original GauGAN architecture. Second, for our hard constrained case, we append the spectral projection layer the end of the architecture for enforcing hard constraints.
For training the model, we use multiresolution discriminator loss as in across discriminators. We train the model with batch size of 18 (across 6 Volta V100 GPUs) with the Adam optimizer using learning rate of 2 4. The soft constrained model uses residue penalty factor of 0.01.
C ADDITIONAL VISUALIZATION
Low-Res Imput Trilinaa
None
Soft
Trinear+PDEL None+PDEL
Soft+PDEL
POEL
Ground Truth
Figure 4: Visualizations for low resolution imputs from the test set, predictions by various models, as well as ground truth flow fields. The flow fields are colored by mapping the three velocity components to RGB channels respectively. The 2d images are slice plots in the dimension.
10


Reviews:
[{"review_id": "S1esiqF4FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a way to efficiently enforce physical constraints expressed by linear PDEs on the output of a neural network. The idea is to have, as a last layer of the network, a projection onto the constrained solution space, and to back-propagate through it. That projection layer is made efficient for high-dimensional outputs via the fast Fourier transform (FFT), exploiting a well-known numerical trick. Importantly, the proposed strategy is very general, and can indeed be used with any PDE constraint that is a linear combination of differential operators.\n\nFirst, I would like to mention to the AC that the authors apparently forgot to run bibtex. The submitted paper had no bibliography and all references are \"?\". So technically the original submission was incomplete and would probably have to be rejected, on the grounds that it was impossible to check whether the literature references are appropriate. The authors have rectified this through a comment pointing to an (anonymous) version with references. For me, this is ok.\n\nThe research direction of the paper is a hot topic: how to reconcile data-driven deep learning with analytic physical models is, arguably, one of the big research questions holding back the wide-spread use of deep networks in several natural sciences (e.g., environmental and climate science, hydrology, etc.). Work in this direction could have a significant impact, and the approach developed the paper is rather general and, as far as I can tell, correct.\n\nThe experiments use the rather challenging task of super-resolving turbulent flow, subject  to the Navier-Stokes constraints. Experiments are run on synthetic data from the JHTD simulation dataset. This is hard to avoid, since dense ground truth for flow fields is impossible to obtain, nevertheless it  would have been more convincing to also show at least qualitative results for a real flow dataset. It is also not completely satisfactory that the experiments must use even as OUTput a significantly downscaled version of the dataset with a resolution that would not be tremendously useful in practice - this implicitly acknowledges that the paper, while reducing the computational cost compared to the direct projection method, did actually not overcome the most critical bottleneck one faces when combining high-dimensional physics simulation with deep learning: namely, that it is not tractable to simply store physical fields as explicit 3D/4D voxel grids if one wants to work with them on the GPU.\n\nAnother slightly bothering choice in the experiments is to compare only distributions. While I see the point that the prediction is, by its nature, ambiguous; I still think one should look at both flow statistics and at the actual flow field difference. Ambiguity is not a very convincing justification to not even try to predict correctly - by the author's definition any super-resolution task is ambiguous, still one aims, for instance, to recover the correct image, not just a plausible one. In a sense it is the whole point of prior knowledge (which the PDEs are in a learning context) to bring the solution closer to the right answer, when the data alone cannot do the job. In practice, a prediction that is very close to the true flow field, but ever so slightly violates the physical constraints is often more useful than one that strictly satisfies the constraints, but is way off.\n\nOne comment on the presentation: I feel that the discussion of soft constraints could be more extensive and more balanced. I agree that they are less principled, they do not devalue the present work. But we know, both for variational methods and for pre-deep learning methods, that soft constraints work rather well in practice, especially with an adaptive weight that gradually tightens the constraints. So it would be in order to not just dismiss the alternative in a half-sentence, but to give it proper consideration - especially since in terms of dissipation, it actually performs better than the hard constraint in the experiments.\n\nOverall, I find the topic important and the presented work is a sensible and nicely generic step forward. On the negative side, the paper does not fully deliver on the promise to make physics constraints in deep networks usable in practice. My rating reflects my impression that the bibliography and references are probably correct - this should be checked before reaching a final decision. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xdjdvhtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use a differentiable FFT layer to enforce hard constraints for results generated by a CNN. This is demonstrated and evaluated for a 3D turbulence data set (an interesting and challenging problem), and evaluated for a single case.\n\nWhile this goal is good by itself, and the domain of applications is a very interesting one, the paper gives the impression of being preliminary, and the claims for the proposed constraints are a bit too generic, in my opinion.\n\nFirst, the FFT effectively only yields a somewhat specialized method for projection onto the set of admissible solutions, and is demonstrated only for a single constraint, i.e., to make the flow field solenoidal. The same goal can actually be reached in different ways, e.g., by inferring a vector potential as proposed by Kim et al. 2019 in the \"DeepFluids\" paper. The latter employs a curl formulation, and as such is less general, but probably faster than the FFT based method proposed here.\n\nIn addition, the paper unfortunately contains only a single example. Here, several variants (no constraint, soft constraint, and the proposed method) are evaluated in addition to simpler interpolation methods. Visually, I could not really make out differences in figure 4. The metrics in table 1 look interesting, although it, e.g., didn't get clear to me what the \"KS stats\" mean. The graphs in figure 3 also paint a somewhat varied picture. While some regions seem to be well represented, others are clearly there in the references, but missing in one of the inferred versions.\n\nI was wondering in general - what is the intuition for divergence-freeness improving the TKE, for example? It's neat to see the metrics improve, but wouldn't one expect that a projection onto divergence free flows rather removes energy from the solutions, and hence maybe yield values that are too low? \n\nI think the paper could be improved by first evaluating the method for a series of smaller two-dimensional examples, before tackling a full 3D flow. This would simplify comparisons to other methods, and help to illustrate the properties of the method. Ideally, other constraints than enforcing divergence-freeness could be demonstrated to show the generality of employing an FFT projection in the loss function. So currently, I think this paper is not quite ready for a conference such as ICLR. It would be important to demonstrate that the result shown here is not an \"outlier\", but that the improvements are a general trend obtained via the proposed method.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1ltqZlCYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis work develops a differentiable spectral projection layer to enforce spatial PDE constraints using spectral methods, to achieve the introduction of the physical constraints in the end-to-end network without damaging the intrinsic property of the network. Analysis of computational cost shows the proposed layer is cheaper than the convolutional layer. The experimental comparison demonstrates the superiority of the proposed method. In my viewpoint, the novelty of this paper is somewhat novel. \n\nThis paper focuses on designing the PDE layer to constrain the network output without additional loss. Some constraint sets for comparison are clearly performed and then authors present the proposed spectral projection layer. From the mechanism of solving, the FFT (IFFT) operator is important component in this layer. It is curious about the role and importance between FFT (IFFT) and spectral projection. If possible, the authors maybe provide some analyses of these two components to deeply recognize the proposed layer. \n\nA series of compared experiments are conducted to verify the effectiveness of the proposed PDEL. But it is a little confusing in Table 1, e.g., why the second column obtains the second better in the all mean values. I know its result has a high score of residue. It will be better if authors can clarify their causes and analysis clearly.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an FFT-based loss function to enforce physical constraints in a CNN-based PDE solver.  The proposed idea seems sensible, but the reviewers agreed that not enough attention was paid to baseline alternatives, and that a single example problem was not enough to understand the pros and cons of this method.
Decision: Reject