Physics-Aware Flow Data Completion Using Neural Inpainting

1 INTRODUCTION . Realistically modeling and predicting fluid phenomena is important to a large number of applications , which may range from optimizing objects ’ aerodynamic properties to creating special effects in movies . Fluids are commonly modelled by numerically solving the Navier-Stokes equations , however computer generated solutions might be discrepant from real phenomena . This happens possibly due to mismatches of the mathematical model , incorrect numerical discretization , poor discrete resolution , or errors on the estimation of parameters . Thus , approaches such as Particle Image Velocimetry or Doppler flow measurements directly measure fluid quantities in real-world settings . Often though , these measurements can not be performed densely due to missing sensors , under-sampled domains or occluded and unreachable areas . Thus , methods for prediction and augmentation of measured flow data are actively researched . Previous approaches are either based on a low-dimensional analysis of the flow field based on Principal Component Analysis ( PCA ) , e.g. , ( Saini et al. , 2016 ) , or are based on physically reconstructing missing areas by solving the unsteady incompressible Navier-Stokes equations , e.g. , ( Sciacchitano et al. , 2012 ) . A main challenge with these traditional techniques is to predict data in large occluded or empty areas , e.g. , where more than 50 % of the data has to be predicted . In such challenging scenarios , already approximate predictions are useful as they could , for example , optimize strategies for sensor placement , guide procedures for human-based scanning , or improve workflows for digital prototyping . This goal of estimating missing flow field data has many similarities with image inpainting , as it is essentially a scene completion process using partial observations . The recent success of data-driven image inpainting algorithms ( Pathak et al. , 2016 ; Iizuka et al. , 2017 ; Liu et al. , 2018 ; Yu et al. , 2018a ; b ) demonstrates the capability of deep neural networks to complete large missing regions in natural images in a plausible fashion . The major difference between flow field inpainting and image inpainting lies in the fact that flow field data inherently follows the solution of the NavierStokes equations , and hence existing image inpainting algorithms can easily fail in physics-aware completion tasks as they never aim to capture the physical laws . In this paper , we formulate the problem of flow data completion in large empty areas as an inpainting problem , but consider the mathematical equations that model the underlying fluid phenomena in the design of the network architecture and loss functions . By synergistically combining deep learning with fluid dynamics , we are able to inpaint data in large and irregular areas . We evaluate our proposed architectures and loss functions using thorough ablation studies both quantitatively and qualitatively . The contributions of this paper can be summarized as : • A DenseBlocks U-Net network architecture based on a stream function formulation to inpaint velocity values ; • A set of physics-derived loss functions representing velocity , velocity gradient , divergence and vorticity ; • A simple but effective way of handling solid obstacles in the learning process . 2 RELATED WORK . Predicting missing data for incompressible Navier-Stokes equations has been studied in the Computational Fluid Dynamics ( CFD ) field . Sciacchitano et al . ( 2012 ) solves the unsteady Navier-Stokes equations locally , and dimensionality reduction approaches such as proper orthogonal decomposition are applied ( Venturi & Karniadakis , 2004 ; Higham et al. , 2016 ; Saini et al. , 2016 ) . Data completion of large occluded and empty areas are difficult to handle with these methods , and their running time might become prohibitively expensive depending on the application . We adapt the idea of image inpainting , which has been intensively studied in the field of learning , to reconstruct missing flow data . ( Pathak et al. , 2016 ) used Context Encoders as one of the first attempts for filling missing image data with a deep convolutional neural network . CNN-based methods are attractive due their ability to reconstruct complex functions with only few sparse samples while being highly efficient . The follow-up work by Iizuka et al . ( 2017 ) proposes a fully convolutional network to complete rectangular missing data regions . The approach , however , still relies on Poisson image blending as a post-processing step . Yu et al . ( 2018b ) introduces contextual attention layer to model long-range dependencies in images and a refinement network for post-processing , enabling end-to-end training . Zeng et al . ( 2019 ) extends previous work by extracting context attention maps in different layers of the encoder and skip connect attention maps to the decoder . These approaches all include adversarial losses computed from a discriminator ( Goodfellow et al. , 2014 ) in order to better reconstruct visually appealing high frequency details . However , high frequency details from adversarial losses can result in mismatches from ground truth data ( Huang et al. , 2017 ) , which can potentially predict missing data that diverge from physical laws . Liu et al . ( 2018 ) designs partial convolution operations for image inpainting , so that the prediction of the missing pixels is only conditioned on the valid pixels in the original image . The operation enables high quality inpainting results without adversarial loss . Inpainting approaches have also been successfully used for scene completion and view path planning using data from sparse input views . Song et al . ( 2017 ) uses an end-to-end network SSCNet for scene completion and Guo & Tong ( 2018 ) a view-volume CNN that extracts geometric features from 2D depth images . Zhang & Funkhouser ( 2018 ) presents an end-to-end architecture for depth inpainting , and Han et al . ( 2019 ) uses multi-view depth completion to predict point cloud representations . A 3D recurrent network has been used to integrate information from only a few input views ( Choy et al. , 2016 ) , and Xu et al . ( 2016 ) uses spatial and temporal structure of sequential observations to predict a view sequence . Neural networks have also recently been applied to fluid simulations . Applications include prediction of the entire dynamics ( Wiewel et al. , 2019 ) , reconstruction of simulations from a set of input parameters ( Kim et al. , 2019b ) , interactive shape design ( Umetani & Bickel , 2018 ) , inferring hidden physics quantities ( Raissi et al. , 2018 ) , and artistic control for visual effects ( Kim et al. , 2019a ) . A complete overview of machine learning for fluid dynamics can be found in Brunton et al . ( 2020 ) . 3 LEARNING FLOW DATA . Our model is inspired by Liu et al . ( 2018 ) from image inpainting . The inpainting task in image space can benefit from the capability of deep neural networks to learn semantic priors in an end-toend fashion . To reconstruct missing flow data , however , inherent laws of fluid dynamics should be considered by the neural network . In this section we detail the proposed physically-derived architecture and loss functions incorporated into our model , which results in improved fluid reconstruction results . 3.1 PHYSICS-AWARE NETWORK . Our goal is to train a network that can fill empty regions of incompressible velocity fields . The input scheme is similar to standard image inpainting tasks . Given a 2D velocity field uin with a missing fluid data region represented by a binary mask M ( 0 for empty and 1 for known regions ) , the network predicts the velocity field with the same dimensionality as the input uout . We adapt the U-Net ( Ronneberger et al. , 2015 ) by adding Dense Blocks ( Huang et al. , 2017 ) at the bottleneck and replacing the normal convolution operations with modified partial convolutions ( Liu et al. , 2018 ) . We modified the original scaling factors of previous modified partial convolutions to the mean of the binary mask M , which leads to sharper velocity profiles in the reconstructed field . The modified partial convolution at every location is defined as x′ = { WT ( X M ) M + b if ∑ ( M ) > 0 , 0 otherwise , ( 1 ) where W are the convolution filter weights , X are the feature values for the current convolution window , b are biases and denotes element-wise multiplication . Additionally to the velocity prediction network ( velocity branch ) , we implement a second network that directly predicts stream function values ( stream function branch ) . This formulation can help to enforce incompressibility of the predicted velocity field and thus guide the network to predict velocities in a physically-aware manner , since ∇ · ( ∇ × Ψ ) = 0 . The curl operator is also fully differentiable . This formulation is therefore suitable to become an output layer for reconstructing incompressible flow data . Thus , the stream function branch takes feature maps from the last and second last layers of the velocity branch as well as the inpainting mask and passes them through 4 densely connected convolution layers with swish activation functions ( Ramachandran et al. , 2017 ) , outputting a stream function Ψ ( x , y ) field . The velocity field can then be reconstructed through predicted stream functions by applying the curl operator : u = ∇×Ψ . ( 2 ) In 2D , the stream function becomes a scalar field , and the resulting velocity components are : ux = ∂ψ ∂y , uy = − ∂ψ ∂x . ( 3 ) Both velocities from velocity branch and stream function branch are concatenated together and passed through a final prediction layer along with the inpainting mask . The final prediction layer is a single convolution layer that mixes velocity predictions from both branches . A detailed illustration of our architecture is shown in Figure 1 . The exact parameters of the network can be found in Appendix A . 3.2 LOSS FUNCTIONS . For incompressible flow data it is important to define a new set of supervised loss functions that model physical properties and constraints . For detailing the employed loss functions , we use û for predicted and u for ground-truth velocities.The L1 velocity loss efficiently reconstructs lowfrequency information and is defined as Lvel = ||û− u||1 . ( 4 ) Inspired by Kim et al . ( 2019b ) , we additionally minimize the difference of the velocity field Jacobian between ground-truth and predicted velocity fields . With a sufficiently smooth flow-field dataset , high-frequency features of the CNN are potentially on the null space of the L1 distance minimization ( Kim et al. , 2019b ) . Thus , matching the Jacobians helps the network to recover highfrequency spectral information , while it also regularizes the reconstructed velocity to match groundtruth derivatives . The velocity Jacobian J ( u ) is defined in 2D as J ( u ) = ( ∂ux ∂x ∂ux ∂y ∂uy ∂x ∂uy ∂y ) , ( 5 ) and the corresponding loss function is simply given as the L1 of vectorized Jacobian between predicted and ground-truth velocities : Ljacobian = ||J ( û ) − J ( u ) ||1 . ( 6 ) Additionally , we compute a loss function that matches the vorticity of predicted and ground-truth velocities . The vorticity field describes the local spinning motion of the velocity field . Similarly to the Jacobian loss , our vorticity loss acts as a directional high-frequency filter that helps to match shearing derivatives of the original data , enhancing the capability of the model to properly match the underlying fluid dynamics . The vorticity loss is given by Lvort = ||∇ × û−∇× u||1 . ( 7 ) Incompressible flows should have zero divergence , however , numerical simulations often produce results that are not strictly divergence-free due to discretization errors . As we combine predictions from velocity and stream function branches , we are able to match the divergence on the original and predicted fields by minimizing Ldiv = ||∇ · û−∇ · u||1 . ( 8 ) Similarly to previous works , each loss function is applied both on the known and unknown regions with potentially different weights . The weight selection is illustrated in Appendix A . We exclude the perceptual Lperceptual , style Lstyle and total variation Ltv losses from the image inpainting model of Liu et al . ( 2018 ) . Although these losses successfully improve the visual quality of predicted images , they are not suited for completing flow-field data , since they match pre-learned filters from image classification architectures .

Summary: In this paper the authors adopt prior work in image inpainting to the problem of 2d fluid velocity field inpainting by extending the network architecture and using additional loss functions. Specifically, the U-net network is extended with a DenseBlock in the middle, and a separate branch of the network is added which predicts the stream function (a different representation of the velocity field which guarantees incompressibility). The additional losses are L1 for various derivatives of the flow field (Jacobian, divergence, vorticity). Experiments presented in the paper show that these new elements improve the flow field error compared to a baseline model originally developed for image inpainting. The suggested application for this model is filling gaps in experimental measurements that are missing or impossible to obtain, and where such a model could be computationally cheaper than an actual fluid solver.