Title: PHYSICS-AWARE FLOW DATA COMPLETION USING NEURAL INPAINTING

Abstract: In this paper we propose a physics-aware neural network for inpainting fluid flow data. We consider that flow field data inherently follows the solution of the Navier-Stokes equations and hence our network is designed to capture physical laws. We use a DenseBlock U-Net architecture combined with a stream function formulation to inpaint missing velocity data. Our loss functions represent the relevant physical quantities velocity, velocity Jacobian, vorticity and divergence. Obstacles are treated as known priors, and each layer of the network receives the relevant information through concatenation with the previous layer's output. Our results demonstrate the network's capability for physics-aware completion tasks, and the presented ablation studies show the effectiveness of each proposed component.

1. INTRODUCTION:

Realistically modeling and predicting fluid phenomena is important to a large number of applications, which may range from optimizing objects' aerodynamic properties to creating special effects in movies. Fluids are commonly modelled by numerically solving the Navier-Stokes equations, however computer generated solutions might be discrepant from real phenomena. This happens possibly due to mismatches of the mathematical model, incorrect numerical discretization, poor discrete resolution, or errors on the estimation of parameters. Thus, approaches such as Particle Image Velocimetry or Doppler flow measurements directly measure fluid quantities in real-world settings. Often though, these measurements cannot be performed densely due to missing sensors, under-sampled domains or occluded and unreachable areas.

Thus, methods for prediction and augmentation of measured flow data are actively researched. Previous approaches are either based on a low-dimensional analysis of the flow field based on Principal Component Analysis (PCA), e.g., (Saini et al., 2016) , or are based on physically reconstructing missing areas by solving the unsteady incompressible Navier-Stokes equations, e.g., (Sciacchitano et al., 2012) . A main challenge with these traditional techniques is to predict data in large occluded or empty areas, e.g., where more than 50% of the data has to be predicted. In such challenging scenarios, already approximate predictions are useful as they could, for example, optimize strategies for sensor placement, guide procedures for human-based scanning, or improve workflows for digital prototyping.

This goal of estimating missing flow field data has many similarities with image inpainting, as it is essentially a scene completion process using partial observations. The recent success of data-driven image inpainting algorithms (Pathak et al., 2016; Iizuka et al., 2017; Liu et al., 2018; Yu et al., 2018a; b) demonstrates the capability of deep neural networks to complete large missing regions in natural images in a plausible fashion. The major difference between flow field inpainting and image inpainting lies in the fact that flow field data inherently follows the solution of the Navier-Stokes equations, and hence existing image inpainting algorithms can easily fail in physics-aware completion tasks as they never aim to capture the physical laws.

In this paper, we formulate the problem of flow data completion in large empty areas as an inpainting problem, but consider the mathematical equations that model the underlying fluid phenomena in the design of the network architecture and loss functions. By synergistically combining deep learning with fluid dynamics, we are able to inpaint data in large and irregular areas. We evaluate our proposed architectures and loss functions using thorough ablation studies both quantitatively and qualitatively. The contributions of this paper can be summarized as:

• A DenseBlocks U-Net network architecture based on a stream function formulation to inpaint velocity values;

• A set of physics-derived loss functions representing velocity, velocity gradient, divergence and vorticity;

• A simple but effective way of handling solid obstacles in the learning process.

2. RELATED WORK:

Predicting missing data for incompressible Navier-Stokes equations has been studied in the Computational Fluid Dynamics (CFD) field. Sciacchitano et al. (2012) solves the unsteady Navier-Stokes equations locally, and dimensionality reduction approaches such as proper orthogonal decomposition are applied (Venturi & Karniadakis, 2004; Higham et al., 2016; Saini et al., 2016) . Data completion of large occluded and empty areas are difficult to handle with these methods, and their running time might become prohibitively expensive depending on the application.

We adapt the idea of image inpainting, which has been intensively studied in the field of learning, to reconstruct missing flow data. (Pathak et al., 2016) used Context Encoders as one of the first attempts for filling missing image data with a deep convolutional neural network. CNN-based methods are attractive due their ability to reconstruct complex functions with only few sparse samples while being highly efficient. The follow-up work by Iizuka et al. (2017) proposes a fully convolutional network to complete rectangular missing data regions. The approach, however, still relies on Poisson image blending as a post-processing step. Yu et al. (2018b) introduces contextual attention layer to model long-range dependencies in images and a refinement network for post-processing, enabling end-to-end training. Zeng et al. (2019) extends previous work by extracting context attention maps in different layers of the encoder and skip connect attention maps to the decoder. These approaches all include adversarial losses computed from a discriminator (Goodfellow et al., 2014) in order to better reconstruct visually appealing high frequency details. However, high frequency details from adversarial losses can result in mismatches from ground truth data (Huang et al., 2017) , which can potentially predict missing data that diverge from physical laws. Liu et al. (2018) designs partial convolution operations for image inpainting, so that the prediction of the missing pixels is only conditioned on the valid pixels in the original image. The operation enables high quality inpainting results without adversarial loss.

Inpainting approaches have also been successfully used for scene completion and view path planning using data from sparse input views. Song et al. (2017) Neural networks have also recently been applied to fluid simulations. Applications include prediction of the entire dynamics (Wiewel et al., 2019) , reconstruction of simulations from a set of input parameters (Kim et al., 2019b) , interactive shape design (Umetani & Bickel, 2018) , inferring hidden physics quantities (Raissi et al., 2018) , and artistic control for visual effects (Kim et al., 2019a) . A complete overview of machine learning for fluid dynamics can be found in Brunton et al. (2020) .

3. LEARNING FLOW DATA:

Our model is inspired by Liu et al. (2018) from image inpainting. The inpainting task in image space can benefit from the capability of deep neural networks to learn semantic priors in an end-toend fashion. To reconstruct missing flow data, however, inherent laws of fluid dynamics should be considered by the neural network. In this section we detail the proposed physically-derived architecture and loss functions incorporated into our model, which results in improved fluid reconstruction results.

3.1. PHYSICS-AWARE NETWORK:

Our goal is to train a network that can fill empty regions of incompressible velocity fields. The input scheme is similar to standard image inpainting tasks. Given a 2D velocity field u in with a missing fluid data region represented by a binary mask M (0 for empty and 1 for known regions), the network predicts the velocity field with the same dimensionality as the input u out . We adapt the U-Net (Ronneberger et al., 2015) by adding Dense Blocks (Huang et al., 2017) at the bottleneck and replacing the normal convolution operations with modified partial convolutions (Liu et al., 2018) . We modified the original scaling factors of previous modified partial convolutions to the mean of the binary mask M , which leads to sharper velocity profiles in the reconstructed field. The modified partial convolution at every location is defined as

EQUATION

where W are the convolution filter weights, X are the feature values for the current convolution window, b are biases and denotes element-wise multiplication.

Additionally to the velocity prediction network (velocity branch), we implement a second network that directly predicts stream function values (stream function branch). This formulation can help to enforce incompressibility of the predicted velocity field and thus guide the network to predict velocities in a physically-aware manner, since ∇ • (∇ × Ψ) = 0. The curl operator is also fully differentiable. This formulation is therefore suitable to become an output layer for reconstructing incompressible flow data. Thus, the stream function branch takes feature maps from the last and second last layers of the velocity branch as well as the inpainting mask and passes them through 4 densely connected convolution layers with swish activation functions (Ramachandran et al., 2017) , outputting a stream function Ψ(x, y) field. The velocity field can then be reconstructed through predicted stream functions by applying the curl operator:

EQUATION

In 2D, the stream function becomes a scalar field, and the resulting velocity components are:

EQUATION

)

Both velocities from velocity branch and stream function branch are concatenated together and passed through a final prediction layer along with the inpainting mask. The final prediction layer is a single convolution layer that mixes velocity predictions from both branches. A detailed illustration of our architecture is shown in Figure 1 . The exact parameters of the network can be found in Appendix A.

Figure 1 : Our architecture for fluid data inpainting consists of a velocity branch using Dense Blocks and a stream function branch to predict incompressible velocities. Our method supports different occlusion and obstacle masks to mimic different real-world settings.

3.2. LOSS FUNCTIONS:

For incompressible flow data it is important to define a new set of supervised loss functions that model physical properties and constraints. For detailing the employed loss functions, we use û for predicted and u for ground-truth velocities.The L 1 velocity loss efficiently reconstructs lowfrequency information and is defined as

EQUATION

Inspired by Kim et al. (2019b) , we additionally minimize the difference of the velocity field Jacobian between ground-truth and predicted velocity fields. With a sufficiently smooth flow-field dataset, high-frequency features of the CNN are potentially on the null space of the L 1 distance minimization (Kim et al., 2019b) . Thus, matching the Jacobians helps the network to recover highfrequency spectral information, while it also regularizes the reconstructed velocity to match groundtruth derivatives. The velocity Jacobian J(u) is defined in 2D as

EQUATION

and the corresponding loss function is simply given as the L 1 of vectorized Jacobian between predicted and ground-truth velocities:

EQUATION

Additionally, we compute a loss function that matches the vorticity of predicted and ground-truth velocities. The vorticity field describes the local spinning motion of the velocity field. Similarly to the Jacobian loss, our vorticity loss acts as a directional high-frequency filter that helps to match shearing derivatives of the original data, enhancing the capability of the model to properly match the underlying fluid dynamics. The vorticity loss is given by

EQUATION

Incompressible flows should have zero divergence, however, numerical simulations often produce results that are not strictly divergence-free due to discretization errors. As we combine predictions from velocity and stream function branches, we are able to match the divergence on the original and predicted fields by minimizing

EQUATION

Similarly to previous works, each loss function is applied both on the known and unknown regions with potentially different weights. The weight selection is illustrated in Appendix A. We exclude the perceptual L perceptual , style L style and total variation L tv losses from the image inpainting model of Liu et al. (2018) . Although these losses successfully improve the visual quality of predicted images, they are not suited for completing flow-field data, since they match pre-learned filters from image classification architectures.

3.3. ENCODING OBSTACLES:

The interaction between fluid and solid obstacles is crucial for fluid dynamics applications. To incorporate solid obstacle information as prior knowledge to the network, we concatenate a binary mask O indicating whether a solid obstacle occupies a cell as an extra input channel. In order to properly propagate the obstacle information to all network layers, the obstacle occupancy information is concatenated to previous layers' output. To accomplish that, we downsample the obstacle map O to match a specific layer dimensions, similarly to the empty region mask M .

4. EXPERIMENTS AND RESULTS:

4.1 DATA GENERATION Due to the lack of publicly available flow data sets captured from real-world experiments, we trained our model on synthetic data. We generated fluid velocity fields with a numerical flow solver for Illustrations of types of masks can be seen in the leftmost column in Figure 3 .

To evaluate the proposed architecture and loss functions, we applied our model on two different datasets, both computed on grid resolutions of 128 × 96. The first wind tunnel dataset implements a scene with turbulent flow around obstacles. We define inflow velocities at bottom and top regions of the domain, while the remaining two sides (left and right) are set as free-flow (open) boundary conditions. The inflow speed is set to random values, and obstacles (6 spheres, 6 rectangles) are randomly positioned, yielding a total of 32,000 simulation frames. The second simple plume dataset implements a smoke rising from a source at the bottom of a fully enclosed box, which represents a common setup in graphics applications. In this dataset, no solid obstacles are included and the source position and size are the parameters that vary across different simulations. In total, 21,000 simulation frames are present in the simple plume dataset.

4.2. ABLATION STUDIES:

To investigate the effects of various components introduced in previous sections, we performed a series of ablation studies, with results shown in Table 1 . We train and evaluate the architectures of different configurations on the wind tunnel dataset, selectively deactivating the DenseBlocks, as well as the stream function and velocity branches. We also compare the effect of the proposed loss functions by progressively adding them to different architecture configurations. Besides evaluating Mean Absolute Error (MAE) over the whole dataset, we also evaluate the model capabilities when varying masking occlusion levels in Figure 2 . Our ablation study shows that the quality of inpainting results can be significantly improved by Dense Blocks (entries (a-c) and (e-g)) when compared with architectures with no DenseBlocks (entries (d) and (h-k)). We also tested adding loss functions progressively to architectures with (entries (a) and (e-g)) and without DenseBlocks (entries (h-k)) to evaluate if those effects are cumulative.

Combining the proposed physically-inspired losses yields better results also in this case, which seems a strong indication that they contribute to better results even in distinct architectures.

The model that employs only a stream function branch (b) performs similarly to our full model (a). Note that since model (b) reconstructs velocities by taking the curl of stream function predictions, incompressibility is guaranteed and thus no divergence loss is used. However, the synthetic velocity field data has discretization errors and it is not truly divergence free. Therefore, the approach with a single stream function branch cannot capture the divergent modes present on the original data, yielding higher MAE than the combined branches approach. We hypothesize that a pure stream function architecture would fare better with velocity fields obtained from real-world experiments or highly accurate flow solvers, where the divergence is closer to zero.

We notice that the model with only the velocity branch (c) has a lower MAE over the whole dataset. However, Figure 2 shows that this model has a low capability of inpainting samples with large empty regions. This indicates that the stream function branch can better guide the network to predict results obeying physical laws, while the velocity branch can help predicting results based on information from a known region. The functionality of the two branches can be more clearly shown in Figure 3 , where the models (a,b,c) are visually compared. To plot the velocity fields, we use a HSV colormap that encodes flow direction and relative velocity in the hue and saturation, respectively. The model in (c) only uses the velocity branch information, and it reconstructs lower frequency content better as the information is easier to infer from surrounding known regions. Using the stream function branch only, model (b) better predicts high frequency information, but mistakenly reconstructs lower frequency regions. Finally, model (a) combines the advantages of models (b) and (c) by using both branches, and predictions are more precise in both higher and lower frequency ranges.

4.3. PREDICTION RESULTS:

Figure 4 shows the results of applying different mask profiles for the simple plume dataset. The top left example shows a scan path mask (left), comparing the reconstructed (middle) and ground truth velocity profiles (right). Even with a sparse scan mask with a occlusion of 0.74, our reconstruction is able recreate velocity profiles that are close to the ground truth, even in regions that are far away from the mask. The top and bottom right images of Figure 4 show similar examples, while the bottom left image depicts a random noise mask to mimic the effects of noisy sampling. In Figure 5 we show corresponding results for the wind tunnel dataset. In this challenging scenario, we use large region masks in combination with obstacles immersed in the fluid. Our results show that our method is able to accurately capture flows around obstacles, even though very sparse velocity field samples were used. We additionally compare the original image inpainting model (Liu et al., 2018) (Table 1 (k)) with our best architecture (Table 1 (a) ). The results are shown in Figure 6 , indicating that our approach can reconstruct more flow structures, especially near immersed obstacles.

5. CONCLUSIONS:

We have presented a physics-aware architecture for inpainting missing velocity data. We have shown that our method is especially powerful for data completion of large areas with more than 50% missing data entries. Using the proposed stream function branch in combination with DenseBlocks has proven to be the key element to reduce Mean Absolute Error (MAE), and augmenting the architectures with our physically-derived loss functions has further improved accuracy. The proposed method outperforms existing image inpainting models when applied to flow data, demonstrating the effectiveness of including knowledge about fluid dynamics in the network design.

We have evaluated our method on 2-dimensional data. Extending the method to 3-dimensional flows that exhibit more turbulent structures is an essential next step. The major challenge with 3D data, however, is the large memory consumption, which is especially critical for high-resolution simulations. Approaches based on progressive patch-based inpainting (Isola et al., 2016) or viewby-view inpainting (Han et al., 2019) could be relevant for reducing the memory footprint. Further tests are also needed with simulation datasets that capture different real-world scenarios, as well as data from real-world measurements. 

A APPENDIX:

A.1 NETWORK PARAMETERS

The U-Net architecture is similar to the one described in (Liu et al., 2018) with the sole difference that stride 2 partial convolutions are only done in every 2nd layer in order to fit the dataset resolution of 128 by 96.

For the DenseBlocks U-Net, we replace layers 7 to 10 with a DenseBlock described in Table 3 . Note that a DenseBlock as described in Huang et al. (2017) has a skip connection between each layer. Outputs of all previous layers inside the block are concatenated and represent the input of the current layer. The encoder part is also modified to achieve a constant compression ratio of 1.5 over each subsequent layer, see Table 2 . The number of features is computed as follows: 64 • 4 l 2

1.5 l-1 with l being the layer number from 1 to 7. The decoder then connects from layer 17 to 22 and is built symmetrically to the encoder. Note that this architecture, although deeper compared to the U-Net (22 vs 16), has only about 28% as many trainable parameters.

The stream function prediction network is a DenseBlock that consists of 5 convolution layers, and is described in Table 4 . Activation is done with the swish function (Ramachandran et al., 2017) Training for all models is done using Adam optimizer with a learning rate of 0.01 and a batch size of 16. During training, all masks are used equally and the occlusion is set uniformly at random to a value between 25% and 99%.

:



Figures and Tables:

Figure 2: Mean Absolute Error of different model configurations at each mask occlusion level.

Figure 3: Inpainted flow fields with different masked inputs and obstacle configurations. From left to right: masked velocity input , output from configuration (a), output velocity from configuration (b), output from configuration (c), ground truth. All the masks used on the examples above have an approximate occlusion rate of 52%.

Figure 4: Examples of predictions on Simple Plume dataset. From left to right of each example: masked velocity input, output from configuration (a), ground truth velocity.

Figure 5: Examples of predictions on Wind Tunnel Dataset. From left to right of each example: masked velocity input, output from configuration (a), ground truth velocity.

Figure 6: Comparison of model configuration (a) with image inpainting model Liu et al. (2018). Image inpainting model is trained on wind tunnel dataset without L style , L perceptual and L tv . From left to right: masked velocity input, output from configuration (a), output from image inpainting model, ground truth velocity.

instead of the conventional ReLU because it turned out to facilitate training and improve accuracy.



Table content: <table><tr><td>uses an end-to-end network SSCNet for scene</td></tr><tr><td>completion and Guo &amp; Tong (2018) a view-volume CNN that extracts geometric features from 2D</td></tr><tr><td>depth images. Zhang &amp; Funkhouser (2018) presents an end-to-end architecture for depth inpainting,</td></tr><tr><td>and Han et al. (2019) uses multi-view depth completion to predict point cloud representations. A 3D</td></tr><tr><td>recurrent network has</td></tr></table>

Ablation study configurations.incompressible fluids (Mantaflow(Thuerey & Pfaff, 2018)). Each training data sample consists of a 2-dimensional vector field containing the velocity components u and v and the empty regions and obstacles masks M and O. During training, different types of empty region masks are generated on the fly with empty to filled area region ratios that vary randomly from 25 to 99 percent. We generate three different types of masks: uniform random noise masks mimic possible sampling noise from real-world velocity measurements; scan path masks simulate paths of a velocity probing; and large region masks model large occluded areas that are not reachable by probes or measurement devices.

Table content: <table><tr><td/><td>Ψ branch u branch DenseBlocks L jacobian Lvort L div MAE</td></tr><tr><td>(a)</td><td>0.268</td></tr><tr><td>(b)</td><td>0.270</td></tr><tr><td>(c)</td><td>0.261</td></tr><tr><td>(d)</td><td>0.353</td></tr><tr><td>(e)</td><td>0.280</td></tr><tr><td>(f)</td><td>0.276</td></tr><tr><td>(g)</td><td>0.280</td></tr><tr><td>(h)</td><td>0.307</td></tr><tr><td>(i)</td><td>0.314</td></tr><tr><td>(j)</td><td>0.332</td></tr><tr><td>(k)</td><td>0.332</td></tr></table>

Peer review (iclr2020):
Summary: This paper proposes a physics-aware variant of a U-Net network for completing missing flow field data. Most notably, the loss functions are motivated by fluid dynamics, which forces the network to remain more consistent with the governing laws. Decision: I found the paper and the idea very exciting. Injecting domain knowledge by forcing the output (or differentiable transformations thereof) to be consistent with the physics is a quite relevant and appealing idea, for which this paper constitutes a nice proof-of-concept. Framing the problem of completing missing flow data as an inpainting task is also original. However, the evaluation of the method does not study important aspects regarding its generalization. The description of the experimental protocol is also missing important information. Further arguments: - I found the discussion in Section 4.2 around method (b) not convincing. I do not understand why the network should be penalized if it does not reproduce the mistakes of the original simulator/solver. Since `div u` should be 0, why not simply penalizing ||div u|| instead of having the loss of Eqn. 8? Isn*t approach (b) the approach which is most *physics-aware* and correct from a physics point of view? - The experiments do not highlight whether the network actually just learn the training distribution or generalize by *understanding* the physics of the problem. A compelling experiment would have been to evaluate whether a network trained on a prior family of obstacles transfer properly to a different family (e.g., training on 6 spherical and 6 rectangle obstacles, but testing on fewer/more obstacles with other shapes). - Similarly, could the network generalize to larger/smaller inputs? Once trained, can it work on grids smaller/larger than 128x96? If not, what do you recommend to do in practice? - The description of the experimental protocol does not specify whether the method was evaluated on independent test data. More worrisome, section 4.2 even states that the MAE for Figure 2 is computed *over the whole dataset*. - The method is not compared against any domain specific baseline. - While quite exciting, I am not confident the contribution is original enough from an ML point of view for ICLR, although it is certainly novel for fluid dynamics. Additional feedback: - Some results reported in Table 1 are quite close of each other. It would have made the experiments much stronger if uncertainties were also reported and discussed. - Fig 3: I would have liked seeing the error maps of the methods. This would have been quite helpful to better tell them apart. - Given my comments above, I am confident an 8th page could be put to good use.