Title: PHYSICS-AWARE FLOW DATA COMPLETION USING NEURAL INPAINTING

Abstract: In this paper we propose a physics-aware neural network for inpainting fluid flow data. We consider that flow field data inherently follows the solution of the Navier-Stokes equations and hence our network is designed to capture physical laws. We use a DenseBlock U-Net architecture combined with a stream function formulation to inpaint missing velocity data. Our loss functions represent the relevant physical quantities velocity, velocity Jacobian, vorticity and divergence. Obstacles are treated as known priors, and each layer of the network receives the relevant information through concatenation with the previous layer's output. Our results demonstrate the network's capability for physics-aware completion tasks, and the presented ablation studies show the effectiveness of each proposed component.

1. INTRODUCTION:

Realistically modeling and predicting fluid phenomena is important to a large number of applications, which may range from optimizing objects' aerodynamic properties to creating special effects in movies. Fluids are commonly modelled by numerically solving the Navier-Stokes equations, however computer generated solutions might be discrepant from real phenomena. This happens possibly due to mismatches of the mathematical model, incorrect numerical discretization, poor discrete resolution, or errors on the estimation of parameters. Thus, approaches such as Particle Image Velocimetry or Doppler flow measurements directly measure fluid quantities in real-world settings. Often though, these measurements cannot be performed densely due to missing sensors, under-sampled domains or occluded and unreachable areas.

Thus, methods for prediction and augmentation of measured flow data are actively researched. Previous approaches are either based on a low-dimensional analysis of the flow field based on Principal Component Analysis (PCA), e.g., (Saini et al., 2016) , or are based on physically reconstructing missing areas by solving the unsteady incompressible Navier-Stokes equations, e.g., (Sciacchitano et al., 2012) . A main challenge with these traditional techniques is to predict data in large occluded or empty areas, e.g., where more than 50% of the data has to be predicted. In such challenging scenarios, already approximate predictions are useful as they could, for example, optimize strategies for sensor placement, guide procedures for human-based scanning, or improve workflows for digital prototyping.

This goal of estimating missing flow field data has many similarities with image inpainting, as it is essentially a scene completion process using partial observations. The recent success of data-driven image inpainting algorithms (Pathak et al., 2016; Iizuka et al., 2017; Liu et al., 2018; Yu et al., 2018a; b) demonstrates the capability of deep neural networks to complete large missing regions in natural images in a plausible fashion. The major difference between flow field inpainting and image inpainting lies in the fact that flow field data inherently follows the solution of the Navier-Stokes equations, and hence existing image inpainting algorithms can easily fail in physics-aware completion tasks as they never aim to capture the physical laws.

In this paper, we formulate the problem of flow data completion in large empty areas as an inpainting problem, but consider the mathematical equations that model the underlying fluid phenomena in the design of the network architecture and loss functions. By synergistically combining deep learning with fluid dynamics, we are able to inpaint data in large and irregular areas. We evaluate our proposed architectures and loss functions using thorough ablation studies both quantitatively and qualitatively. The contributions of this paper can be summarized as:

• A DenseBlocks U-Net network architecture based on a stream function formulation to inpaint velocity values;

• A set of physics-derived loss functions representing velocity, velocity gradient, divergence and vorticity;

• A simple but effective way of handling solid obstacles in the learning process.

2. RELATED WORK:

Predicting missing data for incompressible Navier-Stokes equations has been studied in the Computational Fluid Dynamics (CFD) field. Sciacchitano et al. (2012) solves the unsteady Navier-Stokes equations locally, and dimensionality reduction approaches such as proper orthogonal decomposition are applied (Venturi & Karniadakis, 2004; Higham et al., 2016; Saini et al., 2016) . Data completion of large occluded and empty areas are difficult to handle with these methods, and their running time might become prohibitively expensive depending on the application.

We adapt the idea of image inpainting, which has been intensively studied in the field of learning, to reconstruct missing flow data. (Pathak et al., 2016) used Context Encoders as one of the first attempts for filling missing image data with a deep convolutional neural network. CNN-based methods are attractive due their ability to reconstruct complex functions with only few sparse samples while being highly efficient. The follow-up work by Iizuka et al. (2017) proposes a fully convolutional network to complete rectangular missing data regions. The approach, however, still relies on Poisson image blending as a post-processing step. Yu et al. (2018b) introduces contextual attention layer to model long-range dependencies in images and a refinement network for post-processing, enabling end-to-end training. Zeng et al. (2019) extends previous work by extracting context attention maps in different layers of the encoder and skip connect attention maps to the decoder. These approaches all include adversarial losses computed from a discriminator (Goodfellow et al., 2014) in order to better reconstruct visually appealing high frequency details. However, high frequency details from adversarial losses can result in mismatches from ground truth data (Huang et al., 2017) , which can potentially predict missing data that diverge from physical laws. Liu et al. (2018) designs partial convolution operations for image inpainting, so that the prediction of the missing pixels is only conditioned on the valid pixels in the original image. The operation enables high quality inpainting results without adversarial loss.

Inpainting approaches have also been successfully used for scene completion and view path planning using data from sparse input views. Song et al. (2017) Neural networks have also recently been applied to fluid simulations. Applications include prediction of the entire dynamics (Wiewel et al., 2019) , reconstruction of simulations from a set of input parameters (Kim et al., 2019b) , interactive shape design (Umetani & Bickel, 2018) , inferring hidden physics quantities (Raissi et al., 2018) , and artistic control for visual effects (Kim et al., 2019a) . A complete overview of machine learning for fluid dynamics can be found in Brunton et al. (2020) .

3. LEARNING FLOW DATA:

Our model is inspired by Liu et al. (2018) from image inpainting. The inpainting task in image space can benefit from the capability of deep neural networks to learn semantic priors in an end-toend fashion. To reconstruct missing flow data, however, inherent laws of fluid dynamics should be considered by the neural network. In this section we detail the proposed physically-derived architecture and loss functions incorporated into our model, which results in improved fluid reconstruction results.

3.1. PHYSICS-AWARE NETWORK:

Our goal is to train a network that can fill empty regions of incompressible velocity fields. The input scheme is similar to standard image inpainting tasks. Given a 2D velocity field u in with a missing fluid data region represented by a binary mask M (0 for empty and 1 for known regions), the network predicts the velocity field with the same dimensionality as the input u out . We adapt the U-Net (Ronneberger et al., 2015) by adding Dense Blocks (Huang et al., 2017) at the bottleneck and replacing the normal convolution operations with modified partial convolutions (Liu et al., 2018) . We modified the original scaling factors of previous modified partial convolutions to the mean of the binary mask M , which leads to sharper velocity profiles in the reconstructed field. The modified partial convolution at every location is defined as

EQUATION

where W are the convolution filter weights, X are the feature values for the current convolution window, b are biases and denotes element-wise multiplication.

Additionally to the velocity prediction network (velocity branch), we implement a second network that directly predicts stream function values (stream function branch). This formulation can help to enforce incompressibility of the predicted velocity field and thus guide the network to predict velocities in a physically-aware manner, since ∇ • (∇ × Ψ) = 0. The curl operator is also fully differentiable. This formulation is therefore suitable to become an output layer for reconstructing incompressible flow data. Thus, the stream function branch takes feature maps from the last and second last layers of the velocity branch as well as the inpainting mask and passes them through 4 densely connected convolution layers with swish activation functions (Ramachandran et al., 2017) , outputting a stream function Ψ(x, y) field. The velocity field can then be reconstructed through predicted stream functions by applying the curl operator:

EQUATION

In 2D, the stream function becomes a scalar field, and the resulting velocity components are:

EQUATION

)

Both velocities from velocity branch and stream function branch are concatenated together and passed through a final prediction layer along with the inpainting mask. The final prediction layer is a single convolution layer that mixes velocity predictions from both branches. A detailed illustration of our architecture is shown in Figure 1 . The exact parameters of the network can be found in Appendix A.

Figure 1 : Our architecture for fluid data inpainting consists of a velocity branch using Dense Blocks and a stream function branch to predict incompressible velocities. Our method supports different occlusion and obstacle masks to mimic different real-world settings.

3.2. LOSS FUNCTIONS:

For incompressible flow data it is important to define a new set of supervised loss functions that model physical properties and constraints. For detailing the employed loss functions, we use û for predicted and u for ground-truth velocities.The L 1 velocity loss efficiently reconstructs lowfrequency information and is defined as

EQUATION

Inspired by Kim et al. (2019b) , we additionally minimize the difference of the velocity field Jacobian between ground-truth and predicted velocity fields. With a sufficiently smooth flow-field dataset, high-frequency features of the CNN are potentially on the null space of the L 1 distance minimization (Kim et al., 2019b) . Thus, matching the Jacobians helps the network to recover highfrequency spectral information, while it also regularizes the reconstructed velocity to match groundtruth derivatives. The velocity Jacobian J(u) is defined in 2D as

EQUATION

and the corresponding loss function is simply given as the L 1 of vectorized Jacobian between predicted and ground-truth velocities:

EQUATION

Additionally, we compute a loss function that matches the vorticity of predicted and ground-truth velocities. The vorticity field describes the local spinning motion of the velocity field. Similarly to the Jacobian loss, our vorticity loss acts as a directional high-frequency filter that helps to match shearing derivatives of the original data, enhancing the capability of the model to properly match the underlying fluid dynamics. The vorticity loss is given by

EQUATION

Incompressible flows should have zero divergence, however, numerical simulations often produce results that are not strictly divergence-free due to discretization errors. As we combine predictions from velocity and stream function branches, we are able to match the divergence on the original and predicted fields by minimizing

EQUATION

Similarly to previous works, each loss function is applied both on the known and unknown regions with potentially different weights. The weight selection is illustrated in Appendix A. We exclude the perceptual L perceptual , style L style and total variation L tv losses from the image inpainting model of Liu et al. (2018) . Although these losses successfully improve the visual quality of predicted images, they are not suited for completing flow-field data, since they match pre-learned filters from image classification architectures.

3.3. ENCODING OBSTACLES:

The interaction between fluid and solid obstacles is crucial for fluid dynamics applications. To incorporate solid obstacle information as prior knowledge to the network, we concatenate a binary mask O indicating whether a solid obstacle occupies a cell as an extra input channel. In order to properly propagate the obstacle information to all network layers, the obstacle occupancy information is concatenated to previous layers' output. To accomplish that, we downsample the obstacle map O to match a specific layer dimensions, similarly to the empty region mask M .

4. EXPERIMENTS AND RESULTS:

4.1 DATA GENERATION Due to the lack of publicly available flow data sets captured from real-world experiments, we trained our model on synthetic data. We generated fluid velocity fields with a numerical flow solver for Illustrations of types of masks can be seen in the leftmost column in Figure 3 .

To evaluate the proposed architecture and loss functions, we applied our model on two different datasets, both computed on grid resolutions of 128 × 96. The first wind tunnel dataset implements a scene with turbulent flow around obstacles. We define inflow velocities at bottom and top regions of the domain, while the remaining two sides (left and right) are set as free-flow (open) boundary conditions. The inflow speed is set to random values, and obstacles (6 spheres, 6 rectangles) are randomly positioned, yielding a total of 32,000 simulation frames. The second simple plume dataset implements a smoke rising from a source at the bottom of a fully enclosed box, which represents a common setup in graphics applications. In this dataset, no solid obstacles are included and the source position and size are the parameters that vary across different simulations. In total, 21,000 simulation frames are present in the simple plume dataset.

4.2. ABLATION STUDIES:

To investigate the effects of various components introduced in previous sections, we performed a series of ablation studies, with results shown in Table 1 . We train and evaluate the architectures of different configurations on the wind tunnel dataset, selectively deactivating the DenseBlocks, as well as the stream function and velocity branches. We also compare the effect of the proposed loss functions by progressively adding them to different architecture configurations. Besides evaluating Mean Absolute Error (MAE) over the whole dataset, we also evaluate the model capabilities when varying masking occlusion levels in Figure 2 . Our ablation study shows that the quality of inpainting results can be significantly improved by Dense Blocks (entries (a-c) and (e-g)) when compared with architectures with no DenseBlocks (entries (d) and (h-k)). We also tested adding loss functions progressively to architectures with (entries (a) and (e-g)) and without DenseBlocks (entries (h-k)) to evaluate if those effects are cumulative.

Combining the proposed physically-inspired losses yields better results also in this case, which seems a strong indication that they contribute to better results even in distinct architectures.

The model that employs only a stream function branch (b) performs similarly to our full model (a). Note that since model (b) reconstructs velocities by taking the curl of stream function predictions, incompressibility is guaranteed and thus no divergence loss is used. However, the synthetic velocity field data has discretization errors and it is not truly divergence free. Therefore, the approach with a single stream function branch cannot capture the divergent modes present on the original data, yielding higher MAE than the combined branches approach. We hypothesize that a pure stream function architecture would fare better with velocity fields obtained from real-world experiments or highly accurate flow solvers, where the divergence is closer to zero.

We notice that the model with only the velocity branch (c) has a lower MAE over the whole dataset. However, Figure 2 shows that this model has a low capability of inpainting samples with large empty regions. This indicates that the stream function branch can better guide the network to predict results obeying physical laws, while the velocity branch can help predicting results based on information from a known region. The functionality of the two branches can be more clearly shown in Figure 3 , where the models (a,b,c) are visually compared. To plot the velocity fields, we use a HSV colormap that encodes flow direction and relative velocity in the hue and saturation, respectively. The model in (c) only uses the velocity branch information, and it reconstructs lower frequency content better as the information is easier to infer from surrounding known regions. Using the stream function branch only, model (b) better predicts high frequency information, but mistakenly reconstructs lower frequency regions. Finally, model (a) combines the advantages of models (b) and (c) by using both branches, and predictions are more precise in both higher and lower frequency ranges.

4.3. PREDICTION RESULTS:

Figure 4 shows the results of applying different mask profiles for the simple plume dataset. The top left example shows a scan path mask (left), comparing the reconstructed (middle) and ground truth velocity profiles (right). Even with a sparse scan mask with a occlusion of 0.74, our reconstruction is able recreate velocity profiles that are close to the ground truth, even in regions that are far away from the mask. The top and bottom right images of Figure 4 show similar examples, while the bottom left image depicts a random noise mask to mimic the effects of noisy sampling. In Figure 5 we show corresponding results for the wind tunnel dataset. In this challenging scenario, we use large region masks in combination with obstacles immersed in the fluid. Our results show that our method is able to accurately capture flows around obstacles, even though very sparse velocity field samples were used. We additionally compare the original image inpainting model (Liu et al., 2018) (Table 1 (k)) with our best architecture (Table 1 (a) ). The results are shown in Figure 6 , indicating that our approach can reconstruct more flow structures, especially near immersed obstacles.

5. CONCLUSIONS:

We have presented a physics-aware architecture for inpainting missing velocity data. We have shown that our method is especially powerful for data completion of large areas with more than 50% missing data entries. Using the proposed stream function branch in combination with DenseBlocks has proven to be the key element to reduce Mean Absolute Error (MAE), and augmenting the architectures with our physically-derived loss functions has further improved accuracy. The proposed method outperforms existing image inpainting models when applied to flow data, demonstrating the effectiveness of including knowledge about fluid dynamics in the network design.

We have evaluated our method on 2-dimensional data. Extending the method to 3-dimensional flows that exhibit more turbulent structures is an essential next step. The major challenge with 3D data, however, is the large memory consumption, which is especially critical for high-resolution simulations. Approaches based on progressive patch-based inpainting (Isola et al., 2016) or viewby-view inpainting (Han et al., 2019) could be relevant for reducing the memory footprint. Further tests are also needed with simulation datasets that capture different real-world scenarios, as well as data from real-world measurements. 

A APPENDIX:

A.1 NETWORK PARAMETERS

The U-Net architecture is similar to the one described in (Liu et al., 2018) with the sole difference that stride 2 partial convolutions are only done in every 2nd layer in order to fit the dataset resolution of 128 by 96.

For the DenseBlocks U-Net, we replace layers 7 to 10 with a DenseBlock described in Table 3 . Note that a DenseBlock as described in Huang et al. (2017) has a skip connection between each layer. Outputs of all previous layers inside the block are concatenated and represent the input of the current layer. The encoder part is also modified to achieve a constant compression ratio of 1.5 over each subsequent layer, see Table 2 . The number of features is computed as follows: 64 • 4 l 2

1.5 l-1 with l being the layer number from 1 to 7. The decoder then connects from layer 17 to 22 and is built symmetrically to the encoder. Note that this architecture, although deeper compared to the U-Net (22 vs 16), has only about 28% as many trainable parameters.

The stream function prediction network is a DenseBlock that consists of 5 convolution layers, and is described in Table 4 . Activation is done with the swish function (Ramachandran et al., 2017) Training for all models is done using Adam optimizer with a learning rate of 0.01 and a batch size of 16. During training, all masks are used equally and the occlusion is set uniformly at random to a value between 25% and 99%.

:



Figures and Tables:

Figure 2: Mean Absolute Error of different model configurations at each mask occlusion level.

Figure 3: Inpainted flow fields with different masked inputs and obstacle configurations. From left to right: masked velocity input , output from configuration (a), output velocity from configuration (b), output from configuration (c), ground truth. All the masks used on the examples above have an approximate occlusion rate of 52%.

Figure 4: Examples of predictions on Simple Plume dataset. From left to right of each example: masked velocity input, output from configuration (a), ground truth velocity.

Figure 5: Examples of predictions on Wind Tunnel Dataset. From left to right of each example: masked velocity input, output from configuration (a), ground truth velocity.

Figure 6: Comparison of model configuration (a) with image inpainting model Liu et al. (2018). Image inpainting model is trained on wind tunnel dataset without L style , L perceptual and L tv . From left to right: masked velocity input, output from configuration (a), output from image inpainting model, ground truth velocity.

instead of the conventional ReLU because it turned out to facilitate training and improve accuracy.



Table content: <table><tr><td>uses an end-to-end network SSCNet for scene</td></tr><tr><td>completion and Guo &amp; Tong (2018) a view-volume CNN that extracts geometric features from 2D</td></tr><tr><td>depth images. Zhang &amp; Funkhouser (2018) presents an end-to-end architecture for depth inpainting,</td></tr><tr><td>and Han et al. (2019) uses multi-view depth completion to predict point cloud representations. A 3D</td></tr><tr><td>recurrent network has</td></tr></table>

Ablation study configurations.incompressible fluids (Mantaflow(Thuerey & Pfaff, 2018)). Each training data sample consists of a 2-dimensional vector field containing the velocity components u and v and the empty regions and obstacles masks M and O. During training, different types of empty region masks are generated on the fly with empty to filled area region ratios that vary randomly from 25 to 99 percent. We generate three different types of masks: uniform random noise masks mimic possible sampling noise from real-world velocity measurements; scan path masks simulate paths of a velocity probing; and large region masks model large occluded areas that are not reachable by probes or measurement devices.

Table content: <table><tr><td/><td>Ψ branch u branch DenseBlocks L jacobian Lvort L div MAE</td></tr><tr><td>(a)</td><td>0.268</td></tr><tr><td>(b)</td><td>0.270</td></tr><tr><td>(c)</td><td>0.261</td></tr><tr><td>(d)</td><td>0.353</td></tr><tr><td>(e)</td><td>0.280</td></tr><tr><td>(f)</td><td>0.276</td></tr><tr><td>(g)</td><td>0.280</td></tr><tr><td>(h)</td><td>0.307</td></tr><tr><td>(i)</td><td>0.314</td></tr><tr><td>(j)</td><td>0.332</td></tr><tr><td>(k)</td><td>0.332</td></tr></table>

Peer review (iclr2020):
In this paper the authors adopt prior work in image inpainting to the problem of 2d fluid velocity field inpainting by extending the network architecture and using additional loss functions. Specifically, the U-net network is extended with a DenseBlock in the middle, and a separate branch of the network is added which predicts the stream function (a different representation of the velocity field which guarantees incompressibility). The additional losses are L1 for various derivatives of the flow field (Jacobian, divergence, vorticity). Experiments presented in the paper show that these new elements improve the flow field error compared to a baseline model originally developed for image inpainting. The suggested application for this model is filling gaps in experimental measurements that are missing or impossible to obtain, and where such a model could be computationally cheaper than an actual fluid solver. The paper discusses prior work at the appropriate level of depth, and specifies hyperparameters that were used for training. As is, I do not believe the paper fully meets the bar for novelty and potential impact for acceptance at ICLR (but it could perhaps be a good fit for a more specialized venue). I am specifically not convinced about the practical applicability of the results. The novelty of the approach also seems quite limited, and the findings do not seem particularly surprising or insightful (i.e. that extending the vanilla U-net architecture and adding losses that directly bias the network towards physically meaningful solutions is better than the baseline). Since all training and testing data was generated from simulation, it is unclear how well the networks would cope with real world measurement noise. Furthermore, the authors note that the flow field from the simulation is sometimes not divergence-free. It is surprising that this could be a problem to a level where it could impact evaluations. If it indeed is, then perhaps it would make sense to either simulate the flows with a denser grid (inpainting could still be done on sparsified results). I found it surprising that the authors chose instead to use Eq. 8 and *force* the network to learn a flow field from the solver which is known to not be strictly physically valid. It is also unclear to me that the masking schemes used in the experiments are relevant to actual measurements -- for instance, 2nd row of Fig. 3 or 1st row of Fig. 4 seem completely artificial. I appreciated the ablation study variants (but see also comments below for at least one more configuration that I believe should be discussed in addition to the existing ones). It would however be more informative (and important for potential practical applications) to include some sort of breakdown by mask type and flow field configuration/structures (steady, unsteady, wakes, jets, vortices, etc). The included images show the model does not capture some finer details of the velocity field (e.g. vortex structures in the last row of Fig. 3 and right column of Fig. 4; wakes behind small obstacles in Fig. 5-6, long, narrow, and fast jets in Fig. 6), but it is unclear what impact the proposed extensions (DenseBlock, losses, and stream function branch) have on these structures. Similarly, interpolating missing data points with a fairly dense input where no points are more than a few pixels apart seems like a much easier problem than filling large empty spaces, and it would make sense to do separate analyses for these cases. Questions & suggestions for improvements: * Has the impact of different weight combinations in A.2. been investigated? How was the ratio of 6:1 for empty:valid determined? * The text says that in the stream function pathway, the features are passed *through 4 densely connected convolution layers*. I was confused the first time I read that sentence, and only later I realized that this refers to a DenseNet-like pattern of connectivity. A reference to Huang et al. here or some other clarification would help. * In the figures, please consider also showing the difference between the predictions and ground truth to make it easier to see which features of the flow field are predicted accurately, and which are not. * Is there an L1 loss applied as well directly to the output of the stream prediction branch? * What are the Reynolds numbers used in the simulations? How far from the original Re does the network generalize? * When computing MAE, what are the units? What are the velocity magnitudes? Consider also reporting mean relative error. * Impact of the various L1 deriv. losses seems negligible when the stream function is used, but is more visible when only velocity is being directly predicted. Please comment on why that might be. * Why does (f) (Jacobian only) work better than (e) and (g)? * It is not clear that that the effect of the additional L1 losses is truly cumulative. Please consider testing just (u, DenseBlock, L_div). * It would be an interesting extension to include a non-ML baseline, which could be compared against the current results in terms of flow field quality and computational cost. * Can the network predictions deviate from the known (non-masked) values in the input? If so, please consider including a breakdown of the evaluation that shows how much the error varies between the *valid* and *empty* areas.