Physics-Aware Flow Data Completion Using Neural Inpainting
Authors: Sebastien Foucher
Jingwei Tang
Vinicius da Costa de Azevedo
Byungsoo Kim
Markus Gross
Barbara Solenthaler
Venue:  (2020)
Keywords: neural inpainting
fluid dynamics
flow data completion
physics-aware network

Abstract: In this paper we propose a physics-aware neural network for inpainting fluid flow data. We consider that flow field data inherently follows the solution of the Navier-Stokes equations and hence our network is designed to capture physical laws. We use a DenseBlock U-Net architecture combined with a stream function formulation to inpaint missing velocity data. Our loss functions represent the relevant physical quantities velocity, velocity Jacobian, vorticity and divergence. Obstacles are treated as known priors, and each layer of the network receives the relevant information through concatenation with the previous layer's output. Our results demonstrate the network's capability for physics-aware completion tasks, and the presented ablation studies show the effectiveness of each proposed component.

Under review as a conference paper at ICLR 2020
PHYSICS-AWARE FLOW DATA COMPLETION USING NEURAL INPAINTING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper we propose physics-aware neural network for inpainting fluid flow data. We consider that flow field data inherently follows the solution of the Navier- Stokes equations and hence our network is designed to capture physical laws. We use a DenseBlock U-Net architecture combined with stream function formu- lation inpaint missing velocity data. Our loss functions represent the relevant physical quantities velocity, velocity Jacobian, vorticity and divergence. Obstacles are treated as known priors, and each layer of the network receives the relevant information through concatenation with the previous layer's output. Our results demonstrate the network's 's capability for physics-aware completion tasks, and the presented ablation studies show the effectiveness of each proposed component.
INTRODUCTION
Realistically modeling and predicting fluid phenomena is important to a large number of applica- tions, which may range from optimizing objects' aerodynamic properties to creating special effects in movies. Fluids are commonly modelled by numerically solving the Navier-Stokes equations, however computer generated solutions might be discrepant from real phenomena. This happens possibly due to mismatches of the mathematical model, incorrect numerical discretization, poor discrete resolution, or errors on the estimation of parameters. Thus, approaches such as Particle Image Velocimetry or Doppler flow measurements directly measure fluid quantities in real-world settings. Often though, these measurements cannot be performed densely due to missing sensors, under-sampled domains occluded and unreachable areas.
Thus, methods for prediction and augmentation of measured flow data are actively researched. Pre- vious approaches are either based on a Component Analysis (PCA), e.g., (Saini et al., 2016), or are based on physically reconstructing missing areas by solving the unsteady incompressible Navier-Stokes equations, e.g., (Sciacchitano et al. 2012). A main challenge with these traditional techniques is to predict data in large occluded empty areas, e.g., where more than 50% of the data has to be predicted. In such challenging scenarios, already approximate predictions are useful as they could, for example, optimize strategies for sensor placement, guide procedures for human-based scanning, or improve workflows for digital prototyping.
analysis of the flow field based on Principal
This goal of estimating missing flow field data has many similarities with image inpainting, as is essentially scene completion process using partial observations. The recent success of data-driven image inpainting algorithms (Pathak et al. 2016, Tizuka al., 2017 Liu et al., 2018 Yu et al.) 2018a b) demonstrates the capability of deep neural networks to complete large missing regions in natural images plausible fashion. The major difference between flow field inpainting and image inpainting lies in the fact that flow field data inherently follows the solution of the Navier- Stokes equations, and hence existing image inpainting algorithms can easily fail in physics-awan completion tasks they never aim capture the physical laws.
In this paper, we formulate the problem of flow data completion in large empty areas as an inpaint- ing problem, but consider the mathematical equations that model the underlying fluid phenomena in the design of the network architecture and loss functions. By synergistically combining deep learn- ing with fluid dynamics,
able to inpaint data in large and irregular areas. We evaluate proposed architectures and loss functions using thorough ablation studies both quantitatively and qualitatively. The contributions of this paper can be summarized as:

---

Under review as a conference paper at ICLR 2020
A DenseBlocks U-Net network architecture based stream function formulation to in- paint velocity values;
A set of physics-derived loss functions representing velocity, velocity gradient, divergence and vorticity;
A simple but effective way of handling solid obstacles in the learning process.
RELATED WORK
Predicting missing data for incompressible Navier-Stokes equations has been studied in the Compu- tational Fluid Dynamics (CFD) field. SSciacchitano et al. (2012) solves the unsteady Navier-Stokes equations locally, and dimensionality reduction approaches such as proper orthogonal decompo- sition are applied (Venturi & Karniadakis 2004, Higham et al.] 2016;; (aani et al. 2016). Data completion of large occluded and empty areas are difficult handle with these methods, and their running time might become prohibitively expensive depending on the application.
We adapt the idea of image inpainting, which has been intensively studied in the field of learning, to reconstruct missing flow data. (Pathak et al. (0116) used Context Encoders as one of the first attempts for filling missing image data with deep convolutional neural network. CNN-based methods are attractive due their ability to reconstruct complex functions with only few sparse samples while being highly efficient. The follow-up work by Tizuka et al. (2017) proposes a fully convolutional network to complete rectangular missing data regions. The approach, however, still relies on Poisson image blending as a post post-procsssing step. Yu et al. (2018b) introduces contextual attention layer to model long-range dependencies in images and refinement network for pratpprocssing, enabling end-to-end training. ZZeng et al. (2019) extends previous work by extracting context attention maps in different layers of the encoder and skip connect attention maps to the decoder. These approaches all include adversarial losses computed from discriminator (Goodfellow et al.) 2014) in order to better reconstruct visually appealing high frequency details. However, high frequency details from adversarial losses can result in mismatches from ground truth data (Huang et al. 2017). which can potentially predict missing data that diverge from physical laws. [iu al. (2018) designs partial convolution operations for image inpainting, so that the prediction of the missing pixels is only conditioned on the valid pixels in the original image. The operation enables high quality inpainting results without adversarial loss.
Inpainting approaches have also been successfully used for scene completion and view path planning using data from sparse input views. Song et al. (2017) uses end-to-end network SSCNet for scene completion and Guo & Tong (2018) a view-volume CNN that extracts geometric features from 2D depth images. Zhang & Funkhouser (2018) presents end-to-end architecture for depth inpainting, Han et al. (2019) uses milti-view depth completion to predict point cloud representations. recurrent network has been used integrate information from only few input views (Choy al., 2016), and Xu et al. (2016) uses spatial and temporal structure of sequential observations to predict view sequence.
3D
Neural networks have also recently been applied to fluid simulations. Applications include predic- tion of the entire dynamics (Wiewel al. 2019), reconstruction of simulations from a of input parameters (Kim et al. (011b)), interactive shape design (Umetani & Bickel, (018). inferring hidden physics quantities (Raissi al.) (2018), and artistic control for visual effects (Kim et al.) 2019a). A complete overview of machine learning for fluid dynamics can be found in Brunton et al. (2020).
3 LEARNING FLOW DATA
Our model is inspired by Liu et al. (2018) from image inpainting. The inpainting task in image space can benefit from the capability of deep neural networks to learn semantic priors in end-to- end fashion. To reconstruct missing flow data, however, inherent laws of fluid dynamics should be considered by the neural network. In this section we detail the proposed thysically-derive architec- ture and loss functions incorporated into our model, which results in improved fluid reconstruction results.
2

---

Under review as a conference paper at ICLR 2020
3.1 PHYSICS-AWARE NETWORK
Our goal is to train a network that can fill empty regions of incompressible velocity fields. The input scheme is similar standard image inpainting tasks. Given 2D velocity field uin with missing fluid data region represented by a binary mask M (O for empty and 1 for known regions), the network predicts the velocity field with the same dimensionality U-Net (Ronneberger et al. 2 by adding Dense Blocks (Huang et al. (017)) at the bottleneck and replacing the normal convolution operations with modified partial convolutions (Liu et al. 2018). We modified the original scaling factors of previous modified partial convolutions to the mean of the binary mask M, which leads to sharper velocity profiles in the reconstructed field. The modified partial convolution at every location is defined as
the input Wout- We adapt the
O +
Ef
otherwise,
where W are the convolution filter weights, X are the feature values for the current convolution window, are biases and @ denotes element-wise multiplication.
Additionally to the velocity prediction network (velocity branch), implement second network that directly predicts stream function values (stream function branch). This formulation can help to enforce incompressibility of the predicted velocity field and thus guide the network to predict velocities in a physically-aware manner, since V (V x V) = O. The curl operator is also fully differentiable. This formulation is therefore suitable become output layer for reconstructing incompressible flow data. Thus, the stream function branch takes feature maps from the last and second last layers of the velocity branch as well as the inpainting mask and passes them through densely connected convolution layers with swish activation functions (Ramachandran et al. 2017), outputting stream function (x, y) field. The velocity field can then be reconstructed through predicted stream functions by applying the curl operator:
=
V.
In 2D, the stream function becomes scalar field, and the resulting velocity components are:
Orb Oy uyy
Og
U1
Fr
Both velocities from velocity branch and stream function branch are concatenated together and passed through final prediction layer along with the inpainting mask. The final prediction layer single convolution layer that mixes velocity predictions from both branches. A detailed illustration of our architecture is shown in Figure The exact parameters of the network can be found in Appendix A
Velocity Branch
Dense Decoder
Obstacle Mask
Encoder
Block
Concatenation
Conv2D
Input valocity
Stream Function Branch
Dense
Predicted Velocity
Block
Inpainting Mask
Figure 1: Our architecture for fluid data inpainting consists of velocity branch using Dense Blocks and a stream function branch to predict incompressible velocities. Our method supports different occlusion and obstacle masks mimic different real-world settings.
3

---

Under review as a conference paper at ICLR 2020
3.2 LOSs FUNCTIONS
For incompressible flow data it is important to define a new set of supervised loss functions that model physical properties and constraints. For detailing the employed loss functions, we use for predicted and u for ground-truth velocities. The L velocity loss efficiently reconstructs low- frequency information and defined as
Lue! u11--
(4)
Inspired by Kim al. (2019b), additionally minimize the difference of the velocity field Ja- cobian between ground-truth and predicted velocity fields. With a sufficiently smooth flow-field dataset, high-frequency features of the CNN potentially on the null space of the L distance minimization (Kim et al. 2019b). Thus, matching the Jacobians helps the network to recover high- frequency spectral information, while it also regularizes the reconstructed velocity match ground- truth derivatives. The velocity Jacobian J(u) is defined in 21) as
811
(u)
duy
Ox
Juy
(5)
and the corresponding loss function is simply given as the L of vectorized Jacobian between pre- dicted and ground-trae truth velocities:
jocobian = (J)
(6)
Additionally, we compute a loss function that matches the vorticity of predicted and ground-truth velocities. The vorticity field describes the local spinning motion of the velocity field. Similarly to the Jacobian loss, our vorticity loss acts as directional high-frequency filter that helps to match shearing derivatives of the original data, enhancing the capability of the model to properly match the underlying fluid dynamics. The vorticity loss is given by
Leort
x ú
Incompressible flows should have zero divergence, however, numerical simulations often produce results that not strictly divergence-free due to discretization errors. As combine predictions from velocity and stream function branches, we are able to match the divergence on the original and predicted fields by minimizing
Latio 1 ú u
Similarly previous works, each loss function is applied both on the known and unknown regions with potentially different weights. The weight selection is illustrated in Appendix[A] We exclude the perceptual I Epperceptuat- style Isttye and total variation Tet losses from the image inpainting model of Liu et al. (2018). Although these losses successfully improve the visual quality of predicted images, they are not suited for completing flow-field data, since they match pre-learned filters from image classification architectures.
3.3 ENCODING OBSTACLES
The interaction between fluid and solid obstacles is crucial for fluid dynamics applications. To in- corporate solid obstacle information prior knowledge the network, concatenate binary mask indicating whether a solid obstacle occupies a cell as an extra input channel. In order to properly propagate the obstacle information to all network layers, the obstacle occupancy informa- tion concatenated to previous layers' output. To accomplish that, we downsample the obstacle map O match specific layer dimensions, similarly to the empty region mask M.
4 EXPERIMENTS AND RESULTS
4.1 DATA GENERATION
Due to the lack of publicly available flow data sets captured from real-world experiments, we trained model synthetic data. We generated fluid velocity fields with numerical flow solver for
4

---

Under review as a conference paper at ICLR 2020
V branch
branch
DenseBlocks
Ljacobian
Lyort
Ldii
MAE
(8)
0.268
(b)
0.270
(c)
v
0.261
/
V
0.353
(e)
0.280
V
0.276
0.280
0.307
0.314
0.332
0.332
Table 1: Ablation study configurations.
incompressible fluids (Mantaflow (Thuerey & Pfafi, 2018)). Each training data sample consists of 2-dimensional vector field containing the velocity components and and the empty regions and obstacles masks and O. During training, different types of empty region masks are generated on the fly with empty to filled area region ratios that vary randomly from 25 to 99 percent. We generate three different types of masks: uniform random noise masks mimic possible sampling noise from real-world velocity measurements; scan path masks simulate paths of velocity probing; and large region masks model large occluded areas that are not reachable by probes or measurement devices. Illustrations of types of masks can be seen in the leftmost column in Figure
To evaluate the proposed architecture and loss functions, we applied our model on two different datasets, both computed grid resolutions of 128 96. The first wind tunnel dataset implements a scene with turbulent flow around obstacles. We define inflow velocities at bottom and top regions of the domain, while the remaining two sides (left and right) are set as free-flow (open) boundary conditions. The inflow speed is set to random values, and obstacles (6 spheres, rectangles) are randomly positioned, yielding total of 32,000 simulation frames. The second simple plume dataset implements smoke rising from source at the bottom of fully enclosed box, which represents a common setup graphics applications. In this dataset, no solid obstacles are included and the source position and size are the parameters that vary across different simulations. In total, 21 ,000 simulation frames are present the simple plume dataset.
4.2 ABLATION STUDIES
To investigate the effects of various components introduced in previous sections, performed series of ablation studies, with results shown in Table [ We train and evaluate the architectures of different configurations on the wind tunnel dataset, selectively deactivating the DenseBlocks, as well as the stream function and velocity branches. We also compare the effect of the proposed loss functions by progressively adding them different architecture configurations. Besides evaluating Mean Absolute Error (MAE) over the whole dataset, we also evaluate the model capabilities when varying masking occlusion levels in Figure
0.7
1N1
0.5
0.5
0.5
2.4
0.4
0.2
0.2
0.2
0.1
0.1
0.1
0.0
6.0
Mask
Figure 2: Mean Absolute Error of different model configurations at each mask occlusion level.
Our ablation study shows that the quality of inpainting results can be significantly improved by Dense Blocks (entries (a-c) and (c-g)) when compared with architectures with no DenseBlocks (en- tries (d) and (h-k)). We also tested adding loss functions progressively to architectures with (entries
5

---

Under review as a conference paper at ICLR 2020
(a) and (e-g)) and without DenseBlocks (entries (h-k)) to evaluate if those effects cumulative. Combining the proposed pyssicall-iisppiree losses yields better results also in this case, which seems strong indication that they contribute to better results even in distinct architectures.
The model that employs only a stream function branch (b) performs similarly to our full model (a). Note that since model (b) reconstructs velocities by taking the curl of stream function predictions, incompressibility is guaranteed and thus no divergence loss is used. However, the synthetic velocity field data has discretization errors and it is not truly divergence free. Therefore, the approach with a single stream function branch cannot capture the divergent modes present on the original data, yielding higher MAE than the combined branches approach. We hypothesize that a pure stream function architecture would fare better with velocity fields obtained from real- weal-world experiments or highly accurate flow solvers, where the divergence closer to zero.
We notice that the model with only the velocity branch (c) has lower MAE over the whole dataset. However, Figure[Jshows that this model has low capability of inpainting samples with large empty regions. This indicates that the stream function branch can better guide the network to predict results obeying physical laws, while the velocity branch can help predicting results based on information from a known region. The functionality of the two branches can be more clearly shown in Figure where the models (a,b.c) are visually compared. To plot the velocity fields, we use a HSV colormap that encodes flow direction and relative velocity in the hue and saturation. respectively. The model in (c) only uses the velocity branch information, and it reconstructs lower frequency content better as the information is easier to infer from surrounding known regions. Using the stream function branch only, model (b) better predicts high frequency information, but mistakenly reconstructs lower frequency regions. Finally, model (a) combines the advantages of models (b) and (c) by using both branches, and predictions are more precise in both higher and lower frequency ranges.
4.3 PREDICTION RESULTS
Figure 4 shows the results of applying different mask profiles for the simple plume dataset. The top left example shows a scan path mask (left), comparing the reconstructed (middle) and ground truth velocity profiles (right). Even with a sparse scan mask with a occlusion of 0.74, our reconstruction is able recreate velocity profiles that are close to the ground truth, even in regions that are far away from the mask. The top and bottom right images of Figure 4 show similar examples, while the bottom left image depicts a random noise mask to mimic the effects of noisy sampling. In Figure [ we show corresponding results for the wind tunnel dataset. In this challenging scenario, we use large region masks in combination with obstacles immersed in the fluid. Our results show that our method is able to accurately capture flows around obstacles, even though very sparse velocity field samples were used. We additionally compare the original image inpainting model (Liu et al. 2018) (Table | (k)) with our best architecture (Table (a)). The results are shown in Figure 6 indicating that our approach can reconstruct more flow structures, especially near immersed obstacles.
CONCLUSIONS
We have presented a physics-aware architecture for inpainting missing velocity data. We have shown that our method is especially powerful for data completion of large areas with more than 50% miss- ing data entries. Using the proposed stream function branch in combination with DenseBlocks has proven to be the key element to reduce Mean Absolute Error (MAE), and augmenting the archi- tectures with our physically-derived loss functions has further improved accuracy. The proposed method outperforms existing image inpainting models when applied to flow data, demonstrating the effectiveness of including knowledge about fluid dynamics in the network design.
We have evaluated our method on 2-dimensional data. Extending the method to 3-dimensional flows that exhibit more turbulent structures is an essential next step. The major challenge with 3D data, however, is the large memory consumption, which is especially critical for high-resolution simulations. Approaches based on progressive patch-based inpainting [Isola et al. 2016) or view- by-view inpainting (Han et al. 2019) could be relevant for reducing the memory footprint. Further tests are also needed with simulation datasets that capture different real-world scenarios, as well as data from real-world measurements.

---

Under review as a conference paper at ICLR 2020
Figure 3: Inpainted flow fields with different masked inputs and obstacle configurations. From left to right: masked velocity input, output from configuration (a), output velocity from configuration (b), output from configuration (c), ground truth. All the masks used on the examples above have an approximate occlusion rate of 52%.
Figure 4: Examples of predictions on Simple Plume dataset. From left to right of each example: masked velocity input, output from configuration (a), ground truth velocity.
REFERENCES
Steven L. Brunton, Bernd R. Noack, and Petros Koumoutsakos. Machine learning for fluid mechan- ics. Annual Review of Fluid Mechanics, 52, 2020.
7

---

Under review as a conference paper at ICLR 2020
Figure 5: Examples of predictions on Wind Tunnel Dataset. From left to right of each example: masked velocity input, output from configuration (a), ground truth velocity.
Figure 6: Comparison of model configuration (a) with image inpainting model Liu et al. 2018 Image inpainting model is trained wind tunnel dataset without Lsstles Exprecppuual and Tax- From left to right: masked velocity input, output from configuration (a), output from image inpainting model, ground truth velocity.
Christopher Bongsoo Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d- r2n2: unified approach for single and multi-view 3d object reconstruction. Computer Vision ECCV 2016, pp. 628 628-644, 2016. doi: 10.
38.
8

---

Under review as a conference paper at ICLR 2020
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farkes Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-266620 2680, 2014.
Yuxiao Guo and Xin Tong. View-volums network for semantic scene completion from a single depth image. pp. 726-732, 07 2018. doi: 10.24963/ijcai 201.////0....0//001..
Xiaoguang Han, Zhaoxuan Zhang, Dong Du, Mingdai Yang, Jingming Yu, Pan Pan, Xin Yang. Ligang Liu, Zixiang Xiong, and Shuguang Cui. Deep reinforcement learning of volume volume-guided guided progressive view inpainting for 3d point scene completion from a single depth image, 03 2019.
J. E. Higham, W. Brevis, and C. J. Keylock. A rapid non-iterative proper orthogonal decomposition based outlier detection and correction for PIV data. Measurement Science and Technology, 27 (12), 2016. ISSN 13616501. doi: 10.10880095--033////222223
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. Proceedings 30th IEEE Conference Computer Vision and Pattern Recognition, CVPR 20/7, 201 20017-Jamarry22211222006 2017. doi: 10. 00.1109/CVPR.2017.244
Satoshi lizuka, Edgar Simo Serra, and Hiroshi Ishikawa. Globally and locally consistent image pletion. ACM Transactions on Graphics, 36(4), 2017. ISSN 15577368. doi: 10.1 10.1/45/0772959 3073659.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexci A. Efros. Image-to-imadd translation with conditional adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 5967-5976, 2016.
Byungsoo Kim, Vinicius Azevedo. Markus Gross, and Barbara Solenthaler. Transport-based neural style transfer for smoke simulations. ACM Trans. Graph., 2019a.
Byungsoo Kim, Vinicius C. Azevedo, Nils Thuerey, Theodore Kim, Markus Gross, and Barbara Solenthaler. Deep Fluids: Generative Network for Parameterized Fluid Simulations. Computer Graphics Forum, 38:59-70, 70, 2019b.
Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting Chun Wang, Andrew Tao, and Bryan Catanzaro. Im- age Inpainting for Irregular Holes Using Partial Convolutions. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11215 LNCS:89-100 2018. ISSN 16113349, doi:
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context En- coders: Feature Learning by Inpainting. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 226--eee::256-2222 2544, 2016. ISSN 10636919.
doi: 10.1109/CVPR.2116.21
Maziar Raissi, Alireza Yazdani, and George E. Karniadakis. Hidden fluid mechanics: A navier- stokes informed deep learning framework for assimilating flow visualization data. CoRR, 2018.
Prajit Ramachandran, Barret Zoph, and Quoe V. Le. Searching for Activation Functions. pp. 1-13, 2017. URLpbtp:: wrg. abs C5941
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed-
ical image segmentation. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9351:234 241, 2015. ISSN
16113349. doi: 10. (001007/978-33319-245444 28.
Pankaj Saini, Christoph Amdd, and Adam Steinberg. Development and evaluation of sappy-pod as a data reconstruction technique for noisy piv measurements in gas turbine combustors. Experiments in Fluids, 57, 07 2016. doi: 10. 12100770000448-011--28---
Andrea Sciacchitano, Fulvio Scarano, and Bernhard Wieneke. Multi-fram pyramid correlation for time-resolved PIV. Experiments in Fluids, 53(4):1087-1105, 2012. ISSN 07234864. doi: 10. 10.10077400348--12-3344

---

Under review as a conference paper at ICLR 2020
Shuran Song, Fisher Yu, Andy Zeng, Angel Chang, Manolis Savva, and Thomas Funkhouser. Se- mantic scene completion from a single depth image. pp. 190-198, 07 2017. doi: 10.1109/CVPR. 2017.28.
Nils Thuerey and Tobias Pfaff. MantaFlow, 2018. http://munntatoww.cmm.
Nobuyuki Umetani and Bemd Bickel. Learning diceeniimesio flow for interactive aerodynamic design. ACM Trans. Graph., 37(4):89:1-89:10 2018. ISSN 0730-0301.
Daniele Venturi and George Em Karniadakis. Gappy data and reconstruction procedures for flow past cylinder. Journal of Fluid Mechanics, 519:315-333 2004. ISSN 00221 120.. doi: 10.1017/ $0022112004001338.
Steffen Wiewel, Moritz Becher, and Nils Thuerey. Latent-space Physics: Towards Learning the Temporal Evolution of Fluid Flow. Computer Graphics Forum, 38, 2019.
Kai Xu, Yifei Shi, Lintao Zheng, Junyu Zhang, Min Liu, Hui Huang, Hao Su, Daniel Cohen-Or, and Baoquan Chen. 3d attention-driven depth acquisition for object identification. ACM Trans. Graph., 35(6):238:1-2356 14, 2016.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. Generative Image Inpainting with Contextual Attention. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 5505-5514, 2018a. ISSN 10636919, doi: 10. 1110///P..2018.000
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. Free-Form Image Inpainting with Gated Convolution. 2018b. http:t http:httpp///axiv..oottto hatp://bs//8800 03589
Yanhong Zeng. Jianlong Fu, Hongyang Chao, and Baining Guo. Learning
Encoder Image Inpainting. pp. 1486-1494, 2019. URL http://axii.
Network for
exq/abs/19t 07475
Yinda Zhang and Thomas A. Funkhouser. Deep depth completion of single RGB-D image. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pp. 175-185, 2018. doi: 10.1099/VVPR2018.00002
A APPENDIX
A.I NETWORK PARAMETERS
The U-Net architecture similar to the one described in (Liu et al. 2018) with the sole difference that stride partial convolutions are only done every 2nd layer in order to fit the dataset resolution of 128 by 96.
For the DenseBlocks U-Net, we replace layers 10 with DenseBlock described in Table Note that DenseBlock as described in Huang et al. (2017) has skip connection between each layer. Outputs of all previous layers inside the block are concatenated and represent the input of the current layer. The encoder part is also modified to achieve constant compression ratio of 1.5 over
64 4111
each subsequent layer, Table 2 The number of features is computed follows: 1.5--I with
I being the layer number from 1 To 7. The decoder then connects from layer 17 to 22 and is built symmetrically to the encoder. Note that this architecture, although deeper compared to the U-Net (22 vs 16). has only about 28% many trainable parameters.
The stream function prediction network is DenseBlock that consists of 5 convolution layers, and is described in Table Activation is done with the swish function (Ramachandran al. (017) instead of the conventional ReLU because turned out to facilitate training and improve accuracy.
Training for all models is done using Adam optimizer with learning rate of 0.01 and batch size of 16. During training, all masks are used equally and the occlusion is set uniformly at random to a value between 25% and 99%.
10

---

Under review as a conference paper at ICLR 2020
Layer
2
3
4
5
6
Type
PConv
PConv
PConv
PConv
PConv
PConv
PConv
Output Resolution
128x96
64x48
64x48
32x24
32x24
16x
16x
Features
170
113
303
202
539
359
Kernel size
5
5
3
3
3
3
Stride
2
1
2
1
2
Activation
ReLU
ReLU
ReLU
ReLU
ReLU
ReLU
ReLU
Batch Normalization
No
Yes
Yes
Yes
Yes
Yes
Yes
Table 2: Structure of the encoder part in the velocity branch.
Layer
8-16
Type
PConv
Resolution
16x12
Features
Kernel size
Stride
Activation
ReLU
Batch Normalization
Yes
Table 3: Structure of the DenseBlock in the velocity branch.
Layer
24
Type zero padding
Conv
Conv zero padding
Conv
Padding
zero padding
Resolution
128x96
128x96
128x96
Features
64
32
Kernel size
Stride
Activation
swish
swish
swish
Batch Normalization
No
No
No
Table 4: Structure of the BenseBlock in the stream function prediction network.
A.2 LOSS FUNCTION WEIGHTS
The weights for all loss functions are listed in Table weighted differently on known and empty regions. We choose higher weight on empty regions
Except for Liort and Laive all losses
because they are more important for final results.
Loss Name
Loss Weight
Tyvelantid
Liellmpty
6
L
Ujeooiiawaatd
Ljacobianennly
6
Twwrr
6
Ldiv
Table 5: Loss functions weights. The losses Liel and Ljacobian are weighted differently on empty regions (emtpy) and known regions (valid).
11


Reviews:
[{"review_id": "r1gggNzRYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a physics-aware variant of a U-Net network for completing missing flow field data. Most notably, the loss functions are motivated by fluid dynamics, which forces the network to remain more consistent with the governing laws.\n\nDecision:\nI found the paper and the idea very exciting. Injecting domain knowledge by forcing the output (or differentiable transformations thereof) to be consistent with the physics is a quite relevant and appealing idea, for which this paper constitutes a nice proof-of-concept. Framing the problem of completing missing flow data as an inpainting task is also original. However, the evaluation of the method does not study important aspects regarding its generalization. The description of the experimental protocol is also missing important information.\n\nFurther arguments:\n- I found the discussion in Section 4.2 around method (b) not convincing. I do not understand why the network should be penalized if it does not reproduce the mistakes of the original simulator/solver. Since `div u` should be 0, why not simply penalizing ||div u|| instead of having the loss of Eqn. 8? Isn't approach (b) the approach which is most 'physics-aware' and correct from a physics point of view?\n- The experiments do not highlight whether the network actually just learn the training distribution or generalize by \"understanding\" the physics of the problem. A compelling experiment would have been to evaluate whether a network trained on a prior family of obstacles transfer properly to a different family (e.g., training on 6 spherical and 6 rectangle obstacles, but testing on fewer/more obstacles with other shapes).\n- Similarly, could the network generalize to larger/smaller inputs? Once trained, can it work on grids smaller/larger than 128x96? If not, what do you recommend to do in practice?\n- The description of the experimental protocol does not specify whether the method was evaluated on independent test data. More worrisome, section 4.2 even states that the MAE for Figure 2 is computed \"over the whole dataset\".\n- The method is not compared against any domain specific baseline. \n- While quite exciting, I am not confident the contribution is original enough from an ML point of view for ICLR, although it is certainly novel for fluid dynamics.\n\nAdditional feedback:\n- Some results reported in Table 1 are quite close of each other. It would have made the experiments much stronger if uncertainties were also reported and discussed.\n- Fig 3: I would have liked seeing the error maps of the methods. This would have been quite helpful to better tell them apart.\n- Given my comments above, I am confident an 8th page could be put to good use.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HygM1raHcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am not an expert in recent Navier-Stokes approaches, but note that there is a lot of recent work in physics aware modeling.  Specifically the sections on e.g. loss seem to have a lot of prior work. It’s difficult for me to judge the exact amount of novelty in this paper with respect to the physics. With respects to the DL part it looks like it’s mainly minor modifications to the known U-net architecture. \n* With respect to the introduction of the stream function branch, in table 1 one can observe that the error is actually higher than simply without it. The authors argue that “the synthetic velocity field data has discretisation errors and it is not truly divergence free. Therefore, the approach with a single stream function branch cannot capture the divergent modes present on the original data”. I’m not sure exactly what to think of this .. in my opinion this means they should use a more accurate flow solver for their simulation as otherwise it is hard to draw any definite conclusion here and only speculation remains. \n* I guess the inpainting works decently well, as is expected from previous image inpainting literature and the problem is essentially treated as image completion. However, I can’t really visually tell much of a difference between the images shown for the different approaches/components in figures 3-6. Again, this ties back to my previous component about lack of clarity of the improvement of the introduced individual terms. \n* I find it a little bit weird that there is exactly one reference prior to 2012. Fluid dynamics isn’t exactly a field that was introduced 5 years ago. Also their paper ends before the 8 page limit. I think the authors could have used the remaining space more efficiently. \n\nI generally like the idea of including physical consistency when training to train a neural network for a respective task where this matters. I’m just not sure I have a clear take-away from this paper as the results don’t seem to carry a clear message of the proposed approaches resulting in definite improvements over more naive approaches or including only partial physical consistency. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1ehNl2iYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors adopt prior work in image inpainting to the problem of 2d fluid velocity field inpainting by extending the network architecture and using additional loss functions. Specifically, the U-net network is extended with a DenseBlock in the middle, and a separate branch of the network is added which predicts the stream function (a different representation of the velocity field which guarantees incompressibility). The additional losses are L1 for various derivatives of the flow field (Jacobian, divergence, vorticity). Experiments presented in the paper show that these new elements improve the flow field error compared to a baseline model originally developed for image inpainting. The suggested application for this model is filling gaps in experimental measurements that are missing or impossible to obtain, and where such a model could be computationally cheaper than an actual fluid solver.\n\nThe paper discusses prior work at the appropriate level of depth, and specifies hyperparameters that were used for training.\n\nAs is, I do not believe the paper fully meets the bar for novelty and potential impact for acceptance at ICLR (but it could perhaps be a good fit for a more specialized venue). I am specifically not convinced about the practical applicability of the results. The novelty of the approach also seems quite limited, and the findings do not seem particularly surprising or insightful (i.e. that extending the vanilla U-net architecture and adding losses that directly bias the network towards physically meaningful solutions is better than the baseline).\n\nSince all training and testing data was generated from simulation, it is unclear how well the networks would cope with real world measurement noise. Furthermore, the authors note that the flow field from the simulation is sometimes not divergence-free. It is surprising that this could be a problem to a level where it could impact evaluations. If it indeed is, then perhaps it would make sense to either simulate the flows with a denser grid (inpainting could still be done on sparsified results). I found it surprising that the authors chose instead to use Eq. 8 and \"force\" the network to learn a flow field from the solver which is known to not be strictly physically valid.\n\nIt is also unclear to me that the masking schemes used in the experiments are relevant to actual measurements -- for instance, 2nd row of Fig. 3 or 1st row of Fig. 4 seem completely artificial. I appreciated the ablation study variants (but see also comments below for at least one more configuration that I believe should be discussed in addition to the existing ones). It would however be more informative (and important for potential practical applications) to include some sort of breakdown by mask type and flow field configuration/structures (steady, unsteady, wakes, jets, vortices, etc). The included images show the model does not capture some finer details of the velocity field (e.g. vortex structures in the last row of Fig. 3 and right column of Fig. 4; wakes behind small obstacles in Fig. 5-6, long, narrow, and fast jets in Fig. 6), but it is unclear what impact the proposed extensions (DenseBlock, losses, and stream function branch) have on these structures. Similarly, interpolating missing data points with a fairly dense input where no points are more than a few pixels apart seems like a much easier problem than filling large empty spaces, and it would make sense to do separate analyses for these cases.\n\nQuestions & suggestions for improvements:\n* Has the impact of different weight combinations in A.2. been investigated? How was the ratio of 6:1 for empty:valid determined?\n* The text says that in the stream function pathway, the features are passed \"through 4 densely connected convolution layers\". I was confused the first time I read that sentence, and only later I realized that this refers to a DenseNet-like pattern of connectivity. A reference to Huang et al. here or some other clarification would help.\n* In the figures, please consider also showing the difference between the predictions and ground truth to make it easier to see which features of the flow field are predicted accurately, and which are not.\n* Is there an L1 loss applied as well directly to the output of the stream prediction branch?\n* What are the Reynolds numbers used in the simulations? How far from the original Re does the network generalize?\n* When computing MAE, what are the units? What are the velocity magnitudes? Consider also reporting mean relative error.\n* Impact of the various L1 deriv. losses seems negligible when the stream function is used, but is more visible when only velocity is being directly predicted. Please comment on why that might be.\n* Why does (f) (Jacobian only) work better than (e) and (g)?\n* It is not clear that that the effect of the additional L1 losses is truly cumulative. Please consider testing just (u, DenseBlock, L_div).\n* It would be an interesting extension to include a non-ML baseline, which could be compared against the current results in terms of flow field quality and computational cost. \n* Can the network predictions deviate from the known (non-masked) values in the input? If so, please consider including a breakdown of the evaluation that shows how much the error varies between the \"valid\" and \"empty\" areas.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a physics-aware models for inpainting fluid data. In particular, the authors extend the vanilla U-net architecture and add losses that explicitly  bias the network towards physically meaningful solutions. 

While the reviewers found the work to be interesting, they raised a few questions/objections which are summarised below:

1) Novelty: The reviewers largely found the idea to be novel. I agree that this is indeed novel and a step in the right direction.
2) Experiments: The main objection was to the experimental methodology. In particular, since most of the experiments were on simulated data the reviewers expected simulations where the test conditions were a bit more different than the training conditions. It is not very clear whether the training and test conditions were different and it would have been useful if the authors had clarified this in the rebuttal. The reviewers have also suggested a more thorough ablation study.
3) Organisation: The authors could have used the space more effectively by providing additional details and ablation studies.

Unfortunately, the authors did not engage with the reviewers and respond to their queries. I understand that this could have been because of the poor ratings which would have made the authors believe that a discussion wouldn't help. The reviewers have asked very relevant Qs and made some interesting suggestions about the experimental setup. I strongly recommend the authors to consider these during subsequent submissions. 

Based on the reviewer comments and lack of response from the authors, I recommend that the paper cannot be accepted. 
Decision: Reject