A New Perspective on Fluid Simulation: An Image-to-Image Translation Task via Neural Networks

1 INTRODUCTION . 1.1 TARGET ISSUE . Simulating fluids , streams , and flows is a task in many fields of science . In most cases , this is done by numerical approaches like the method of finite elements ( FEM ) ( Quarteroni & Valli , 2008 ) or Lattice Boltzmann methods ( LBM ) ( Mohamad , 2011 ) . These methods are providing great advantages like configurable accuracy and fine-grained adaptability regarding the specific given task . However , to get close-to-reality simulations , some issues have to be targeted as well . One of the most important issues emerges from the approximation property of these methods . This property limits the configuration of accuracy in both directions : if the accuracy becomes too low , the approximation will become to coarse — in other words , it will no longer reflect reality respectively fulfill the given task . In contrast , if it becomes too high , effects like the curse of dimension will set in , causing exploding problem sizes . In the result , these methods will have an only minimal adjustable computation time range , as the accuracy strongly connects to the problem size and therefore also to the computation time , and may exceed every given maximum time span . To avoid this issue , a different approach might be more satisfying . As our focus is on problems with no need of high accuracy , we decided to follow a more approximate path by involving the human observer . This observer typically isn ’ t interested in the numerical representation of the simulation , but in the easy-to-interpret image-based one . That leads directly to the idea to focus on an imageto-image translation for each time step of the simulation . The results of such an approach will be much more inaccurate as typical numerical results — the very limited resolution of the color spaces alone results in high errors compared to FEM or LBM results — but for a wide class of problems the results will be good enough under the condition that the computations are very fast . Therefore , the main task for this approach is to translate an image representation of time step n into the image representation of step n+ 1 as fast and as accurate as possible . For translation , we decided to use a neural network based on good results for image-to-image translation ( Isola et al. , 2017 ) . In total , this approach — starting with an image representation of the starting values and translate it recursively into the next time steps of the simulation — leads to the following question : • Is it possible ... regarding computation time and accuracy . Can we get results fast enough to accept the additional approximation ? Is the accuracy high enough to get useful results ? Will this accuracy remain over many simulation time steps ? • ... to get accurate simulations ... regarding the real world and the human observer . How much noise in the picture is too much ? When and why does our approach begin to fail ? • ... very fast ... regarding the computation time and deployed hardware . After training , is it possible to run a complex simulation on my laptop at home ? Will it be fast enough to meet real-time conditions ? • ... with this approach ... regarding the recursive usage of neural networks as well as an unedited image input . Can we use unedited data , or do we need preprocessing steps like FFT ? Is the recursive approach expedient ? Are image processing steps like morphological filtering needed between each step ? Is the cGAN approach with a UNET architecture reasonable ? Do we need additional LSTM units ? • ... while using as little as possible parameters ? regarding the input-output-ratio . How much data is needed to produce the same output as LBM or FEM ? How generalizable is our approach ? Do we need to train every explicit model , geometry , and structure or is it possible to transfer results ? Some emerging questions aren ’ t trivial at all , and this paper can ’ t answer all of them . With our work , we want to show the benefits of the mentioned approach and highlight opportunities and limits . 1.2 STATE OF THE ART . As our approach covers multiple topics , this state-of-the-art section will compare our work to numerical approaches , image translation methods , architectures of neural networks , comparable approaches for fluid dynamics , and previous work our approach is based on . For a better overview , we will headline the corresponding sections and summarize the latest developments . Numerical Approaches : Standard numerical methods for solving PDEs , like FEM ( Quarteroni & Valli , 2008 ) or LBM ( Mohamad , 2011 ) , are widely known . Ideas like approximate preconditioning ( Anzt et al. , 2018 ) or multi-precision solvers of systems of equations ( Gratton et al. , 2019 ; Aliaga et al. , 2020 ) are emerging over the last years . These approaches can have a great impact on a specific part of the numerical PDE solver , but this doesn ’ t necessarily lead to faster run-times for the whole PDE solver . There are also some more global approaches to speed up a complete PDE solving method thanks to mathematical optimization , like Gracie et al . ( 2006 ) ; Du & Wang ( 2015 ) ; Etzmuss et al . ( 2003 ) , but these approaches are highly adapted to a specific problem . Based on these findings , a pure mathematical approach doesn ’ t seem to be the right way . Image Translation : Our basic idea is based on image translation with a neural network . In the last years many approaches in this direction appear , manly ( but not only ) to manipulate images or movies in real-time ( Liu et al. , 2017 ; Radford et al. , 2015 ; Zhao et al. , 2020 ) . Most of these approaches are not really matching our task , which results in unsuitable network architectures or unrealizable constraints . But there is one matching approach with Isola et al . ( 2017 ) . Our cGAN approach is inspired by the ideas and the excellent results given there and in additional work on cGAN structures like Karras et al . ( 2017 ) ; Zhang et al . ( 2017 ) . Neural Network Architectures : In addition to the cGAN approach , we need to find the right architecture for our neural networks . Based on Isola et al . ( 2017 ) we used a PatchGAN architecture ( Li & Wand , 2016 ) for the discriminator part . For the generator part , we used the proposed U-Net architecture ( Isola et al. , 2017 ) . Regarding the iterative structure of our translation task , we added the idea of long short-term memory modules ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) to improve the U-NET structure . Neural Fluid Dynamics and PDE Solvers : Combining numerical methods with machine learning algorithms is not entirely new , even in the field of fluid dynamics . Two very up-to-date examples are Li et al . ( 2020 ) and Pfaff et al . ( 2021 ) . While the first one focuses on the numerical operators and numerical errors , the second one tries to work with the numerical discretization . Both – as other approaches before – are following approaches starting within the numerical solving pipeline . Our approach to work solely on the image representation clearly separates us from previous approaches and is a unique characteristic of this paper . Unfortunately , our work is not yet at the point to be fully comparable with these approaches , but we are aiming to it in the near future . Additional Influences : Finally , there is one more idea we have to mention as part of the base of our work . In Lehmann et al . ( 2020 ) the authors showed , why and how a binary map is a good option to define a sharp separation of areas within an image regarding the image translation task . We adopted this option for our approach , as we need a sharp separation between the streaming area and the environment area in the image representation . 1.3 CONTENT ORGANIZATION . In detail , we will provide data for the following main findings in this paper : • Using a pix2pix-approach with a U-NET structure and a cGAN training is a useful way to get approximated simulation results • Results from recursive application of neuronal networks are useful over tenth to hundreds of iterations , including an only moderate increasing additional approximation error • The speed-up can be upto 9 ( on GPU-based hardware ) compared to a FEM-based simulation • Different color spaces , boundary mappings , and input data sets are possible and may lead to data-driven approaches We structured this paper as the following : We start with some basic knowledge about FEM and image comparison , followed by explanations of our neuronal network architecture , and our data generation in section 2 . Section 3 will cover our results for our test setting ( see A.3 ) and will give a look over the edge of this specific configured environment . We close this work with a conclusion and summary in section 4 as well as a view on our future directions . The statement of r/eproducibility can found at the very end . 2 THEORETICAL BACKGROUND . 2.1 NUMERICAL BASIS AND MESSURMENT . Looking into our approach , everything starts with a given PDE ( partial differential equation ) , the mathematical model behind descriptions of physical phenomenons . In our case , this is the NavierStokes equations for incompressible flows ( Oymak & Selcuk , 1996 ) . The starting conditions , boundary conditions , and physical parameters are chosen in a way to get a Kármán vortex street within a canal with an obstacle ( fig . 1 ) ( Schäfer et al. , 1996 ) . For discretization , the method of lines ( Oymak & Selcuk , 1996 ) is used , which leads to discrete time steps each equipped with the same discrete space grid . The values derived from the chosen solvers1 are finally mapped to a chosen color space ( in our case it is the [ 0 , 255 ] grayscale ) . This very rough spotlight on the basic method shows where the main issue can be located : There are a lot of approximation errors in the reformulation , discretization , numerical solvers , and in the final mapping to the color space ( Quarteroni & Valli , 2008 ) . Additionally , the single errors may ( or may not ) accumulate , which is the reason it can be very difficult to change a single solver or transformation method out of the solving pipeline . Especially in the case of neural networks , where 1Θ-step methods for the time direction ( Berzins & Furzeland , 1992 ) and a variant of Newton ’ s method ( Nemec & Zingg , 2002 ) for the non-linear equations in space . no useful formal error limit guaranties can be given , a global replacement approach seems to be the more promising than changing a single pipeline step . As this means to replace the whole numerical method with a neural network and only keep the starting values to create the first input data , we have to solve the issue to define a meaningful quality measurement for our results . For run time a simple time difference , respectively a speed-up measurement , is good enough . For accuracy , comparing the numerically created image of a time step with one created by a neural network pixel by pixel may lead to high errors , even if the images are indistinguishable by a human observer . However , the same measurement may only result in small errors even if one can find obvious false streaming data ( with small changes in the colors ) . As we found in previous testing , this problem will occur in our case also for average pixel errors and correlation-based measurements . Therefore , the most promising quality measurement for us is based on the mean square error ( MSE ) and originally developed to evaluate the quality of lossy compression algorithms . It is called peak signal-to-noise ratio ( PSNR ) ( Korhonen & You , 2012 ) : εPSNR = 10 · log10 ( 2552 εMSE ) [ dB ] . ( 1 ) In theory , a higher PSNR value means less detectable differences in the images , and values above 30 dB should result in only undetectable differences for a human observer ( Mehra , 2016 ) . In practice , we noticed that in many cases , values below this significant value might be acceptable as well . The reason is that the first observable errors are irrelevant ones regarding our setting , like discolored vertical or horizontal pixel lines that obviously can not be interpreted as streaming data . Therefore , we marked the theoretical limit in our charts and stopped our iteration some steps later .

Summary: The authors pose fluid simulation as an image-to-image translation task. From this perspective, approximating fluid flow using a cGAN can potentially improve speed (at the expense of accuracy) over FEM methods. This can be useful in situations where accuracy can be traded off for speed (e.g. video games). The authors show a U-net can produce good future predictions for Karman vortex sheets, and is faster than an off the shelf multiphysics simulator. In addition the authors ablate various architectural design choices.