Frequency-aware Interface Dynamics with Generative Adversarial Networks

1 INTRODUCTION . Complex and chaotic physical phenomena such as liquids , gels and goo are still very challenging when it comes to representing them as detailed and realistically as possible . A variety of numerical methods have been proposed to simulate such materials , from purely Eulerian methods ( Harlow & Welch , 1965 ; Stam , 1999 ) , over particle based methods ( Gingold & Monaghan , 1977 ; Ihmsen et al. , 2014 ) , to hybrids ( Zhu & Bridson , 2005 ; Stomakhin et al. , 2013 ) . Such simulations have also been targeted with deep learning methods ( Tompson et al. , 2017 ; Mrowca et al. , 2018 ; Li et al. , 2019 ) , but despite significant advances , they remain very time-consuming and highly challenging to solve . One approach to speed up the necessary calculations and to allow for more control is to employ super-sampling . This can be seen as a form of post-processing where one simulates only a lowresolution simulation and uses an up-sampling technique to approximate the behavior of a highresolution simulation . Neural networks are of special interest here because of their capability to efficiently approximate the strongly nonlinear behavior of physical simulations . Applying neural networks to space-time data sets of physical simulations has seen strongly growing interest in recent years ( Ladicky et al. , 2015 ; Kim et al. , 2020 ) , and is particularly interesting in this context to incorporate additional constraints , e.g. , for temporal coherence ( Xie et al. , 2018 ) , or for physical plausibility ( Tompson et al. , 2017 ; Kim et al. , 2019 ) . An important aspect here is that methods based on simple distance losses , such as mean square errors , quickly reach their limits . The generated data tends to be smooth without the necessary small-scale features . Generative adversarial networks ( GANs ) have been proposed to overcome this issue ( Goodfellow , 2016 ) . They are characterized by the fact that , apart from a generative network , they also make use of a discriminator that classifies the results of the generator with respect to the ground-truth data . Via a joint training , the distribution of solutions of the generator is guided to approximate the ground-truth data distribution . As the quality of the results is primarily determined by the discriminator network , it remains an open problem to accurately evaluate the quality of the inferred results . In our work we propose to evaluate the problem in the Fourier space . In this way , we are able to evaluate the given methods reliably , and it allows us to design improved learning algorithms that more faithfully recover the small scale details of the reference data . For the core of our method , we build on an existing GAN-based architecture that employs two discriminator networks , one for the spatial and one for the temporal behaviour ( Xie et al. , 2018 ) . In terms of ground truth data , we focus on multi-phase ( solid-fluid-air ) interactions with a sharp fluid-air interface . Unlike single-phase flow whose details are visible and relevant solely due to transparency throughout the volume , the details of our data are in most cases only visible on the surface . Of course , the internal dynamics in the volume also play a role , but they are mostly hidden from the viewer , only the effects on the surface are visible . Furthermore , we consider phenomena that build up and take place over the course of several frames . Thus , as we will outline below , we employ a recurrent approach that is conditioned on a previous output in order to produce the solution for a subsequent timestep . In order to represent and process fine details , we treat such detail as high-frequency displacements of a low-frequency surface , and correspondingly formulate the problem in Fourier space . The transformation into Fourier space yields an isolated view of the individual frequencies , and thus allows for a much improved analysis of the results achieved by different methods . E.g. , it robustly identifies the strong smoothing behavior of L2 metrics , and can detect mode collapse problems of adversarial training runs . We also demonstrate how frequency information can be incorporated into the learning objective in order to improve results . To summarize , the central contributions of our work are : ( 1 ) A method for frequency evaluation with a consideration of spatial properties , ( 2 ) A novel frequency aware loss formulation , ( 3 ) A simple , yet intuitive evaluation of different generative methods , ( 4 ) A time consistent spatio-temporal upsampling of complex physical surfaces . Related Work Deep learning methods in conjunction with physical models were employed in variety of contexts , ranging from learning models for physical intuition ( Battaglia et al. , 2016 ; Sanchez-Gonzalez et al. , 2018 ) , over robotic control ( Schenck & Fox , 2018 ; Hu et al. , 2019 ) to engineering applications ( Ling et al. , 2016 ; Morton et al. , 2018 ) . In the following , we focus on fluid-like materials with continuous descriptions , which encompass a wide range of behavior and pose challenging tasks for learning methods ( Mrowca et al. , 2018 ; Li et al. , 2019 ) . For fluid flows in particular , a variety of learning methods were proposed ( Tompson et al. , 2017 ; Prantl et al. , 2017 ; Um et al. , 2018 ) . A common approach to reduce the high computational cost of a simulation is to employ super-resolution techniques ( Dong et al. , 2016 ; Chu & Thuerey , 2017 ; Bai et al. , 2019 ) . In this context , our work targets the up-sampling for physics-based animations , for which we leverage the approach proposed by Xie et al . ( 2018 ) . However , in contrast to this work , we target phenomena with clear interfaces , which motivates the frequency-based viewpoint of our work . For sharp interfaces , Lagrangian models are a very popular discretization of continuum mechanical systems . E.g. , smoothed particle hydrodynamics ( SPH ) ( Gingold & Monaghan , 1977 ; Koschier et al. , 2019 ) is a widely-used particle-based simulation method . While points and particles are likewise frequently used representations for physical deep learning ( Li et al. , 2019 ; Ummenhofer et al. , 2019 ; Sanchez-Gonzalez et al. , 2020 ) , Eulerian , i.e. , grid-based representations offer advantages in terms of efficient and robust kernel evaluations . We employ generative adversarial networks ( Goodfellow , 2016 ) , as a powerful and established method for learning generative models . Here , ” unconditional ” GANs typically rely on a synthetic input vector from Gaussian noise to produce the desired output distribution , e.g. , the DC-GAN approach ( Radford et al. , 2016 ) . Conditional GANs ( Mirza & Osindero , 2014 ) were introduced to provide the network with an input that allows the neural network to steer the generation of the output . Hence super-resolution tasks for natural images ( Ledig et al. , 2016 ) , or image translation tasks ( Isola et al. , 2017 ) employ conditional GANs . The time dimension was also taken into account in natural imaging works , e.g. , by Saito et al . in the form of a temporal generator ( Saito et al. , 2017 ) , or via a stochastic sequence generator ( Yu et al. , 2017 ) . Other works have included direct L2 loss terms as temporal regularizers ( Bhattacharjee & Das , 2017 ; Chen et al. , 2017 ) , which , however , typically strongly restricts the changes over time . Similar to flow advection , video networks also often use warping information to align data over time ( Liu et al. , 2017 ; de Bezenac et al. , 2017 ) . We will demonstrate that recurrent architectures similar to those used for video super-resolution ( Sajjadi et al. , 2018 ) are likewise very amenable for physical problems over time . 2 METHOD . The input for our method is a coarsely approximated source simulation , with the learning objective to infer the surface of a target simulation over space and time . This target is typically computed via a potentially very costly , finely resolved simulation run for the same physical setup . When it comes to the possibilities of simulation representations , there is a great variance . In our case we have chosen an implicit representation of the data , by a signed-distance field ( SDF ) denoted by g : R3 → R. An SDF returns , for a given point , the signed distance to the surface , with negative being inside the medium . Such a function is realized in practice by a grid X ∈ RMx×My×Mz , storing the pre-computed signed distance values , where M∗ , ∗ ∈ { x , y , z } specifies the size of the grid in the respective dimension x , y or z . We have chosen this representation because most neural network layers are designed for array-like representations , and the loss functions on grid-based data are very efficient to evaluate . Additionally , an implicit representation via a grid can leverage tools from the field of level-set processing ( Adalsteinsson & Sethian , 1999 ) , and facilitates the frequency viewpoint via a Fourier transformation . Additional values , like the velocity , are also mapped on a grid V ∈ RMx×My×Mz×3 . Our goal is to let a generative network G : RMx×My×Mz×4 → RNx×Ny×Nz infer a grid Ỹ which approximates a desired high-resolution simulation Y ∈ RNx×Ny×Nz with N∗ = kM∗ , N∗ ∈ N { x , y , z } and up-sample factor k ∈ N , i.e . G ( X ) = Ỹ ≈ Y . As our method only requires position and velocity data from a simulation , it is largely agnostic to the type of solver or physical model for generating the source and target particle data . 2.1 NEURAL NETWORK FORMULATION . Our method is based on a generative , neural network with a 3D fully-convolutional ResNet architecture ( He et al. , 2016 ) that produces an output field at a single instance in time . The low-resolution input data is first up-sampled with a tri-linear up-sampling and then processed with several convolutional layers , as shown in Figure 1a . We use leaky ReLU as activation function after each layer , except for the last layer , where we use a tanh activation . In our case , the input data consists of the implicitly represented geometry data Xt , the velocity Vt of the simulation as well as the results of a previous pass Ỹt . The previously generated data is advected with the low-resolution velocity before further processing . Through this feedback loop we train our network recurrently by iterating over a sequence of T = 10 frames . This yields stability over longer periods of time and gives better insights about temporal behaviour . Furthermore , the recurrent training is important to enable persistent behavior over time , such as the progression of fine surface waves . Unlike the process for generating the input data , the network training can not resort to a physical simulation with full resolution , and hence can not uniquely determine the evolution of future states . Therefore , its main learning objective is to capture the dynamics of the target simulations beyond that basic motion computed with an advection step . For initialization of the undefined first frame Ỹ−1 we use a tri-linear up-sampled version of the input . To train our network we have to define first a loss function that allows us to evaluate the differences between generated and ground-truth data . The most basic loss function is a simple mean squared error ( MSE ) : Ls = ||Y − Ỹ ||22 . ( 1 ) This has the big disadvantage that it is ill-suited to measure the similarity or differences of solutions . For example , considering a function with multiple solutions for a given input , i.e. , a multi-modal setting , a method that trains with an MSE loss will learn the expected value of the output distribution , i.e , the average of the different solutions . However , the average is typically not a part of the solution set . Thus , the MSE loss often does not correspond to the correct distance in solution space , based on significant factors corresponding to the distribution of the solutions . Our super-sampling setup is such a problem : Due to the low resolution input , the high resolution details can not be determined uniquely , resulting in a variety of possible solutions when up-sampling . Via physical properties of the material and its temporal sequence , some solutions can be eliminated , but nonetheless the space of solutions typically remains infinitely large . If an MSE loss is used , all such samples from the training data set are simply averaged to obtain a mean value , so that the result does no longer reflect the level of detail of the ground-truth data . The MSE loss nevertheless gives a rough direction , and provides a stable learning target . Hence , we still use it as a component in the final loss formulation , in combination with an adversarial loss . In contrast to a direct distance metric , the adversarial loss approximates the ground-truth distribution . Hence , the network no longer learns one mean value , but chooses one valid solution out of the possible ones . We define a discriminator Ds that takes as input a high-resolution version of a simulation frame and classifies it , distinguishing between ground-truth and generated frames . It does this through a binary output , where 0 is ” fake ” and 1 is ” real ” . Its task is to provide the generator with feedback on the correctness of the given data . The special feature is that the discriminators are trained together with the generator , thus creating a competitive interaction where both parties improve each other . As loss for the discriminator we use a binary cross-entropy : Lbce = y log ( ỹ ) + ( 1− y ) log ( 1− ỹ ) , ( 2 ) where y is the ground-truth and ỹ is the generated value from the discriminator . For complex tasks , GANs can be unstable and difficult to control . For this reason we additionally use the recent Spectral Normalization ( Miyato et al. , 2018 ) , which we found to provide more stable adversarial training . While we have primarily focused on spatial content , i.e. , the surface of the material so far , the temporal behavior likewise plays a crucial role , and poses similar difficulties in our multi-modal setting . On the one hand , the generation of details can quickly lead to temporally incoherent results , which is characterized by unappealing flickering . On the other hand , our network also should be able to match and recreate spatial solutions over time that reflect the physical behavior . Following previous work ( Xie et al. , 2018 ) , we use an additional discriminator Dt to classify the temporal behavior of data . This is done by passing three corresponding frames , which are aligned with each other using advection A : RNx×Ny×Nz×3 × RMx×My×Mz×3 → RNx×Ny×Nz×3 . Apart from this , the temporal discriminator closely follows the structure of the spatial discriminator . Both discriminators ( Figure 1b ) use a typical funnel structure , where the dimension is increasingly reduced using strided convolutional layers , with a last fully connected layer computing the classification result . We likewise use leaky ReLU activations , with a sigmoid function for the last layer . The classification of the discriminators is included in the loss formulation of the generator : LDs = 1 T T∑ t Ds ( G ( tX ) , Xt ) , LDt = Dt ( A ( G ( Xt−1 ) , Vt ) , G ( X ) , A ( G ( Xt+1 ) , −Vt ) , X ) , ( 3 ) which gives the final loss function : LG = Ls + αLDs + βLDt , ( 4 ) where α and β indicate the weighting of the individual loss terms . An additional benefit of the adversarial loss is that it allows for learning from unpaired data . A common problem for up-sampling methods is the generation of paired ground truth data for training . Due to different numerical approximations , and hence potentially differing physical behavior , the easiest solution is to simulate at high resolution , and down-sample the data . While at training time the down-sampled data is used , at test time , the model needs to be applied to data from a lowresolution simulation instead . This typically leads to large distribution shifts , and correspondingly impaired inference quality . Therefore , we take an unpaired training approach into account that decouples the low and high resolution data . The feedback from the discriminators is still based on the ground-truth data , which makes the output conditionally dependent on the input , but also approximates the behavior of the reference data . However , there is no direct supervision in the generator anymore : the output is no longer compared with a matching ground-truth in the loss , but only related to the input . This is done by down-sampling the output and comparing it with the input : L∗s = ||X − p ( Ỹ ) ||22 , ( 5 ) where p : RNx×Ny×Nz×3 → RMx×My×Mz×3 is a down-sampling function based on average pooling . This effectively removes the need for paired low- and high-resolution samples at training time , and fully relies on the discriminator to match both distributions . To indicate the focus on surface structures , we refer to the final version of our generative network as surfGAN . For a more detailed description of the training and the network architecture we refer to the appendix A.1 and A.2 .

Summary: This paper presents a GAN framework to learn spatial and temporal representation on complex physical surfaces to apply to simulation. The method represents data in a SDF-like way so it is agnostic to the properties of material and simulation model. The network is built upon conventional GAN, with two discriminator for temporal and spatial. A loss function is proposed to evaluates on a grid-based Fourier transform of the output and ground truth to better preserve high frequency details (temporally and spatially). Results show performance on a physical simulation with opaque, elasto-plastic materials.