Generating Liquid Simulations with Deformation-aware Neural Networks

1 INTRODUCTION . Learning physical functions is an area of growing interest within the research community , with applications ranging from physical priors for computer vision problems Kyriazis & Argyros ( 2013 ) , over robotic control Schenck & Fox ( 2017 ) , to fast approximations for numerical solvers Tompson et al . ( 2017 ) . While the underlying model equations for many physics problems are known , finding solutions is often prohibitively expensive for phenomena on human scales . At the same time , the availability of model equations allows for the creation of reliable ground truth data for training , if enough computational resources can be allocated . Water , and liquids in general , are ubiquitous in our world . At the same time , they represent an especially tough class of physics problems , as the constantly changing boundary conditions at the liquid-gas interface result in a complex space of surface motions and configurations . In this work we present a novel approach to capture parametrized spaces of liquid behavior that is based on space-time deformations . We represent a single 3D input surface over time as a four-dimensional signed-distance function ( SDF ) , which we deform in both space and time with learned deformations to recover the desired physical behavior . To calculate and represent these deformations efficiently , we take a two-stage approach : First , we span the sides of the original parameter region with precomputed deformations , and infer a suitable weighting function . In a second step , we synthesize a dense deformation field for refinement . As both the parameter weighting problem and the deformation synthesis are highly non-linear problems , we demonstrate that neural networks are a particularly suitable solver to robustly find solutions . We will demonstrate that it is possible to incorporate the non-linear effects of weighted deformations into the loss functions of neural networks . In particular , we put emphasis on incorporating the influence of deformation alignment into the loss gradients . This alignment step is necessary to ensure the correct application of multiple consecutive deformations fields . The second stage of our algorithm is a generative model for deformation fields , for which we rely on a known parametrization of the inputs . Thus , in contrast to other generative models which learn to represent unknown parametrization of data sets Radford et al . ( 2016 ) , our models are trained with a known range and dimensionality to parameter range , which serves as input . Once trained , the models can be evaluated very efficiently to synthesize new implicit surface configurations . To demonstrate its performance , we have implemented a proof-of-concept version for mobile devices , and a demo app is available for Android devices in the Google Play store . Our approach generates liquid animations several orders of magnitude faster than a traditional simulator , and achieves effective speed up factors of more than 2000 , as we will outline in Sec . 5 . The central contributions of our work are : • A novel deformation-aware neural network approach to very efficiently represent large collections of space-time surfaces with complex behavior . • We show how to compute suitable loss gradient approximations for the sub-problems of parameter and deformation inference . • In addition we showcase the high performance of our approach with a mobile device implementation that generates liquid simulations interactively . 2 RELATED WORK . Capturing physical behavior with learning has a long history in the field of learning . Early examples targeted minimization problems to determine physical trajectories or contact forces Bhat et al . ( 2002 ) ; Brubaker et al . ( 2009 ) , or plausible physics for interacting objects Kyriazis & Argyros ( 2013 ; 2014 ) . Since initial experiments with physically-based animation and neural networks Grzeszczuk et al . ( 1998 ) , a variety of new deep learning based works have been proposed to learn physical models from videos Battaglia et al . ( 2016 ) ; Chang et al . ( 2016 ) ; Watters et al . ( 2017 ) . Others have targeted this goal in specific settings such as robotic interactions Finn et al . ( 2016 ) , sliding and colliding objects Wu et al . ( 2015 ; 2016 ) , billiard Fragkiadaki et al . ( 2015 ) , or trajectories in height fields Ehrhardt et al . ( 2017 ) . The prediction of forces to infer image-space motions has likewise been targeted Mottaghi et al . ( 2016a ; b ) , while other researchers demonstrated that the stability of stacked objects can be determined from single images Lerer et al . ( 2016 ) ; Li et al . ( 2016 ) . In addition , the unsupervised inference of physical functions poses interesting problems Stewart & Ermon ( 2017 ) . While these methods have achieved impressive results , an important difference to our method is that we omit the projection to images . I.e. , we directly work with three-dimensional data sets over time . In the context of robotic control , physics play a particularly important role , and learning object properties from poking motions Agrawal et al . ( 2016 ) , or interactions with liquids Schenck & Fox ( 2017 ) were targeted in previous work . Learning physical principles was also demonstrated for automated experiments with reinforcement learning Denil et al . ( 2016 ) . Recently , first works have also addressed replacing numerical solvers with trained models for more generic PDEs Farimani et al . ( 2017 ) ; Long et al . ( 2017 ) . In our work we target a more narrow case : that of surfaces deforming based on physical principles . However , thanks to this narrow scope and our specialized loss functions we can generate very complex physical effects in 3D plus time . Our method can be seen as a generative approach representing samples from a chaotic process . In this context , the learned latent-space representation of regular generative models Masci et al . ( 2011 ) ; Rezende et al . ( 2014 ) ; Goodfellow et al . ( 2014 ) ; Radford et al . ( 2016 ) ; Isola et al . ( 2017 ) is replaced by the chosen parametrization of a numerical solver . Our model shares the goal to learn flow physics based on examples with other methods Ladicky et al . ( 2015 ) ; Tompson et al . ( 2017 ) ; Chu & Thuerey ( 2017 ) , but in contrast to these we focus on 4D volumes of physics data , instead of localized windows at single instances of time . Alternative methods have been proposed to work with 4D simulation data Raveendran et al . ( 2014 ) ; Thuerey ( 2017 ) , however , without being able to capture larger spaces of physical behavior . Due to its focus on deformations , our work also shares similarities with methods for optical flow inference and image correspondences Bailer et al . ( 2016 ) ; Dosovitskiy et al . ( 2015 ) ; Ranjan & Black ( 2016 ) ; Ilg et al . ( 2016 ) . A difference to these approaches is that we learn deformations and their weighting in an unsupervised manner , without explicit ground truth data . Thus , our method shares similarities with spatial transformer networks ( STNs ) Jaderberg et al . ( 2015 ) , and unsupervised approaches for optical flow Meister et al . ( 2017 ) . However , instead of aligning two data sets , our method aims for representing larger , parametrized spaces of deformations . 3 LEARNING DEFORMATIONS . We first explain our formulation of the simulation parameter space , which replaces the latent space of other generative models such as autoencoders Masci et al . ( 2011 ) , GANs Radford et al . ( 2016 ) , or auto-regressive models Van Oord et al . ( 2016 ) . Given a Navier-Stokes boundary value problem with a liquid-gas interface , we treat the interface over time as a single space-time surface . We work with a set of these space-time surfaces , defined over a chosen region of the N-dimensional simulation parameters α . We assume the parameters to be normalized , i.e . α ∈ [ 0 , 1 ] N . In practice , α could contain any set of parameters of the simulation , e.g . initial positions , or even physical parameters such as viscosity . We choose implicit functions φ ( α ) ∈ R4 → R to represent specific instances , such that Γ ( α ) = { x ∈ R4 ; φ ( α , x ) = 0 } is the space of surfaces parametrized by α that our generative model should capture . In the following , φ and ψ will denote four-dimensional signed distance functions . We will typically abbreviate φ ( α ) with φα to indicate that φα represents a set of constant reference surfaces . While we will later on discretize all functions and operators on regular Cartesian grids , we will first show the continuous formulation in the following . Deforming Implicit Surfaces Representing the whole set φα is very challenging . Due to bifurcations and discretization artifacts φα represents a process that exhibits noisy and chaotic behavior , an example for which can be found in Fig . 1 . Despite the complex changes in the data , our goal is to find a manageable and reduced representation . Thus , we generate an approximation of φα by deforming a single initial surface in space and time . We apply space-time deformations u : R4 → R4 to the initial surface ψ0 ( x ) . As a single deformation is limited in terms of its representative power to produce different shapes , we make use of N sequential , pre-computed deformations , thus ui with i ∈ [ 1 · · ·N ] , each of which is scaled by a scalar weight parameter βi , whose values are normally between 0 and 1 . This gives us the freedom to choose how much of ui to apply . The initial surface deformed by , e.g. , u1 is given by ψ0 ( x− β1u1 ) . The sequence of deformed surfaces by sequentially applying all pre-computed ui is given by ψi ( x , β ) = ψi−1 ( x − βiui ) . It is crucial to align such sequences of Eulerian deformations with each other . Here we employ the alignment from previous work Thuerey ( 2017 ) , which we briefly summarize in the following , as it influences the gradient calculation below . Each deformation i relies on a certain spatial configuration for the input surface from deformation i− 1 . Thus , when applying ui−1 with a weight βi−1 < 1 , we have to align ui correspondingly . Given the combined deformation vsum ( x , α ) = ∑N i=1 βiu ∗ i ( x ) , with intermediate deformation fields u ∗ i−1 ( x ) = ui−1 ( x − u∗i ( x ) ) , we compute an inversely weighted offset field as vinv ( x , α ) = − ∑N i=1 ( 1 − βi ) u∗i ( x ) . This offset field is used to align the accumulated deformations to compute the final deformation as vfin ( x + vinv ( x , β ) , β ) = vsum ( x , β ) . vfin now represents all weighted deformations βiui merged into a single vector field . Intuitively , this process moves all deformation vectors to the right location for the initial surface ψ0 , such that they can be weighted and accumulated . To achieve the goal of representing the full set of target implicit surfaces φα our goal is to compute two functions : the first one aims for an optimal weighting for each of the deformations in the sequence , i.e . β , while the second function computes a final refinement deformation w after the weighted sequence has been applied . We will employ two neural networks to approximate the two functions , which we will denote as fp , and fd below . Both functions depend only on the simulation parameters space α , i.e. , fp ( α ) = β , and fd ( α ) = w. Splitting the problem into fp and fd is important , as each of the pre-computed deformations weighted by fp only covers a single trajectory in the space of deformed surfaces . In the follow- ing , we employ an optical flow solve from previous work to pre-compute deformations between the inputs Thuerey ( 2017 ) , which deform the input for the extrema of each original parameter dimension αi . E.g. , for a two-dimensional parameter space this yields two deformations along the sides of the unit cube . This first step robustly covers rough , large scale deformations . As a second step , we employ fd , which is realized as a generative CNN , to infer a deformation for refining the solution . Below we will explain the resulting equations for training . The full equations for applying the deformations , and a full derivation of our loss functions can be found in the supplemental materials . To shorten the notation , we introduce the helper function D ( xi , α ) , which yields a deformed set of coordinates in R4 depending on α that incorporates the deformation sequence weighted by fp ( α ) , and a refinement deformation from fd ( α ) . We express the overall goal in terms of minimizing the L2 distance between the deformed and the target implicit surfaces for all possible values in the parameter space α , using β and w as degrees of freedom : argmin β , w L , L = ∫ ‖ψ0 ( D ( xi , α ) ) − φα‖22 dα . ( 1 ) Our work addresses the problem of how to compute weighting of the deformations and on synthesizing the refinement field . The main difficulty lies in the non-linearity of the deformations , which is why we propose a novel method to robustly approximate both functions with NNs : fp will be represented by the parameter network to compute β , and we make use of a deformation network that to generate w. We employ relatively simple neural networks for both functions . Key for training them is encoding the effect of deformations in the loss functions to make the training process aware of their influence . Hence , we will focus on describing the loss functions for both networks and the corresponding discretized gradients in the following . Learning Deformation Weights For training the NNs we propose the following objective function , which measures the similarity of a known reference surface φα and the corresponding , approximated result ψ0 ( x , α ) for a parameter value α . We introduce the numerical equivalent of the continuous L2 loss from Eq . ( 1 ) as L = 1 2 ∑ i ( ψ0 ( D ( xi , α ) ) − φα ( xi ) ) 2 ∆xi , ( 2 ) which approximates the spatial integral via the sum over all sample points i with corresponding evaluation positions xi ∈ R4 , where ∆xi = ‖xi − xi−1‖ is constant in our case . This corresponds to a regular grid structure , where the loss is accumulated per cell . The central challenge here is to compute reliable gradients for D , which encapsulates a series of highly non-linear deformation steps . We first focus on inferring β , with w = 0 . The gradient of Eq . ( 2 ) with respect to one component of the deformation parameters βj is then given by d dβj L = ∑ i ( − ∂ ∂βj vfin ( xi + vinv ( xi , β ) , β ) · ∇ψ0 ( xi − vfin ( xi , β ) ) ) ( ψ0 ( xi , β ) − φα ( xi ) ) , ( 3 ) where the first term on the sum over i in parentheses represents the gradient of the deformed initial surface . Here we compute the derivative of the full deformation as ∂∂βj vfin ( xi + vinv ( xi , β ) , β ) = u∗i ( xi ) . The offset by vinv on the left hand side indicates that we perform a forward-advection step for this term . Details of this step , and for the full derivation of the gradient are given in Appendix B.1 . After parameter network + Deformation network Reference surface w ↵ A trained NN with this loss functions yields an instance of fp , with which we can infer adjusted deformation weights fp ( α ) = β . Learning to Generate Deformations Based on β , we apply the deformation sequence ui . The goal of our second network , the deformation network fd , is to compute the refinement deformation w. In contrast to the pre-computed ui , fd ( α ) = w now directly depends on α , and can thus capture the interior of the parameter space . Given the initial surface ψ0 deformed by the set of βiui , which we will denote as ψ̃ below , the refinement deformation is applied with a final deformation step as ψ ( x ) = ψ̃ ( x−w ( x , α ) ) . In order to compute the gradient of the deformation loss , we introduce the indicator function χj ( x ) for a single deformation vector wj of w. We found it useful to use a fine discretization for the implicit surfaces , such as ψ , and lower resolutions for w. Hence , each discrete entry wj can act on multiple cells of ψ , which we enumerate with the help of χj . Now the derivative of Eq . ( 1 ) for a fixed β with respect to a single deformation vector wj of w is given by d dwj L = − ∑ i χj ( xi ) ∇ψ̃ ( xi −w ( xi , α ) ) ( ψ̃ ( xi , α ) − φα ( xi ) ) . ( 4 ) The full derivation of this gradient is given in Appendix B.2 . Our approach for deformation learning can be regarded as an extension of STNs Jaderberg et al . ( 2015 ) for dense , weighted fields , and semi-Lagrangian advection methods . The parameter network corresponds to an STN which learns to combine and weight known deformation fields . The deformation network , on the other hand , resembles the thin plate spline STNs , where a network generates an offset for each cell center , which is then used to sample a deformed image or grid . Note that in our case , this sampling process corresponds to the semi-Lagrangian advection of a fluid simulation . Training Details For fp we use a simple structure with two fully connected layers , while fd likewise contains two fully connected layers , followed by two or more four-dimensonal de-convolution layers . All layers use ReLU activation functions . Details can be found in App . B , Fig . 12 . In practice , we also found that a small amount of weight decay andL2 regularization of the generated deformations can help to ensure smoothness . Thus , the loss function of the deformation network , with regularization parameters γ1 and γ2 is Lt = L+ γ1||θ||2 + γ2||w||2 , ( 5 ) where θ denotes the network weights . In addition , regular SDFs can lead to overly large loss values far away from the surface due to linearly increasing distances . Thus , we apply the tanh ( ) function to the SDF values , in order to put more emphasis on the surface region . Special care is required for boundary conditions , i.e , the sides of our domain . Assuming constant values outside of the discretized domain , i.e . ∂ψ ( x ) /∂n = 0 for all x at the domain sides leads to vanishing gradients ∇ψ ( x ) = 0 in App . B , Eq . ( 4 ) . We found this causes artificial minima and maxima in the loss function impeding the training process . Hence , we extrapolate the SDF values with ∂ψ ( x ) /∂n = ±1 in order to retrieve non zero gradients at the domain sides . To train both networks we use stochastic gradient descent with an ADAM optimizer and a learning rate of 10−3 . Training is performed separately for both networks , with typically 1000 steps for fd , 1 Initial surface Reference surface+ Parameter network 0 ↵ and another ca . 9000 steps for fd . Full parameters can be found in App . B , Table 2 . As training data we generate sets of implicit surfaces from liquid simulations with the FLIP method Bridson ( 2015 ) . For our 2D inputs , we use single time steps , while our 4D data concatenates 3D surfaces over time to assemble a space-time surface . Working in conjunction , our two networks capture highly complex behavior of the fluid space-time surface φα over the whole parameter domain . We will evaluate the influence of the networks in isolation and combination in the following .

Summary: This paper introduces a deep learning approach for physical simulation. The approach combines two networks for synthesizing 4D data that represents 3D physical simulations. Here the first network outputs an initial guess, and the second network adds details. The first network utilizes a set of precomputed deformations, while the weights can be set to generate different output shapes. The precomputed deformations are applied in a recurrent manner. The second network is a variant of STN.