Generating Liquid Simulations with Deformation-aware Neural Networks

1 INTRODUCTION . Learning physical functions is an area of growing interest within the research community , with applications ranging from physical priors for computer vision problems Kyriazis & Argyros ( 2013 ) , over robotic control Schenck & Fox ( 2017 ) , to fast approximations for numerical solvers Tompson et al . ( 2017 ) . While the underlying model equations for many physics problems are known , finding solutions is often prohibitively expensive for phenomena on human scales . At the same time , the availability of model equations allows for the creation of reliable ground truth data for training , if enough computational resources can be allocated . Water , and liquids in general , are ubiquitous in our world . At the same time , they represent an especially tough class of physics problems , as the constantly changing boundary conditions at the liquid-gas interface result in a complex space of surface motions and configurations . In this work we present a novel approach to capture parametrized spaces of liquid behavior that is based on space-time deformations . We represent a single 3D input surface over time as a four-dimensional signed-distance function ( SDF ) , which we deform in both space and time with learned deformations to recover the desired physical behavior . To calculate and represent these deformations efficiently , we take a two-stage approach : First , we span the sides of the original parameter region with precomputed deformations , and infer a suitable weighting function . In a second step , we synthesize a dense deformation field for refinement . As both the parameter weighting problem and the deformation synthesis are highly non-linear problems , we demonstrate that neural networks are a particularly suitable solver to robustly find solutions . We will demonstrate that it is possible to incorporate the non-linear effects of weighted deformations into the loss functions of neural networks . In particular , we put emphasis on incorporating the influence of deformation alignment into the loss gradients . This alignment step is necessary to ensure the correct application of multiple consecutive deformations fields . The second stage of our algorithm is a generative model for deformation fields , for which we rely on a known parametrization of the inputs . Thus , in contrast to other generative models which learn to represent unknown parametrization of data sets Radford et al . ( 2016 ) , our models are trained with a known range and dimensionality to parameter range , which serves as input . Once trained , the models can be evaluated very efficiently to synthesize new implicit surface configurations . To demonstrate its performance , we have implemented a proof-of-concept version for mobile devices , and a demo app is available for Android devices in the Google Play store . Our approach generates liquid animations several orders of magnitude faster than a traditional simulator , and achieves effective speed up factors of more than 2000 , as we will outline in Sec . 5 . The central contributions of our work are : • A novel deformation-aware neural network approach to very efficiently represent large collections of space-time surfaces with complex behavior . • We show how to compute suitable loss gradient approximations for the sub-problems of parameter and deformation inference . • In addition we showcase the high performance of our approach with a mobile device implementation that generates liquid simulations interactively . 2 RELATED WORK . Capturing physical behavior with learning has a long history in the field of learning . Early examples targeted minimization problems to determine physical trajectories or contact forces Bhat et al . ( 2002 ) ; Brubaker et al . ( 2009 ) , or plausible physics for interacting objects Kyriazis & Argyros ( 2013 ; 2014 ) . Since initial experiments with physically-based animation and neural networks Grzeszczuk et al . ( 1998 ) , a variety of new deep learning based works have been proposed to learn physical models from videos Battaglia et al . ( 2016 ) ; Chang et al . ( 2016 ) ; Watters et al . ( 2017 ) . Others have targeted this goal in specific settings such as robotic interactions Finn et al . ( 2016 ) , sliding and colliding objects Wu et al . ( 2015 ; 2016 ) , billiard Fragkiadaki et al . ( 2015 ) , or trajectories in height fields Ehrhardt et al . ( 2017 ) . The prediction of forces to infer image-space motions has likewise been targeted Mottaghi et al . ( 2016a ; b ) , while other researchers demonstrated that the stability of stacked objects can be determined from single images Lerer et al . ( 2016 ) ; Li et al . ( 2016 ) . In addition , the unsupervised inference of physical functions poses interesting problems Stewart & Ermon ( 2017 ) . While these methods have achieved impressive results , an important difference to our method is that we omit the projection to images . I.e. , we directly work with three-dimensional data sets over time . In the context of robotic control , physics play a particularly important role , and learning object properties from poking motions Agrawal et al . ( 2016 ) , or interactions with liquids Schenck & Fox ( 2017 ) were targeted in previous work . Learning physical principles was also demonstrated for automated experiments with reinforcement learning Denil et al . ( 2016 ) . Recently , first works have also addressed replacing numerical solvers with trained models for more generic PDEs Farimani et al . ( 2017 ) ; Long et al . ( 2017 ) . In our work we target a more narrow case : that of surfaces deforming based on physical principles . However , thanks to this narrow scope and our specialized loss functions we can generate very complex physical effects in 3D plus time . Our method can be seen as a generative approach representing samples from a chaotic process . In this context , the learned latent-space representation of regular generative models Masci et al . ( 2011 ) ; Rezende et al . ( 2014 ) ; Goodfellow et al . ( 2014 ) ; Radford et al . ( 2016 ) ; Isola et al . ( 2017 ) is replaced by the chosen parametrization of a numerical solver . Our model shares the goal to learn flow physics based on examples with other methods Ladicky et al . ( 2015 ) ; Tompson et al . ( 2017 ) ; Chu & Thuerey ( 2017 ) , but in contrast to these we focus on 4D volumes of physics data , instead of localized windows at single instances of time . Alternative methods have been proposed to work with 4D simulation data Raveendran et al . ( 2014 ) ; Thuerey ( 2017 ) , however , without being able to capture larger spaces of physical behavior . Due to its focus on deformations , our work also shares similarities with methods for optical flow inference and image correspondences Bailer et al . ( 2016 ) ; Dosovitskiy et al . ( 2015 ) ; Ranjan & Black ( 2016 ) ; Ilg et al . ( 2016 ) . A difference to these approaches is that we learn deformations and their weighting in an unsupervised manner , without explicit ground truth data . Thus , our method shares similarities with spatial transformer networks ( STNs ) Jaderberg et al . ( 2015 ) , and unsupervised approaches for optical flow Meister et al . ( 2017 ) . However , instead of aligning two data sets , our method aims for representing larger , parametrized spaces of deformations . 3 LEARNING DEFORMATIONS . We first explain our formulation of the simulation parameter space , which replaces the latent space of other generative models such as autoencoders Masci et al . ( 2011 ) , GANs Radford et al . ( 2016 ) , or auto-regressive models Van Oord et al . ( 2016 ) . Given a Navier-Stokes boundary value problem with a liquid-gas interface , we treat the interface over time as a single space-time surface . We work with a set of these space-time surfaces , defined over a chosen region of the N-dimensional simulation parameters α . We assume the parameters to be normalized , i.e . α ∈ [ 0 , 1 ] N . In practice , α could contain any set of parameters of the simulation , e.g . initial positions , or even physical parameters such as viscosity . We choose implicit functions φ ( α ) ∈ R4 → R to represent specific instances , such that Γ ( α ) = { x ∈ R4 ; φ ( α , x ) = 0 } is the space of surfaces parametrized by α that our generative model should capture . In the following , φ and ψ will denote four-dimensional signed distance functions . We will typically abbreviate φ ( α ) with φα to indicate that φα represents a set of constant reference surfaces . While we will later on discretize all functions and operators on regular Cartesian grids , we will first show the continuous formulation in the following . Deforming Implicit Surfaces Representing the whole set φα is very challenging . Due to bifurcations and discretization artifacts φα represents a process that exhibits noisy and chaotic behavior , an example for which can be found in Fig . 1 . Despite the complex changes in the data , our goal is to find a manageable and reduced representation . Thus , we generate an approximation of φα by deforming a single initial surface in space and time . We apply space-time deformations u : R4 → R4 to the initial surface ψ0 ( x ) . As a single deformation is limited in terms of its representative power to produce different shapes , we make use of N sequential , pre-computed deformations , thus ui with i ∈ [ 1 · · ·N ] , each of which is scaled by a scalar weight parameter βi , whose values are normally between 0 and 1 . This gives us the freedom to choose how much of ui to apply . The initial surface deformed by , e.g. , u1 is given by ψ0 ( x− β1u1 ) . The sequence of deformed surfaces by sequentially applying all pre-computed ui is given by ψi ( x , β ) = ψi−1 ( x − βiui ) . It is crucial to align such sequences of Eulerian deformations with each other . Here we employ the alignment from previous work Thuerey ( 2017 ) , which we briefly summarize in the following , as it influences the gradient calculation below . Each deformation i relies on a certain spatial configuration for the input surface from deformation i− 1 . Thus , when applying ui−1 with a weight βi−1 < 1 , we have to align ui correspondingly . Given the combined deformation vsum ( x , α ) = ∑N i=1 βiu ∗ i ( x ) , with intermediate deformation fields u ∗ i−1 ( x ) = ui−1 ( x − u∗i ( x ) ) , we compute an inversely weighted offset field as vinv ( x , α ) = − ∑N i=1 ( 1 − βi ) u∗i ( x ) . This offset field is used to align the accumulated deformations to compute the final deformation as vfin ( x + vinv ( x , β ) , β ) = vsum ( x , β ) . vfin now represents all weighted deformations βiui merged into a single vector field . Intuitively , this process moves all deformation vectors to the right location for the initial surface ψ0 , such that they can be weighted and accumulated . To achieve the goal of representing the full set of target implicit surfaces φα our goal is to compute two functions : the first one aims for an optimal weighting for each of the deformations in the sequence , i.e . β , while the second function computes a final refinement deformation w after the weighted sequence has been applied . We will employ two neural networks to approximate the two functions , which we will denote as fp , and fd below . Both functions depend only on the simulation parameters space α , i.e. , fp ( α ) = β , and fd ( α ) = w. Splitting the problem into fp and fd is important , as each of the pre-computed deformations weighted by fp only covers a single trajectory in the space of deformed surfaces . In the follow- ing , we employ an optical flow solve from previous work to pre-compute deformations between the inputs Thuerey ( 2017 ) , which deform the input for the extrema of each original parameter dimension αi . E.g. , for a two-dimensional parameter space this yields two deformations along the sides of the unit cube . This first step robustly covers rough , large scale deformations . As a second step , we employ fd , which is realized as a generative CNN , to infer a deformation for refining the solution . Below we will explain the resulting equations for training . The full equations for applying the deformations , and a full derivation of our loss functions can be found in the supplemental materials . To shorten the notation , we introduce the helper function D ( xi , α ) , which yields a deformed set of coordinates in R4 depending on α that incorporates the deformation sequence weighted by fp ( α ) , and a refinement deformation from fd ( α ) . We express the overall goal in terms of minimizing the L2 distance between the deformed and the target implicit surfaces for all possible values in the parameter space α , using β and w as degrees of freedom : argmin β , w L , L = ∫ ‖ψ0 ( D ( xi , α ) ) − φα‖22 dα . ( 1 ) Our work addresses the problem of how to compute weighting of the deformations and on synthesizing the refinement field . The main difficulty lies in the non-linearity of the deformations , which is why we propose a novel method to robustly approximate both functions with NNs : fp will be represented by the parameter network to compute β , and we make use of a deformation network that to generate w. We employ relatively simple neural networks for both functions . Key for training them is encoding the effect of deformations in the loss functions to make the training process aware of their influence . Hence , we will focus on describing the loss functions for both networks and the corresponding discretized gradients in the following . Learning Deformation Weights For training the NNs we propose the following objective function , which measures the similarity of a known reference surface φα and the corresponding , approximated result ψ0 ( x , α ) for a parameter value α . We introduce the numerical equivalent of the continuous L2 loss from Eq . ( 1 ) as L = 1 2 ∑ i ( ψ0 ( D ( xi , α ) ) − φα ( xi ) ) 2 ∆xi , ( 2 ) which approximates the spatial integral via the sum over all sample points i with corresponding evaluation positions xi ∈ R4 , where ∆xi = ‖xi − xi−1‖ is constant in our case . This corresponds to a regular grid structure , where the loss is accumulated per cell . The central challenge here is to compute reliable gradients for D , which encapsulates a series of highly non-linear deformation steps . We first focus on inferring β , with w = 0 . The gradient of Eq . ( 2 ) with respect to one component of the deformation parameters βj is then given by d dβj L = ∑ i ( − ∂ ∂βj vfin ( xi + vinv ( xi , β ) , β ) · ∇ψ0 ( xi − vfin ( xi , β ) ) ) ( ψ0 ( xi , β ) − φα ( xi ) ) , ( 3 ) where the first term on the sum over i in parentheses represents the gradient of the deformed initial surface . Here we compute the derivative of the full deformation as ∂∂βj vfin ( xi + vinv ( xi , β ) , β ) = u∗i ( xi ) . The offset by vinv on the left hand side indicates that we perform a forward-advection step for this term . Details of this step , and for the full derivation of the gradient are given in Appendix B.1 . After parameter network + Deformation network Reference surface w ↵ A trained NN with this loss functions yields an instance of fp , with which we can infer adjusted deformation weights fp ( α ) = β . Learning to Generate Deformations Based on β , we apply the deformation sequence ui . The goal of our second network , the deformation network fd , is to compute the refinement deformation w. In contrast to the pre-computed ui , fd ( α ) = w now directly depends on α , and can thus capture the interior of the parameter space . Given the initial surface ψ0 deformed by the set of βiui , which we will denote as ψ̃ below , the refinement deformation is applied with a final deformation step as ψ ( x ) = ψ̃ ( x−w ( x , α ) ) . In order to compute the gradient of the deformation loss , we introduce the indicator function χj ( x ) for a single deformation vector wj of w. We found it useful to use a fine discretization for the implicit surfaces , such as ψ , and lower resolutions for w. Hence , each discrete entry wj can act on multiple cells of ψ , which we enumerate with the help of χj . Now the derivative of Eq . ( 1 ) for a fixed β with respect to a single deformation vector wj of w is given by d dwj L = − ∑ i χj ( xi ) ∇ψ̃ ( xi −w ( xi , α ) ) ( ψ̃ ( xi , α ) − φα ( xi ) ) . ( 4 ) The full derivation of this gradient is given in Appendix B.2 . Our approach for deformation learning can be regarded as an extension of STNs Jaderberg et al . ( 2015 ) for dense , weighted fields , and semi-Lagrangian advection methods . The parameter network corresponds to an STN which learns to combine and weight known deformation fields . The deformation network , on the other hand , resembles the thin plate spline STNs , where a network generates an offset for each cell center , which is then used to sample a deformed image or grid . Note that in our case , this sampling process corresponds to the semi-Lagrangian advection of a fluid simulation . Training Details For fp we use a simple structure with two fully connected layers , while fd likewise contains two fully connected layers , followed by two or more four-dimensonal de-convolution layers . All layers use ReLU activation functions . Details can be found in App . B , Fig . 12 . In practice , we also found that a small amount of weight decay andL2 regularization of the generated deformations can help to ensure smoothness . Thus , the loss function of the deformation network , with regularization parameters γ1 and γ2 is Lt = L+ γ1||θ||2 + γ2||w||2 , ( 5 ) where θ denotes the network weights . In addition , regular SDFs can lead to overly large loss values far away from the surface due to linearly increasing distances . Thus , we apply the tanh ( ) function to the SDF values , in order to put more emphasis on the surface region . Special care is required for boundary conditions , i.e , the sides of our domain . Assuming constant values outside of the discretized domain , i.e . ∂ψ ( x ) /∂n = 0 for all x at the domain sides leads to vanishing gradients ∇ψ ( x ) = 0 in App . B , Eq . ( 4 ) . We found this causes artificial minima and maxima in the loss function impeding the training process . Hence , we extrapolate the SDF values with ∂ψ ( x ) /∂n = ±1 in order to retrieve non zero gradients at the domain sides . To train both networks we use stochastic gradient descent with an ADAM optimizer and a learning rate of 10−3 . Training is performed separately for both networks , with typically 1000 steps for fd , 1 Initial surface Reference surface+ Parameter network 0 ↵ and another ca . 9000 steps for fd . Full parameters can be found in App . B , Table 2 . As training data we generate sets of implicit surfaces from liquid simulations with the FLIP method Bridson ( 2015 ) . For our 2D inputs , we use single time steps , while our 4D data concatenates 3D surfaces over time to assemble a space-time surface . Working in conjunction , our two networks capture highly complex behavior of the fluid space-time surface φα over the whole parameter domain . We will evaluate the influence of the networks in isolation and combination in the following .

Summary: This is an application paper on dense volumetric synthesis of liquids and smoke. Given densely registered 4D implicit surfaces (volumes over time) for a structured scene, a neural-network based model is used to interpolate simulations for novel scene conditions (e.g. position and size of dropped water ball). The interpolation model composes two components -- given these conditions, it first regresses weights combining a set of precomputed deformation fields, and then a second model regresses dense volumetric deformation corrections -- these are helpful as some events are not easily modeled with a set of basis deformations.