Learning Partially Observed PDE Dynamics with Neural Networks

Spatio-Temporal processes bear a central importance in many applied scientific fields . Generally , differential equations are used to describe these processes . In this work , we address the problem of learning spatio-temporal dynamics with neural networks when only partial information on the system ’ s state is available . Taking inspiration from the dynamical system approach , we outline a general framework in which complex dynamics generated by families of differential equations can be learned in a principled way . Two models are derived from this framework . We demonstrate how they can be applied in practice by considering the problem of forecasting fluid flows . We show how the underlying equations fit into our formalism and evaluate our method by comparing with standard baselines . 1 INTRODUCTION . We consider the problem of learning complex dynamics typical of physical systems using deep learning models . Modeling these dynamics is central to almost any problem in applied physics . The classical approach to this problem is model based : the dynamics of the physical system is usually formulated using Partial Differential Equations ( PDEs ) describing the evolution of state variables . PDEs are then solved through some clever numerical scheme ( as in Meerschaert & Tadjeran ( 2006 ) for example ) . This methodology is used in many application domains like climate science , financial engineering , etc .. Although it has been extensively developed , several challenges remain open . Modeling physical phenomena accurately can be tedious , if not unfeasible : the relations between state variables are often complex and , even when those are known , specifying a functional form for the dynamics can be difficult ( Lions et al . ( 1992 ) , Peebles ( 1980 ) ) . Moreover , designing accurate , robust and computationally feasible numerical schemes for PDEs is usually a difficult problem for which current solutions rely on extensive domain specific background . The availability of huge data quantities gathered from different types of sensors or simulations is a strong incentive to develop data driven approaches , as an alternative or complement to standard model based methods . For many real world problems , the underlying physics of the phenomenon is not always known in its entirety and data-driven methods , inspired by our current physical knowledge , appear as a particularly promising alternative to model based approaches , see Bezenac et al . ( 2018 ) , Ling et al . ( 2016 ) , Stewart & Ermon ( 2016 ) or Rudy et al . ( 2017b ) for some examples . On the other hand , mature fields such as differential equations could provide machine learning with useful insights and intuitions . We advocate here a ML approach incorporating structural priors inspired from such a well developed domain . We consider the problem of learning dynamical systems where available observations only provide partial information on the system state and dynamics which is a typical scenario in real world applications , be it in weather forecasting , financial engineering , or even video prediction . The objective is to learn from such observations the underlying system dynamics so as to forecast future observations . We focus on data generated by complex highly non-linear differential equations and analyze whether these dynamics can be learned by Deep Learning models . This context is particularly interesting since simulations relying on PDEs are essential in many applied disciplines and since this allows us to perform controlled experiments providing access to all the variables of the problem . Our contributions are the following : • We propose a general framework for modeling partially observed dynamical systems governed by PDEs with neural networks . This framework is inspired by the way physicists incorporate observations into their forecasting pipeline . It should be flexible enough to accommodate a large class of modeling problems other than forecasting observations . • From this framework we derive two instances of training ( PT and JT ) , with two different levels of prior injection , and two methods for multi-step forecasting ( SSE and MSRE ) . • As a case study , we consider two PDEs describing the evolution of fluid flow , namely the Euler and Navier-Stokes equations , and show how they can be formulated within this formalism . We study the performance and behaviour of our models on the problem of forecasting fluid flow using two common baseline models and focusing on the interplay between injection of prior information and performance . We introduce our framework in section 2 , detail the framework and its instances in section 3 and show how PDEs fit within this framework in section 4 . Experimental results and analysis are described in section 5 and recent related work is reviewed in section 6 . 2 BACKGROUND AND PROBLEM SETTING . 2.1 LEARNING FROM PARTIALLY OBSERVED DATA . A dynamical system in our context can be broadly defined as a function X with values in Ω which obeys a set of differential equations . We consider spatio-temporal dynamics for which X can be written as a function of ( t , x ) ∈ R×Rd where t and x are respectively the time and space variables . The spatial vector-valued functionXt contains the quantities of interest describing a studied physical system at time t and gives the evolution in time and space of the system ’ s state . More precisely , we will be interested in differential equations which can be written as : dXt dt = F ( Xt , DxXt ) ( 1 ) where Xt denotes the state vector at time t , DxXt the spatial derivatives of Xt and F is a function with values in Ω ( common instances for F involve the gradient operator ∇X , the Laplacian ∇2X , ... ) . By specifying a particular structure on F , one can describe the dynamics of many different classes of systems : an important example is that of the Navier-Stokes equations from fluid dynamics which bear a central importance in many applied sciences and which will be our application example . 2.1.1 NEURAL IMPLEMENTATION OF NUMERICAL SCHEMES . Solving differential equations usually requires using a numerical scheme in order to approximate the solution . Finite difference methods for example make use of discrete approximations of the derivatives . Let us suppose available a sequence of states { ( Xt ) t } , each Xt being a function of the space variable x , Xt andXt+δt being separated by a short time interval δt for all t. A straightforward discretization of equation 1 is the forward Euler scheme : Xt+δt −Xt ≈ δt · F ( Xt , DxXt ) where δt is the discretization timestep . Using a parameterized neural network rθ , we can train it in order to fit the relation above : Xt+δt ≈ Xt + δt · rθ ( Xt , DxXt ) This is reminiscent of the skip connection structures used in the residual block of ResNet or in some RNN models . Linear multi-step methods extend this idea by performing several integration steps . A simple multi-step extension of the Euler scheme could then be : Xt+kδt ≈ Xt + δt k−1∑ l=0 rθ ( Xt+lδt , DxXt+lδt ) This sequential process could be implemented by a ResNet architecture so that the latter could be used as a parametric model for approximating solutions of differential equations . Moreover , spatial derivatives of Xt at a given point x can classically be expressed with finite difference schemes over the values of the function at neighbouring points Vx : DXt ( x ) = K ( { Xt ( x′ ) | x′ ∈ Vx } ) where K is a linear function . Thus , we can express each component of DxXt as a convolution over Xt , motivating the use of convolutional networks . Thus , given a sufficiently large training set of state sequences and supposing that we are able to learn their dynamics , we could , just by knowing the initial state of our system , use the learned operator and forecast its behaviour for any time horizon . One could constrain the structure ofK to express a particular set of derivatives . This would be a way to add additional structure and knowledge into the forecasting operator . One could also adopt a more hybrid approach such as in Long et al . ( 2018 ) . For the examples we studied in this paper , we use standard convolutional ResNets as we wanted to disentangle as much as possible the contributions of the elements we introduce , particularly when injecting priors into the estimator . However , it would be interesting in future research to analyze the effects of more structural constraints added into the forecasting operator . 2.1.2 LINKING STATES TO PARTIAL OBSERVATIONS . In realistic settings , the state is generally only partially observed e.g. , when studying the ocean ’ s circulation , variables contained in the system ’ s state such as temperature or salinity are observable while others such as velocity or pressure are not . In other words , the measured data is only a projection of the complete state X . We model this measurement process with a fixed operator H linking the system ’ s state Xt to the corresponding observation Yt : Yt = H ( Xt ) In the following , H is supposed to be known , fixed and differentiable . This setting is the one which we focus on in this paper . 3 FRAMEWORK . To summarize , observations Yt are supposed to be generated by a state space model with variable Xt and the main objective is the prediction of future observations Yt+k for a certain horizon k. The general form of the model for a prediction at horizon 1 is then : initial state X0 Xt+1 = g ( Xt ) Yt = H ( Xt ) ( 2 ) We make the simplifying hypothesis that the information loss through the projection H is so that a state Xt can be deterministically reconstructed using a long enough sequence . In other words , there exists an integer k and an operator E such that : Xt = E ( Yt−k+1 , ... , Yt ) ( 3 ) For more details on this assumption , refer to appendix C. For now , let us consider only one step predictions : given a sequence of initial observations ( Y1 , ... , Yk ) , we want to find the best estimation Ŷk+1 for the observation Yk+1 , where observations Y are supposed to be generated an underlying process X . We propose a framework where predicting Ŷk+1 is decomposed into three steps : • Prediction of the current state : Given past observations , we use a learnable operator eω parameterized by ω to produce a ( complete ) state vector X̃t . • Prediction of the future state : X̃t is given as input to a second learnable operator fθ parameterized by θ which produces the next state X̂t+1 . • Projection of the future observation : X̂t+1 is mapped onto the space of observations using operatorH to predict future observation Ŷt+1 . This is summarized in the following system : X̃t = eω ( Yt−k+1 , ... , Yt ) X̂t+1 = fθ ( X̃t ) Ŷt+1 = H ( X̃t+1 ) ( 4 ) The learning problems amounts to optimizing the following loss : min θ , ω E ( Y1 , ... , Yk+1 ) ∈Data [ d ( H ( fθ ( eω ( Y1 , ... , Yk ) ) ) , Yk+1 ) ] ( 5 ) where d is the loss metric and e and f are functions belonging to appropriate parametric families . Time lag parameter k can be set up by cross-validation . As discussed in section 2 , a natural choice for f is a convolutional residual network modeling the system ’ s dynamics whereas , for e , designing the right family of functions will depend on the structure of the studied data .

Summary: I very much like the aim of this work. This is a problem of interest to a wide community, which as far as I'm aware hasn't yet had much focus from the deep learning community. However, perhaps in part because of this, the paper reads as naive in places. Pages 1-4 are all background saying nothing new, but ignoring the effort made on this problem by other communities. There has been some work done o this problem within statistics, and within the Gaussian process community, to which no reference is made at all by the paper.