Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting

1 INTRODUCTION . Modeling and forecasting complex dynamical systems is a major challenge in domains such as environment and climate ( Rolnick et al. , 2019 ) , health science ( Choi et al. , 2016 ) , and in many industrial applications ( Toubeau et al. , 2018 ) . Model Based ( MB ) approaches typically rely on partial or ordinary differential equations ( PDE/ODE ) and stem from a deep understanding of the underlying physical phenomena . Machine learning ( ML ) and deep learning methods are more prior agnostic yet have become state-of-the-art for several spatio-temporal prediction tasks ( Shi et al. , 2015 ; Wang et al. , 2018 ; Oreshkin et al. , 2020 ; Donà et al. , 2020 ) , and connections have been drawn between deep architectures and numerical ODE solvers , e.g . neural ODEs ( Chen et al. , 2018 ; Ayed et al. , 2019b ) . However , modeling complex physical dynamics is still beyond the scope of pure ML methods , which often can not properly extrapolate to new conditions as MB approaches do . Combining the MB and ML paradigms is an emerging trend to develop the interplay between the two paradigms . For example , Brunton et al . ( 2016 ) ; Long et al . ( 2018b ) learn the explicit form of PDEs directly from data , Raissi et al . ( 2019 ) ; Sirignano & Spiliopoulos ( 2018 ) use NNs as implicit methods for solving PDEs , Seo et al . ( 2020 ) learn spatial differences with a graph network , Ummenhofer et al . ( 2020 ) introduce continuous convolutions for fluid simulations , de Bézenac et al . ( 2018 ) learn the ∗Equal contribution , authors sorted by reverse alphabetical order . velocity field of an advection-diffusion system , Greydanus et al . ( 2019 ) ; Chen et al . ( 2020 ) enforce conservation laws in the network architecture or in the loss function . The large majority of aforementioned MB/ML hybrid approaches assume that the physical model adequately describes the observed dynamics . This assumption is , however , commonly violated in practice . This may be due to various factors , e.g . idealized assumptions and difficulty to explain processes from first principles ( Gentine et al. , 2018 ) , computational constraints prescribing a fine grain modeling of the system ( Ayed et al. , 2019a ) , unknown external factors , forces and sources which are present ( Large & Yeager , 2004 ) . In this paper , we aim at leveraging prior dynamical ODE/PDE knowledge in situations where this physical model is incomplete , i.e . unable to represent the whole complexity of observed data . To handle this case , we introduce a principled learning framework to Augment incomplete PHYsical models for ideNtIfying and forecasTing complex dYnamics ( APHYNITY ) . The rationale of APHYNITY , illustrated in Figure 1 on the pendulum problem , is to augment the physical model when—and only when—it falls short . Designing a general method for combining MB and ML approaches is still a widely open problem , and a clear problem formulation for the latter is lacking ( Reichstein et al. , 2019 ) . Our contributions towards these goals are the following : • We introduce a simple yet principled framework for combining both approaches . We decompose the data into a physical and a data-driven term such that the data-driven component only models information that can not be captured by the physical model . We provide existence and uniqueness guarantees ( Section 3.1 ) for the decomposition given mild conditions , and show that this formulation ensures interpretability and benefits generalization . • We propose a trajectory-based training formulation ( Section 3.2 ) along with an adaptive optimization scheme ( Section 3.3 ) enabling end-to-end learning for both physical and deep learning components . This allows APHYNITY to automatically adjust the complexity of the neural network to different approximation levels of the physical model , paving the way to flexible learned hybrid models . • We demonstrate the generality of the approach on three use cases ( reaction-diffusion , wave equations and the pendulum ) representative of different PDE families ( parabolic , hyperbolic ) , having a wide spectrum of application domains , e.g . acoustics , electromagnetism , chemistry , biology , physics ( Section 4 ) . We show that APHYNITY is able to achieve performances close to complete physical models by augmenting incomplete ones , both in terms of forecasting accuracy and physical parameter identification . Moreover , APHYNITY can also be successfully extended to the partially observable setting ( see discussion in Section 5 ) . 2 RELATED WORK . Correction in data assimilation Prediction under approximate physical models has been tackled by traditional statistical calibration techniques , which often rely on Bayesian methods ( Pernot & Cailliez , 2017 ) . Data assimilation techniques , e.g . the Kalman filter ( Kalman , 1960 ; Becker et al. , 2019 ) , 4D-var ( Courtier et al. , 1994 ) , prediction errors are modeled probabilistically and a correction using observed data is applied after each prediction step . Similar residual correction procedures are commonly used in robotics and optimal control ( Chen , 2004 ; Li et al. , 2014 ) . However , these sequential ( two-stage ) procedures prevent the cooperation between prediction and correction . Besides , in model-based reinforcement learning , model deficiencies are typically handled by considering only short-term rollouts ( Janner et al. , 2019 ) or by model predictive control ( Nagabandi et al. , 2018 ) . The originality of APHYNITY is to leverage model-based prior knowledge by augmenting it with neurally parametrized dynamics . It does so while ensuring optimal cooperation between the prior model and the augmentation . Augmented physical models Combining physical models with machine learning ( gray-box or hybrid modeling ) was first explored from the 1990 ’ s : Psichogios & Ungar ( 1992 ) ; Thompson & Kramer ( 1994 ) ; Rico-Martinez et al . ( 1994 ) use neural networks to predict the unknown parameters of physical models . The challenge of proper MB/ML cooperation was already raised as a limitation of gray-box approaches but not addressed . Moreover these methods were evaluated on specific applications with a residual targeted to the form of the equation . In the last few years , there has been a renewed interest in deep hybrid models bridging data assimilation techniques and machine learning to identify complex PDE parameters using cautiously constrained forward model ( Long et al. , 2018b ; de Bézenac et al. , 2018 ) , as discussed in introduction . Recently , some approaches have specifically targetted the MB/ML cooperation . HybridNet ( Long et al. , 2018a ) and PhICNet ( Saha et al. , 2020 ) both use data-driven networks to learn additive perturbations or source terms to a given PDE . The former considers the favorable context where the perturbations can be accessed , and the latter the special case of additive noise on the input . Wang et al . ( 2019 ) ; Mehta et al . ( 2020 ) propose several empirical fusion strategies with deep neural networks but lack theoretical groundings . PhyDNet ( Le Guen & Thome , 2020 ) tackles augmentation in partially-observed settings , but with specific recurrent architectures dedicated to video prediction . Crucially , all the aforementioned approaches do not address the issues of uniqueness of the decomposition or of proper cooperation for correct parameter identification . Besides , we found experimentally that this vanilla cooperation is inferior to the APHYNITY learning scheme in terms of forecasting and parameter identification performances ( see experiments in Section 4.2 ) . 3 THE APHYNITY MODEL . In the following , we study dynamics driven by an equation of the form : dXt dt = F ( Xt ) ( 1 ) defined over a finite time interval [ 0 , T ] , where the state X is either vector-valued , i.e . we have Xt ∈ Rd for every t , ( pendulum equations in Section 4 ) , or Xt is a d-dimensional vector field over a spatial domain Ω ⊂ Rk , with k ∈ { 2 , 3 } , i.e . Xt ( x ) ∈ Rd for every ( t , x ) ∈ [ 0 , T ] × Ω ( reaction-diffusion and wave equations in Section 4 ) . We suppose that we have access to a set of observed trajectories D = { X· : [ 0 , T ] → A | ∀t ∈ [ 0 , T ] , dXt/dt = F ( Xt ) } , where A is the set of X values ( either Rd or vector field ) . In our case , the unknown F has A as domain and we only assume that F ∈ F , with ( F , ‖ · ‖ ) a normed vector space . 3.1 DECOMPOSING DYNAMICS INTO PHYSICAL AND AUGMENTED TERMS . As introduced in Section 1 , we consider the common situation where incomplete information is available on the dynamics , under the form of a family of ODEs or PDEs characterized by their temporal evolution Fp ∈ Fp ⊂ F . The APHYNITY framework leverages the knowledge of Fp while mitigating the approximations induced by this simplified model through the combination of physical and data-driven components . F being a vector space , we can write : F = Fp + Fa where Fp ∈ Fp encodes the incomplete physical knowledge and Fa ∈ F is the data-driven augmentation term complementing Fp . The incomplete physical prior is supposed to belong to a known family , but the physical parameters ( e.g . propagation speed for the wave equation ) are unknown and need to be estimated from data . Both Fp and Fa parameters are estimated by fitting the trajectories from D. The decomposition F = Fp + Fa is in general not unique . For example , all the dynamics could be captured by the Fa component . This decomposition is thus ill-defined , which hampers the interpretability and the extrapolation abilities of the model . In other words , one wants the estimated parameters of Fp to be as close as possible to the true parameter values of the physical model and Fa to play only a complementary role w.r.t Fp , so as to model only the information that can not be captured by the physical prior . For example , when F ∈ Fp , the data can be fully described by the physical model , and in this case it is sensible to desire Fa to be nullified ; this is of central importance in a setting where one wishes to identify physical quantities , and for the model to generalize and extrapolate to new conditions . In a more general setting where the physical model is incomplete , the action of Fa on the dynamics , as measured through its norm , should be as small as possible . This general idea is embedded in the following optimization problem : min Fp∈Fp , Fa∈F ‖Fa‖ subject to ∀X ∈ D , ∀t , dXt dt = ( Fp + Fa ) ( Xt ) ( 2 ) The originality of APHYNITY is to leverage model-based prior knowledge by augmenting it with neurally parametrized dynamics . It does so while ensuring optimal cooperation between the prior model and the augmentation . A first key question is whether the minimum in Eq . ( 2 ) is indeed well-defined , in other words whether there exists indeed a decomposition with a minimal norm Fa . The answer actually depends on the geometry of Fp , and is formulated in the following proposition proven in Appendix B : Proposition 1 ( Existence of a minimizing pair ) . If Fp is a proximinal set1 , there exists a decomposition minimizing Eq . ( 2 ) . Proximinality is a mild condition which , as shown through the proof of the proposition , can not be weakened . It is a property verified by any boundedly compact set . In particular , it is true for closed subsets of finite dimensional spaces . However , if only existence is guaranteed , while forecasts would be expected to be accurate , non-uniqueness of the decomposition would hamper the interpretability of Fp and this would mean that the identified physical parameters are not uniquely determined . It is then natural to ask under which conditions solving problem Eq . ( 2 ) leads to a unique decomposition into a physical and a data-driven component . The following result provides guarantees on the existence and uniqueness of the decomposition under mild conditions . The proof is given in Appendix B : Proposition 2 ( Uniqueness of the minimizing pair ) . If Fp is a Chebyshev set1 , Eq . ( 2 ) admits a unique minimizer . The Fp in this minimizer pair is the metric projection of the unknown F onto Fp . The Chebyshev assumption condition is strictly stronger than proximinality but is still quite mild and necessary . Indeed , in practice , many sets of interest are Chebyshev , including all closed convex spaces in strict normed spaces and , if F = L2 , Fp can be any closed convex set , including all finite dimensional subspaces . In particular , all examples considered in the experiments are Chebyshev sets . Propositions 1 and 2 provide , under mild conditions , the theoretical guarantees for the APHYNITY formulation to infer the correct MB/ML decomposition , thus enabling both recovering the proper physical parameters and accurate forecasting .

Summary: This paper outlines a method for forecasting and parameter estimation when you have a partial physics model (possibly with unknown parameters) and time series data. This is a hybrid approach where the data-driven (deep learning) approach only learns the parts not accounted for by the physical model. A key feature is being able to decompose the problem in such a way that the data-driven model only models what cannot be captured by the physical model. The parameters of these two models must be fit jointly so that the physical model's parameters are more correct. They prove existence and uniqueness for this decomposition.