Constrained Physical-Statistics Models for Dynamical System Identification and Prediction

1 INTRODUCTION . Dynamical systems prediction and identification find crucial applications ranging from medicine and the study of tumors ( Hanahan & Weinberg , 2011 ; Lu & Fei , 2014 ) to oceanic and climate forecasting ( Oreskes et al. , 1994 ; Caers , 2011 ) . The modeling of such systems traditionally rely on ordinary or partial differential equations ( ODE/PDE ) ( Madec , 2008 ; Marti et al. , 2010 ) , and their resolution via numerical solvers and data assimilation ( Ghil & Malanotte-Rizzoli , 1991 ) . In real world applications , two main pitfalls occur : first the dynamics may only be partially known and thus do not fully represent the studied phenomena ( Rio & Santoleri , 2018 ) ; second , the system state may only be partially observed as in ocean models ( Gaultier et al. , 2013 ) . Machine learning ( ML ) has become a complementary approach to traditional physics based models ( denoted MB for model based ) ( Reichstein et al. , 2019 ; Dueben & Bauer , 2018 ) . Both offer advantages : whereas MB approaches generalize and extrapolate better , ML high expressivity approaches benefit from the ongoing explosive growth of available data such as satellite observations , with reduced costs compared to classical data assimilation . In that perspective , recent lines of work tackle the learning of hybrid models relying on prior physical knowledge and machine learning ( Yin et al. , 2021 ; Mehta et al. , 2020 ) . Efficiently learning such decompositions actually means solving two different tasks : system identification , i.e . estimating the parameters of the physical model , and prediction , i.e . recovering the trajectories associated to the dynamics . Both are essential for hybrid MB/ML models of dynamical systems . Whereas prediction aims at robust extrapolation , identification accounts for physical interpretability of the MB/ML model . While solving both problems using model-based formulation admits well-known numerical solutions , for example using the adjoint method ( Le Dimet & Talagrand , 1986 ; Courtier et al. , 1994 ) , the combination of physical models and deep learning is still an open area of research . In this context , ML applications mainly focus on the prediction task , at the expense of the system identification : Willard et al . ( 2020 ) underlines the lack of generalizability of black-box ML models and their inability to produce physically sound results . Indeed , Ayed et al . ( 2020 ) show that without any prior knowledge , the recovered estimates of a dynamical system states are not physically plausible despite accurate predictions . Moreover , as noted by Yin et al . ( 2021 ) , learning a linear MB/ML decompo- sition with the sole supervision on the system trajectories is ill-posed and admits an infinite number of decompositions . Such observations highlight the need to incorporate physically motivated constraints in the learning of hybrid models , e.g . through regularization penalties , and several works propose additional constraints to guide the model towards physical solutions ( Jia et al. , 2019 ; Yin et al. , 2021 ; Linial et al. , 2021 ) . Finally , to complete prior dynamical knowledge with a data-driven component and ensure interpretability of the decomposition , we work out a principled framework that generalizes previous attempts in the regularization of hybrid models . Our contributions are : • In section 3.1 , we introduce a novel way to recover well-posedness and interpretability in the learning of hybrid MB/ML models through the control of an upper bound . We further extend our framework to incorporate auxiliary data when available to handle complex realworld problems . • In section 3.2 , we propose a novel alternate-optimization algorithm to learn hybrid models . • In section 3.3 , we provide an analysis of the convergence of the proposed algorithm on a simplified case and experimentally evidence the soundness of our approach on more complex settings of increasing difficulty including challenging real world problems ( section 4 ) . 2 BACKGROUND AND PROBLEM SETUP . We consider a dynamical system with state at time t denoted Zt = Z ( t ) . Zt might be fully or only partially observed : we write Zt = ( Xt , Yt ) , where Xt is the observed component and Yt the unobserved one . The evolution of Z is governed by a differential equation with dynamics : dZt dt = d dt ( Xt Yt ) = ( fX ( Zt ) fY ( Zt ) ) ( 1 ) The objective is to predict trajectories of X , i.e . to model the evolution of the observable part following dXtdt = fX ( Zt ) . For simplicity , we omit the index X in fX and write f ( . ) ∆ = fX ( . ) . Dynamical Hypothesis We assume partial knowledge of the dynamics of the observed Xt : dXt dt = f ( Zt ) = fk ( Zt ) + fu ( Zt ) ( 2 ) where fk ∈ Hk is a known operator with unknown parameters θ∗ , and fu ∈ Hu is the unknown residual dynamics . Hk and Hu denote function spaces . Note that the additive hypothesis in eq . ( 2 ) is not restrictive ( see discussion in appendix B ) . Learning Problem Our objective is to approximate f with a function h learned from the observed data . According to eq . ( 2 ) , we assume h = hk +hu . hk ∈ Hk belongs to the same hypothesis space as fk , i.e . it has the same parametric form . Its parameters are denoted θk . Note that hk ( . , θ∗ ) = fk . hu ∈ Hu is represented by a free form functional with parameters θu , e.g . a neural network . The learning problem is then to estimate from data the parameters of hk so that they match the true physical ones and those of hu so as to approximate at best the unknown dynamics f . In this regard , an intuitive training objective is to minimize a distance d between h = hk + hu and f : d ( h , f ) = EZ∼pZ‖h ( Z ) − f ( Z ) ‖2 , ( 3 ) where pZ is the distribution of the state values Z that accounts for varying initial states . Each Z then defines a training sample . Minimizing eq . ( 3 ) with h = hk + hu enables to predict accurate trajectories but may have an infinite number of solutions . For instance , hu may bypass the physical hypothesis hk . Thus , interpretability is not guaranteed . We develop in the following section a method to overcome this ill-posedness . 3 METHOD . In hybrid modeling , two criteria are essentials : 1. identifiability , i.e . the estimated parameters of hk should correspond to the true physical ones ; 2. prediction power , i.e . the statistical component hu should complete hk so that h = hk + hu performs accurate prediction over the system states . To control the contribution of each term hk and hu , we work upper bounds out of eq . ( 3 ) ( section 3.1 ) . We then propose to minimize d ( h , f ) while constraining the upper bounds , which provide us with a well-posed learning framework ( section 3.2 ) . Besides , we show that several previous works that introduced constrained optimization to solve related problems are specific cases of our formulation ( Yin et al. , 2021 ; Jia et al. , 2019 ; Linial et al. , 2021 ) . Finally , we introduce an alternate optimization algorithm which convergence is shown in section 3.3 for a linear approximation of f . 3.1 STRUCTURAL CONSTRAINTS FOR DYNAMICAL SYSTEMS . To ensure identifiability , we derive regularizations on hk and hu flowing from the control of an upper bound of d ( h , f ) . In particular , to minimize d ( hk , fk ) would enable us to accurately interpret hk as the true fk , and thus hu as the residual dynamics fu . However , since we do not access the parameters of fk , computing d ( hk , fk ) is not tractable . We then consider two possible situations . In the first one , the only available information on the physical system is the parametric form of fk ( or equivalently of hk ) , training thus only relies on observed trajectories ( section 3.1.1 ) . In the second one , we consider that auxiliary information about fk is available and will be used to minimize the distance between hk and fk ( section 3.1.2 ) . While the first setting is the more general , the physical prior it relies on is often insufficient to effectively handle real world situations . The second setting makes use of more informative priors and better corresponds to real cases as will be shown in the experimental section ( see section 4.2 ) . 3.1.1 CONTROLLING THE ML COMPONENT AND THE MB HYPOTHESIS . We propose a general approach to constrain the learning of hybrid models when one solely access the functional form of hk . In this case , to make hk accountable in our observed phenomena , a solution is to minimize d ( hk , f ) . Following the triangle inequality we link up both errors d ( h , f ) and d ( hk , f ) ( computations available in appendix C.1 ) : d ( h , f ) ≤ d ( h , hk ) + d ( hk , f ) = d ( hu , 0 ) + d ( hk , f ) ( 4 ) We want the physical-statistical model h = hk + hu to provide high quality forecasts . Minimizing the sole upper bound does not ensure such aim , as hu is only penalized through d ( hu , 0 ) and is not optimized to contribute to predictions . We thus propose to minimize d ( h , f ) while controlling both d ( hu , 0 ) and d ( hk , f ) . Such a control of the upper bound of eq . ( 4 ) amounts to balancing the contribution of the ML and the MB components . This will be formally introduced in section 3.2 . Link to the Literature The least action principle on the ML component i.e . constraining d ( hu , 0 ) is invoked for a geometric argument in ( Yin et al. , 2021 ) , and appears as a co-product of the introduction of d ( hk , f ) in eq . ( 4 ) . Optimizing d ( hk , f ) to match the physical model with observations is investigated in ( Forssell & Lindskog , 1997 ) . The general approach of eq . ( 4 ) allows us to perform prediction ( via h ) and system identification ( via hk ) on simple problems ( see section 4.1 ) . The learning of real-world complex dynamics , via data-driven hybrid models , often fails at yielding a physically sound estimation , as illustrated in section 4.2 . This suggests that learning complex dynamics requires additional information . In many real-world cases , auxiliary information is available in the form of measurements providing complementary information on fk . Indeed , a common issue in physics is to infer an unobserved variable of interest ( in our case fk parameters θ ? ) from indirect or noisy measurements that we refer to as proxy data . For instance , one can access a physical quantity but only at a coarse resolution , as in ( Um et al. , 2020 ; Belbute-Peres et al. , 2020 ) and in the real world example detailed in section 4.2 . We show in the next subsection how to incorporate such an information in order to approximate d ( hk , fk ) .

Summary: The paper proposes a method for hybrid model-based/machine-leanring learning, wherein a model is decomposed into an interpretable parametric prior and a residual (typically represented by a neural net). The paper demonstrates that prediction error minimization does not accurately identify the parametric component and proposes an alternating optimization method that augments prediction error loss with component-specific losses. The paper also introduces a variant where auxiliary data can be used to define the loss.  Theoretical guarantees are provided where auxiliary losses are given as upper-bound constraints and exact minimization is performed. Also, uniqueness and convergence guarantees are provided for the linear approximation case using a more practical algorithm.  Experiments show the efficacy of the method in linear and non-linear settings, in both identification and prediction.