D-CODE: Discovering Closed-form ODEs from Observed Trajectories

1 INTRODUCTION . An ordinary differential equation ( ODE ) links the state of a continuous-time dynamical system x ( t ) to its time derivative ẋ ( t ) via a function f , e.g . ẋ ( t ) = f ( x ( t ) , t ) . The ODE is closed-form when f has a concise and analytical expression ( e.g . f ( x , t ) = −t log ( x ) ) . Closed-form ODEs are ubiquitous in science and engineering . The popularity is partly due to their transparency and interpretability to human experts ( Petersen et al. , 2019 ) . Their concise functional form also facilitates the analysis of many key properties of the dynamical system , e.g . asymptotic stability ( Lyapunov , 1992 ) . However , the discovery of closed-form ODEs has been laborious and time-consuming as it heavily relies on human experts ( Simmons , 1972 ) . The goal of this work is to discover closed-form ODEs in an automated and data-driven way . We envision that such a tool would accelerate the quantitative modeling of dynamical systems , with applications ranging from discovering the kinetics of biochemical reactions to modeling tumor growth ( Schuster , 2019 ; Geng et al. , 2017 ) . Symbolic regression is an established approach to discover a closed-form function a = f ( b ) for predicting the label a from the feature b . It operates in the supervised learning setting and requires a dataset of label-feature pairs for training , i.e . D = { ai , bi } Ni=1 ( Schmidt & Lipson , 2009 ) . However , symbolic regression is not directly applicable to ODEs because we usually do not observe the time derivative ẋ ( t ) , i.e . the labels for regression are not available in the data . In contrast , we only have access to the measurements of state y ( t ) = x ( t ) + at some discrete time points and with noise . For instance , the datasets for tumor growth modeling typically contain tumor volumes measured at different clinical visits with substantial noise ( Wilkerson et al. , 2017 ; Mazaheri et al. , 2009 ) . We refer to this problem as the equation-data mismatch . A simple way to resolve this mismatch is to estimate ẋ ( t ) from data ( e.g . by numerical differentiation ( Bickley , 1941 ) ) , and use the estimated derivatives as labels for regression ( Gaucel et al. , 2014 ) . However , it is very challenging to recover the time derivative under high measurement noise or infrequent sampling ( Cullum , 1971 ) , and symbolic regression tends to perform poorly with inaccurate labels ( Žegklitz & Pošı́k , 2021 ) . This is an substantial drawback because many applications , such as healthcare , may involve noisy or infrequently sampled data ( Jensen et al. , 2014 ) . In this work , we develop the Discovery of Closed-form ODE framework ( D-CODE ) , which extends symbolic regression beyond the supervised learning setting . The key insight behind D-CODE is the variational formulation of ODE ( Hackbusch , 2017 ) , which establishes a direct link between the trajectory x ( t ) and the ODE f while bypassing the unobservable time derivative ẋ ( t ) . We develop a novel objective function based on this insight , and prove that it is a valid proxy for the estimation error of the true ( but unknown ) ODE . We demonstrate via extensive experiments that D-CODE can uncover the governing equations for a diverse range of dynamical systems while being substantially more robust to measurement artifacts than the alternative methods . Finally , D-CODE is designed as a general framework , where some of its components can be flexibly adapted based on the application . 2 BACKGROUND AND PROBLEM SETTING . Curating a dataset of trajectories for ODE discovery involves many decisions , e.g . What variables to include ? When and for how long to take measurements ? ( we discuss dataset curation in Appendix C. ) In this work , we assume the dataset is given and the variables can be modeled by a system of first-order autonomous ODEs ( Eq . 1 ) . We note that higher-order or time-dependent ODEs can be represented in this form by curating a dataset with additional variables ( Simmons , 1972 ) ( Appendix B ) . The system with J ∈ N+ variables is defined as ẋj ( t ) = fj ( x ( t ) ) , ∀j = 1 , . . . , J , ∀t ∈ [ 0 , T ] ( 1 ) where we use Newton ’ s notation ẋj ( t ) for the time derivative . The functions fj : RJ → R will be sometimes referred to as the ODEs directly . We denote T ∈ R+ as the maximum time horizon we have data for . We highlight the following distinction : the trajectory xj : [ 0 , T ] → R is a function of time , whereas the state xj ( t ) ∈ R , ∀t ∈ [ 0 , T ] is a point on the trajectory1 . We denote the state vector x ( t ) : = [ x1 ( t ) , . . . , xJ ( t ) ] > ∈ RJ and the vector-valued trajectory function x : = [ x1 , . . . , xJ ] . Let f∗j ’ s be the true but unknown ODEs to be uncovered , and xi : [ 0 , T ] → RJ , i ≤ N , N ∈ N+ be the true trajectories that satisfy f∗j ’ s . In practice , we only measure the true trajectories at discrete times and with noise . Denote the measurement of trajectory i at time t as yi ( t ) ∈ RJ ; we assume yi ( t ) = xi ( t ) + i ( t ) , ∀i ≤ N , t ∈ T ( 2 ) where i ( t ) ∈ RJ is zero-mean noise with standard deviation σ . The measurements are made at time t ∈ T = { t1 , t2 , . . . , T } . We denote the dataset as D = { yi ( t ) |i ≤ N , t ∈ T } . Closed form . The function fj : RJ → R has a closed form if it can be expressed as a finite sequence of operations ( + , ÷ , log , . . . ) , input variables ( x1 , x2 , . . . ) and numeric constants ( 1.5 , 0.8 , . . . ) ( Borwein et al. , 2013 ) . The functional form of fj is the expression with all the numeric constants replaced by placeholders θk ’ s , e.g . fj ( x ) = θ1x · log ( θ2x ) . To fully uncover f∗j , we need to infer its functional form and estimate the unknown constants θk ’ s ( if any ) . Variational formulation . The variational formulation provides a direct link between the trajectory x and the ODE fj without involving ẋ ( Hackbusch , 2017 ) . We start with the following definition . Definition 1 . Consider J ∈ N+ , T ∈ R+ , continuous functions x : [ 0 , T ] → RJ , f : RJ → R , and g ∈ C1 [ 0 , T ] , where C1 is the set of continuously differentiable functions . We define the functionals Cj ( f , x , g ) : = ∫ T 0 f ( x ( t ) ) g ( t ) dt+ ∫ T 0 xj ( t ) ġ ( t ) dt ; ∀j ∈ { 1 , 2 , . . . , J } ( 3 ) Importantly , the functional Cj depends on the testing function g ( t ) and its derivative ġ ( t ) but not ẋj . Proposition 1 below provides the variational formulation of ODE . Essentially it specifies the infinitely many constraints that x has to satisfy in order to be a solution to the ODE . ( see Appendix A ) . 1We restrict the domain of the function xj to [ 0 , T ] ( rather than R+ ) because we only have data up to T . However , after learning the ODE , we can extrapolate xj beyond T ( Appendix D ) . Proposition 1 . ( Hackbusch , 2017 ) Consider J ∈ N+ , T ∈ R+ , a continuously differentiable function x : [ 0 , T ] → RJ , and continuous functions fj : RJ → R for j = 1 , . . . , J . Then x is the solution to the system of ODEs in Equation 1 if and only if Cj ( fj , x , g ) = 0 , ∀j ∈ { 1 , . . . , J } , ∀g ∈ C1 [ 0 , T ] , g ( 0 ) = g ( T ) = 0 ( 4 ) 3 RELATED WORK . 3.1 SYMBOLIC REGRESSION . Symbolic regression attempts to uncover a closed-form prediction function a = f ( b ) using supervised learning ( Table 1 ) . The main challenge is optimization : searching for the optimal f is thought to be NP-hard because the space of closed-form functions is vast and complex—it is combinatorial in the functional form and continuous in the numeric constants ( Lu et al. , 2016 ) . Therefore , most existing work focuses on developing optimization algorithms . Genetic programming has been one of the most successfully and widely used methods ( Koza , 1994 ; Schmidt & Lipson , 2009 ) . It represents f as a tree where the internal nodes are operations and the leaves are variables or constants ( Figure 1 B ) . It then applies genetic algorithm to search for the best tree representation ( Forrest , 1993 ) . More recently , AI Feynman introduces a set of heuristics to directly prune the search space , thereby improving the optimization efficiency ( Udrescu & Tegmark , 2020 ) . Optimization methods based on reinforcement learning ( Petersen et al. , 2019 ) , pre-trained neural networks ( Biggio et al. , 2021 ) , and Meijer G-functions ( Alaa & van der Schaar , 2019 ) have also been proposed . All methods above use prediction error ( e.g . RMSE ) as the objective , which is not applicable to ODE discovery because the label ( time derivative ) is not observed . D-CODE advances symbolic regression beyond the supervised learning setting and addresses the unique challenges for ODE discovery : the equation-data mismatch and the sensitivity to noisy or infrequent observations . 3.2 DATA-DRIVEN DISCOVERY OF CLOSED-FORM ODES . Two-step sparse regression performs sparse regression on the estimated time derivative ̂̇xj ( Brunton et al. , 2016 ; Rudy et al. , 2017 ) . It assumes that the true ODE can be symbolically expressed in a linear form : ẋj ( t ) = ∑K k=1 θkhk ( x ( t ) ) where θk ∈ R are unknown constants and hk : RD → R are pre-specified candidate functions , such as monomials x , x2 , . . . ( Table 1 ) . The method uses L1 regularization on θk to ensure the learned function only contains a few terms . We review the methods to estimate time derivative in Appendix E. However , the assumed linear form is very restrictive ( e.g . x3/2 or 1/ ( x + 2 ) are not allowed ) . In fact , it can not symbolically represent many well-known ODEs ( e.g . generalized logistic model ) . Furthermore , the choice of candidate functions heavily depends on the human experts , which deviates from our goal of automated and data-driven ODE discovery . Two-step symbolic regression also uses the estimated time derivatives ̂̇xj as the label but applies symbolic regression instead ( Gaucel et al. , 2014 ) ( Table 1 ) . It may use any optimization method discussed in Section 3.1 to search for the optimal function . In this way , it removes the linear assumption on the functional form and can discover closed-form functions in general . The main difficulty is that the true label ( the derivative ) may not be accurately recoverable due to measurement and discretization error , but symbolic regression is sensitive to inaccurate labels because of the large search space ( Agapitos et al. , 2012 ) . As we show experimentally in Section 5 , even with the state-of-the-art methods for estimating derivatives , this approach still often fails when the noise is relatively low . We briefly review the methods for estimating derivatives from y ( t ) in Appendix E and show why this is generally a very challenging problem in Section 4.2 . The function approximator approach learns the true ODE with a function approximator f̂ , such as a neural network ( NN ) ( Chen et al. , 2018 ; Rubanova et al. , 2019 ) or a Gaussian process ( GP ) ( Heinonen et al. , 2018 ) . It estimates the unknown initial condition x̂ ( 0 ) and predicts the entire trajectory x̂ by solving the initial value problem ( IVP ) of the approximated ODE f̂ . The model is trained by minimizing the prediction error between predicted and the observed trajectories ( Table 1 ) . Importantly , the function approximator approach does not give a concise closed-form expression to describe the dynamics , which is the main objective of this work . Furthermore , its performance strongly depends on the prediction horizon used for training . Using an inappropriate horizon may fail with certain type of systems , e.g . chaotic systems ( Ditto & Munakata , 1995 ) . We further discuss this issue and provide experimental results in Appendix H.2 .

Summary: The paper addresses the problem of learning closed form ODEs from observational data, when the observation can be noisy and not frequent enough that instantaneous derivatives can be estimated with low enough error. The work uses a variational criterion on the solution of the ODE, circumventing the need to evaluate instantaneous derivatives. It uses existing symbolic regression techniques but instead of the regression loss, it minimizes a loss term quantifying the violation of the variational constraints. It is a work combining existing results in a novel way, but offers new perspective and there are synergies between the different parts applied.