D-CODE: Discovering Closed-form ODEs from Observed Trajectories

1 INTRODUCTION . An ordinary differential equation ( ODE ) links the state of a continuous-time dynamical system x ( t ) to its time derivative ẋ ( t ) via a function f , e.g . ẋ ( t ) = f ( x ( t ) , t ) . The ODE is closed-form when f has a concise and analytical expression ( e.g . f ( x , t ) = −t log ( x ) ) . Closed-form ODEs are ubiquitous in science and engineering . The popularity is partly due to their transparency and interpretability to human experts ( Petersen et al. , 2019 ) . Their concise functional form also facilitates the analysis of many key properties of the dynamical system , e.g . asymptotic stability ( Lyapunov , 1992 ) . However , the discovery of closed-form ODEs has been laborious and time-consuming as it heavily relies on human experts ( Simmons , 1972 ) . The goal of this work is to discover closed-form ODEs in an automated and data-driven way . We envision that such a tool would accelerate the quantitative modeling of dynamical systems , with applications ranging from discovering the kinetics of biochemical reactions to modeling tumor growth ( Schuster , 2019 ; Geng et al. , 2017 ) . Symbolic regression is an established approach to discover a closed-form function a = f ( b ) for predicting the label a from the feature b . It operates in the supervised learning setting and requires a dataset of label-feature pairs for training , i.e . D = { ai , bi } Ni=1 ( Schmidt & Lipson , 2009 ) . However , symbolic regression is not directly applicable to ODEs because we usually do not observe the time derivative ẋ ( t ) , i.e . the labels for regression are not available in the data . In contrast , we only have access to the measurements of state y ( t ) = x ( t ) + at some discrete time points and with noise . For instance , the datasets for tumor growth modeling typically contain tumor volumes measured at different clinical visits with substantial noise ( Wilkerson et al. , 2017 ; Mazaheri et al. , 2009 ) . We refer to this problem as the equation-data mismatch . A simple way to resolve this mismatch is to estimate ẋ ( t ) from data ( e.g . by numerical differentiation ( Bickley , 1941 ) ) , and use the estimated derivatives as labels for regression ( Gaucel et al. , 2014 ) . However , it is very challenging to recover the time derivative under high measurement noise or infrequent sampling ( Cullum , 1971 ) , and symbolic regression tends to perform poorly with inaccurate labels ( Žegklitz & Pošı́k , 2021 ) . This is an substantial drawback because many applications , such as healthcare , may involve noisy or infrequently sampled data ( Jensen et al. , 2014 ) . In this work , we develop the Discovery of Closed-form ODE framework ( D-CODE ) , which extends symbolic regression beyond the supervised learning setting . The key insight behind D-CODE is the variational formulation of ODE ( Hackbusch , 2017 ) , which establishes a direct link between the trajectory x ( t ) and the ODE f while bypassing the unobservable time derivative ẋ ( t ) . We develop a novel objective function based on this insight , and prove that it is a valid proxy for the estimation error of the true ( but unknown ) ODE . We demonstrate via extensive experiments that D-CODE can uncover the governing equations for a diverse range of dynamical systems while being substantially more robust to measurement artifacts than the alternative methods . Finally , D-CODE is designed as a general framework , where some of its components can be flexibly adapted based on the application . 2 BACKGROUND AND PROBLEM SETTING . Curating a dataset of trajectories for ODE discovery involves many decisions , e.g . What variables to include ? When and for how long to take measurements ? ( we discuss dataset curation in Appendix C. ) In this work , we assume the dataset is given and the variables can be modeled by a system of first-order autonomous ODEs ( Eq . 1 ) . We note that higher-order or time-dependent ODEs can be represented in this form by curating a dataset with additional variables ( Simmons , 1972 ) ( Appendix B ) . The system with J ∈ N+ variables is defined as ẋj ( t ) = fj ( x ( t ) ) , ∀j = 1 , . . . , J , ∀t ∈ [ 0 , T ] ( 1 ) where we use Newton ’ s notation ẋj ( t ) for the time derivative . The functions fj : RJ → R will be sometimes referred to as the ODEs directly . We denote T ∈ R+ as the maximum time horizon we have data for . We highlight the following distinction : the trajectory xj : [ 0 , T ] → R is a function of time , whereas the state xj ( t ) ∈ R , ∀t ∈ [ 0 , T ] is a point on the trajectory1 . We denote the state vector x ( t ) : = [ x1 ( t ) , . . . , xJ ( t ) ] > ∈ RJ and the vector-valued trajectory function x : = [ x1 , . . . , xJ ] . Let f∗j ’ s be the true but unknown ODEs to be uncovered , and xi : [ 0 , T ] → RJ , i ≤ N , N ∈ N+ be the true trajectories that satisfy f∗j ’ s . In practice , we only measure the true trajectories at discrete times and with noise . Denote the measurement of trajectory i at time t as yi ( t ) ∈ RJ ; we assume yi ( t ) = xi ( t ) + i ( t ) , ∀i ≤ N , t ∈ T ( 2 ) where i ( t ) ∈ RJ is zero-mean noise with standard deviation σ . The measurements are made at time t ∈ T = { t1 , t2 , . . . , T } . We denote the dataset as D = { yi ( t ) |i ≤ N , t ∈ T } . Closed form . The function fj : RJ → R has a closed form if it can be expressed as a finite sequence of operations ( + , ÷ , log , . . . ) , input variables ( x1 , x2 , . . . ) and numeric constants ( 1.5 , 0.8 , . . . ) ( Borwein et al. , 2013 ) . The functional form of fj is the expression with all the numeric constants replaced by placeholders θk ’ s , e.g . fj ( x ) = θ1x · log ( θ2x ) . To fully uncover f∗j , we need to infer its functional form and estimate the unknown constants θk ’ s ( if any ) . Variational formulation . The variational formulation provides a direct link between the trajectory x and the ODE fj without involving ẋ ( Hackbusch , 2017 ) . We start with the following definition . Definition 1 . Consider J ∈ N+ , T ∈ R+ , continuous functions x : [ 0 , T ] → RJ , f : RJ → R , and g ∈ C1 [ 0 , T ] , where C1 is the set of continuously differentiable functions . We define the functionals Cj ( f , x , g ) : = ∫ T 0 f ( x ( t ) ) g ( t ) dt+ ∫ T 0 xj ( t ) ġ ( t ) dt ; ∀j ∈ { 1 , 2 , . . . , J } ( 3 ) Importantly , the functional Cj depends on the testing function g ( t ) and its derivative ġ ( t ) but not ẋj . Proposition 1 below provides the variational formulation of ODE . Essentially it specifies the infinitely many constraints that x has to satisfy in order to be a solution to the ODE . ( see Appendix A ) . 1We restrict the domain of the function xj to [ 0 , T ] ( rather than R+ ) because we only have data up to T . However , after learning the ODE , we can extrapolate xj beyond T ( Appendix D ) . Proposition 1 . ( Hackbusch , 2017 ) Consider J ∈ N+ , T ∈ R+ , a continuously differentiable function x : [ 0 , T ] → RJ , and continuous functions fj : RJ → R for j = 1 , . . . , J . Then x is the solution to the system of ODEs in Equation 1 if and only if Cj ( fj , x , g ) = 0 , ∀j ∈ { 1 , . . . , J } , ∀g ∈ C1 [ 0 , T ] , g ( 0 ) = g ( T ) = 0 ( 4 ) 3 RELATED WORK . 3.1 SYMBOLIC REGRESSION . Symbolic regression attempts to uncover a closed-form prediction function a = f ( b ) using supervised learning ( Table 1 ) . The main challenge is optimization : searching for the optimal f is thought to be NP-hard because the space of closed-form functions is vast and complex—it is combinatorial in the functional form and continuous in the numeric constants ( Lu et al. , 2016 ) . Therefore , most existing work focuses on developing optimization algorithms . Genetic programming has been one of the most successfully and widely used methods ( Koza , 1994 ; Schmidt & Lipson , 2009 ) . It represents f as a tree where the internal nodes are operations and the leaves are variables or constants ( Figure 1 B ) . It then applies genetic algorithm to search for the best tree representation ( Forrest , 1993 ) . More recently , AI Feynman introduces a set of heuristics to directly prune the search space , thereby improving the optimization efficiency ( Udrescu & Tegmark , 2020 ) . Optimization methods based on reinforcement learning ( Petersen et al. , 2019 ) , pre-trained neural networks ( Biggio et al. , 2021 ) , and Meijer G-functions ( Alaa & van der Schaar , 2019 ) have also been proposed . All methods above use prediction error ( e.g . RMSE ) as the objective , which is not applicable to ODE discovery because the label ( time derivative ) is not observed . D-CODE advances symbolic regression beyond the supervised learning setting and addresses the unique challenges for ODE discovery : the equation-data mismatch and the sensitivity to noisy or infrequent observations . 3.2 DATA-DRIVEN DISCOVERY OF CLOSED-FORM ODES . Two-step sparse regression performs sparse regression on the estimated time derivative ̂̇xj ( Brunton et al. , 2016 ; Rudy et al. , 2017 ) . It assumes that the true ODE can be symbolically expressed in a linear form : ẋj ( t ) = ∑K k=1 θkhk ( x ( t ) ) where θk ∈ R are unknown constants and hk : RD → R are pre-specified candidate functions , such as monomials x , x2 , . . . ( Table 1 ) . The method uses L1 regularization on θk to ensure the learned function only contains a few terms . We review the methods to estimate time derivative in Appendix E. However , the assumed linear form is very restrictive ( e.g . x3/2 or 1/ ( x + 2 ) are not allowed ) . In fact , it can not symbolically represent many well-known ODEs ( e.g . generalized logistic model ) . Furthermore , the choice of candidate functions heavily depends on the human experts , which deviates from our goal of automated and data-driven ODE discovery . Two-step symbolic regression also uses the estimated time derivatives ̂̇xj as the label but applies symbolic regression instead ( Gaucel et al. , 2014 ) ( Table 1 ) . It may use any optimization method discussed in Section 3.1 to search for the optimal function . In this way , it removes the linear assumption on the functional form and can discover closed-form functions in general . The main difficulty is that the true label ( the derivative ) may not be accurately recoverable due to measurement and discretization error , but symbolic regression is sensitive to inaccurate labels because of the large search space ( Agapitos et al. , 2012 ) . As we show experimentally in Section 5 , even with the state-of-the-art methods for estimating derivatives , this approach still often fails when the noise is relatively low . We briefly review the methods for estimating derivatives from y ( t ) in Appendix E and show why this is generally a very challenging problem in Section 4.2 . The function approximator approach learns the true ODE with a function approximator f̂ , such as a neural network ( NN ) ( Chen et al. , 2018 ; Rubanova et al. , 2019 ) or a Gaussian process ( GP ) ( Heinonen et al. , 2018 ) . It estimates the unknown initial condition x̂ ( 0 ) and predicts the entire trajectory x̂ by solving the initial value problem ( IVP ) of the approximated ODE f̂ . The model is trained by minimizing the prediction error between predicted and the observed trajectories ( Table 1 ) . Importantly , the function approximator approach does not give a concise closed-form expression to describe the dynamics , which is the main objective of this work . Furthermore , its performance strongly depends on the prediction horizon used for training . Using an inappropriate horizon may fail with certain type of systems , e.g . chaotic systems ( Ditto & Munakata , 1995 ) . We further discuss this issue and provide experimental results in Appendix H.2 .

Summary: Authors propose D-CODE, a framework to approximate closed-form ODEs using symbolic regression. The method first estimate the trajectory by smoothing the observations (using any convenient smoother) before inferring a closed form (finite sequence of operations) transition function. This allows for better interpretability of the system.   Authors test D-Code on 5 simulated datasets, where they show high performance of the method, and on one real dataset where they use D-Code to model the temporal effect of chemotherapy on the tumor volume, and compare it to Total variation regularized differentiation (SR-T), the closest method in the literature.