Sparsistent Model Discovery

1 INTRODUCTION . Mathematical models are central in modelling complex dynamical processes such as climate change , the spread of an epidemic or in designing aircrafts . To derive such models , conservation laws , physical principles and phenomenological behaviors are key . However , some systems are too complex to model with a purely bottom up approach . In such situations , and when observational data is present , automated model discovery tools are becoming increasingly more useful to derive partial differential equations ( PDEs ) directly from the data . For example , to model the ocean dynamics in Sanchez-Pi et al . ( 2020 ) and for embryo patterning in Maddu et al . ( 2020 ) . From the mathematical point of view , model discovery of PDEs consists in finding F such that , ut = F ( 1 , u , ux , uxx , ... ) , where ut is the temporal derivative of the field u and u , ux , uxx , ... are higher order spatial derivatives . Usually F is identified based on an experiment consisting of n samples of the field u , see Brunton et al . ( 2016 ) ; Rudy et al . ( 2017 ) ; Schaeffer ( 2017 ) ; Raissi et al . ( 2017 ) . Some recent approaches use symbolic regression to find F , see Maslyaev et al . ( 2019 ) , but so far the most popular approach to perform model discovery is by linear regression which was first introduced in Rudy et al . ( 2017 ) and consists in considering F as a linear combination of some candidate terms , ut = Θ · ξ , where each column in Θ is a candidate term for the underlying equation , typically a combination of polynomial and spatial derivative functions ( e.g . u , ux , uux ) . In order to obtain a parsimonious PDE , many of the coefficients ξ must be zero , motivating the use of a sparse regression method to infer the equation , ξ̂ = arg min ξ ‖ut −Θ · ξ‖22 + λ ∑ i ‖ξi‖ρ . The best subset selection is obtained for ρ = 0 , Hastie et al . ( 2015 ) . However it requires solving a nonconvex and combinatorial optimisation problem . The Lasso , Tibshirani ( 1996 ) , is special in the sense that ρ = 1 is the smallest value of ρ which leads to a convex constraint region and hence a 1Data , code and results shared on : https : //anonymous.4open.science/r/sparsistent_ model_disco-56F8/ convex optimisation problem - for which very efficient solvers exist . Furthermore , there is a large corpus of theoretical work available for the Lasso from which the model discovery community could benefit . It is anecdotaly known in the model discovery community that the Lasso does not perform very well compared to other relaxations of the original problem , see Rudy et al . ( 2017 ) , Li et al . ( 2019 ) , Rudy et al . ( 2019 ) and Maddu et al . ( 2019 ) , however it has never been studied why . We trace back this lack of performance to the potential variable selection inconsistency of the Lasso . Let us focus on variable selection consistency or sparsistency , a property defined as : when the number of samples n −→∞ , the estimated vector ξ̂ contains the same nonzero terms as the true vector ξ . By first revisiting a geometric interpretation of the irrepresentability condition ( IRC ) of the Lasso in the context of model discovery , we provide some theoretical insights on Lasso ’ s inconsistency but also on the design of the libraries generated for sparse regression based model discovery . In addition , these libraries have to be estimated using methods that introduce some deterministic noise into the sparse regression problem . We show when and why the adaptive Lasso Zou ( 2006 ) design matrices might have more chances of satisfying the IRC and introduce the randomised adaptive Lasso to perform variable selection under violated IRC . The latter is tuned by stability selection Meinshausen & Bühlmann ( 2010 ) , which has been used in the past for model discovery in pure sparse regression based approaches Li et al . ( 2019 ) ; Maddu et al . ( 2019 ) but without any variable selection error control . Furthermore , purely relying on sparse regression such as Rudy et al . ( 2017 ) , Li et al . ( 2019 ) and Maddu et al . ( 2019 ) heavily limits to low noise and dense data sets , due to the differentiation method used to build the library ( typically numerical differentiation or splines ) . Purely relying on deep learning , see Raissi et al . ( 2017 ) , to discover F will result in hardly interpretable equations . In Both et al . ( 2021a ) and Chen et al . ( 2020 ) , the two problems are tackled by concurrently learning a solution of the PDE using a physics informed neural network and inferring an explicit equation by performing sparse regression on a library built by automatic differentiation . However , deep learning model discovery frameworks typically require manually tuning many hyperparameters which are sensitive to the input data : ( 1 ) the original DeepMod Both et al . ( 2021b ) requires to tune a threshold to prune coefficients with small magnitudes , ( 2 ) PiDL Chen et al . ( 2020 ) introduces a couple of multipliers to parameter the amount of physics informed regularisation and the amount of regularisation in the sparsity estimator . Our results show how once the randomised adaptive Lasso with stability selection is integrated within the deep learning model discovery framework DeepMod , a single set of hyperparameters can be used to recover a wide variety of PDEs . Contributions • We show the design matrices used to infer the equations by sparse regression can violate the irrepresentability condition ( IRC ) of the Lasso , even when derived from analytical PDE solutions , i.e . without additional noise . This implies any sparse regression based model discovery framework needs to deal with highly correlated irrelevant variables with relevant ones , no matter the differentiation method used to compute the library . • We introduce a randomised adaptive Lasso ( rAdaLasso ) with stability selection and error control algorithm , to recover the true underlying PDE in the presence of design matrices that are highly correlated and violate the IRC . • By integrating rAdaLasso within the deep learning model discovery framework DeepMod , we show a wide variety of nonlinear and chaotic canonical PDEs can be recovered : ( 1 ) at higher noise-to-sample ratios than state-of-the-art algorithms , ( 2 ) with a single set of hyperparameters , paving the road towards truly automated model discovery . 2 THEORY . Sparse regression based model discovery sometimes fails to discover the correct underlying PDE from a data set , even when the model is present in the library and contains little noise . To illustrate this , we present a two-soliton solution2 of the Korteweg-de-Vries ( KdV ) equation in figure 1 ( a ) . Considering a library Θ of p =12 terms , the Lasso fails to select the correct terms of the underlying 2Obtained from an analytical solution , see details in Appendix D. PDE , see figure 1 ( b ) , even in the absence of noise . In this section , we explain why it occurs and introduce the irrepresentable condition to identify the cause . 2.1 ON LASSO ’ S INCONSISTENCY The Lasso is known to be variable selection consistent under the irrepresentable condition ( IRC ) , see3 Hastie et al . ( 2015 ) , which requires the existence of η > 0 such that , max j∈N || ( ΘTTΘT ) −1ΘTTΘF , j ||1 < 1− η , ( 1 ) where ΘT is the subset of the design matrix Θ that contains the true model and ΘF , j a column of the subset of the design matrix that contains the rest of vectors . Conversely , the Lasso will not consistently select the true variables for any design matrix , even if they are present in the library . Geometric interpretation it can be remarked that ( ΘTTΘT ) −1ΘTTΘF , j is the least squares solution of , ΘF , j = ΘT ξ + , ( 2 ) with Gaussian noise , see figure 2 ( a ) . The IRC will be violated if any of the projections of ΘF , j onto the column space of ΘT is larger than 1 , meaning they lie outside the unit sphere see figure 2 ( b ) . The IRC will be trivially satisfied when ΘF , j is orthogonal to the column space of ΘT , in which case η = 1 : meaning irrelevant vectors are orthogonal to the relevant vectors Hastie et al . ( 2015 ) . Furthermore , it becomes clear from the example on figure 2 , that if the relevant vectors are orthogonal to each other then the projections of irrelevant terms onto the space of relevant vectors will be exactly the correlations and the IRC will be less likely to be violated . While it is obvious the IRC will more likely be violated in the case the data is highly correlated : correlations among irrelevant vectors will not lead to a violation of the IRC . 3Many formulations of the IRC can be found in literature such as the seminal work from Zhao & Yu ( 2006 ) and Meinshausen & Bühlmann ( 2010 ) - we do not pretend to be exhaustive on the matter but rather choose the one that better suits our purpose . A diagnostics metric ∆ based on the IRC we introduce , ∆ ( Θ , T ) = max j∈N || ( ΘTTΘT ) −1ΘTTΘF , j ||1 , ( 3 ) where T are the vectors of the support set S and F on its complementary set Sc . In practice , the true support S is unknown . Its estimation Ŝ can be obtained for example by taking for granted the result from a first variable selection . ∆ ( Θ , T̂ ) can help us determining if our library Θ is sufficiently well designed : if ∆ < 1 we know a Lasso can distinguish Ŝ from Ŝc , otherwise the variable selection result should be taken with caution . It is worth pointing out , that by no means if ∆ ( Θ , T̂ ) < 1 we can claim T̂ = T . 2.2 ON THE ADAPTIVE LASSO ’ S SPARSISTENCY . We have seen in the previous section that the Lasso might not be variable selection consistent when ∆ ( Θ , T ) > 1 , meaning that even if the true model is present in the library it might not be selected . Instead , we propose to use the adaptive Lasso which is known to preserve the consistency , see Zou ( 2006 ) . In figure 1 we illustrate that applying it , results in the correct underlying KdV equation to be recovered . Let us give insights of why it might perform better on model discovery problems . The adaptive Lasso is a two-step estimation procedure where first an initial estimation of the coefficients is obtained to derive a weight vector ŵi = 1/|ξ̂i|γ . We fix γ = 2 throughout this work . ξ̂i is preferably obtained using a Ridge regression to handle multicollinearity . In the second step , a Lasso is applied on the weighted coefficient , ŵi , penalising the terms with their respective weights , i.e. , ˆ̃ ξ = arg min ξ̃ ( 1 2n ||∂tu− Θ̃ξ̃||22 + λ p∑ i=1 ||ξ̃i||1 ) , ( 4 ) where Θ̃i = Θi/ŵi and ξ̃i = ŵiξi . Let us consider the impact of the transformation on the design matrix by the adaptive weights . Proposition 1. by assuming all relevant coefficients are larger than the irrelevant ones in magnitude and γ ≥ 1 , then as a result of the transformation , the projection of the jthirrelevant vector onto the kth relevant vector will shrink by a factor |ξ̂F , j/ξ̂T , k|γ and , ∆ ( Θ̃ , T ) ≤ ∆ ( Θ , T ) . ( 5 ) The design matrix Θ̃ will therefore have more chances to verify the IRC than Θ . See proof on appendix A . By coming back to the example presented on figure 1 , ∆ ( Θ , T ) = 1.69 while ∆ ( Θ̃ , T ) = 2e−14 , which illustrates inequality 5 . It is worth insisting the correlation matrices of Θ and Θ̃ are identical . However , the projection of the irrelevant vector ux into the space of relevant vectors ( uux , uxx ) becomes very small , and small enough in this case for the IRC not to be violated anymore ( by replacing Θ with Θ̃ ) . This gives insights why the adaptive Lasso manages to identify the true underlying model when the Lasso would not .

Summary: This paper deals with the problem of model discovery based on sparse regression. Authors propose a new randomised adaptive lasso for that selects the terms to be used in the PDEs selection that is used in conjunction with Deep learning framework. Authors then experimentally verify on 4 PDEs that their method perform better for model selection and at least as well when looking at MSE loss.   For a detailed summary, authors first verify the violation of IRC which is defined as in eq (1) for Lasso on an analytic PDE (Korteweg-de-Vries (KdV) equation). As their first contribution, authors develop a diagnostic metric that indicates the violation of Lasso when one know the true support of the design matrix. In proposition 1, authors show that under certain assumptions on weights, the adaptive Lasso, where one reweighs the design matrix has better diagnostic score than Lasso.  Since the design matrix is typically approximated, if we incur a deterministic error term, authors empirically show that adaptive Lasso is also susceptible to the violation of the IRC. In lieu of this experiment, authors propose randomised adaptive Lasso, where each column of the design matrix of adaptive Lasso is randomly scaled with $\beta(1,2)$. Author integrate it in deep learning model discovery framework and empirically verify the performance of of rAdaLasso for 1) MSE loss and 2) model selection error.