Sparsistent Model Discovery
Authors: Georges Tod
Gert-Jan Both
Remy Kusters
Venue: ICLR 2022 Submitted (2022)
Keywords: model discovery
sparse regression
sparsistency
physics informed deep learning
partial differential equations

Abstract: Discovering the partial differential equations underlying spatio-temporal datasets from very limited and highly noisy observations is of paramount interest in many scientific fields. However, it remains an open question to know when model discovery algorithms based on sparse regression can actually recover the underlying physical processes. In this work, we show the design matrices used to infer the equations by sparse regression can violate the irrepresentability condition (IRC) of the Lasso, even when derived from analytical PDE solutions (i.e. without additional noise). Sparse regression techniques which can recover the true underlying model under violated IRC conditions are therefore required, leading to the introduction of the randomised adaptive Lasso. We show once the latter is integrated within the deep learning model discovery framework DeepMod, a wide variety of nonlinear and chaotic canonical PDEs can be recovered: (1) up to $\mathcal{O}(2)$ higher noise-to-sample ratios than state-of-the-art algorithms, (2) with a single set of hyperparameters, which paves the road towards truly automated model discovery.

SPARSISTENT MODEL DISCOVERY
Anonymous authors
Paper under double-blind review
ABSTRACT
Discovering the partial differential equations underlying spatio-temporal datasets from very limited and highly noisy observations is of paramount interest in many scientific fields. However, it remains an open question to know when model dis- covery algorithms based on sparse regression physical processes. In this work, we show the design matrices used to infer the equations by sparse regression the Lasso, even when derived from analytical PDE solutions (i.e. without addi- tional noise). Sparse regression techniques which can recover the underlying model under violated IRC conditions are therefore required, leading to the intro- duction of the randomised adaptive Lasso. We show within the deep learning model discovery framework Deep.Mdd) a wide variety of nonlinear and chaotic canonical PDEs can be recovered: (1) up to ( higher noise-to tample ratios than state-of-the algorithms, (2) with single set of hy- perparameters, which paves the road towards truly automated model discovery.
actually recover the underlying
violate the irrepresentability condition (IRC)
the latter is integrated
INTRODUCTION
Mathematical models are central in modelling complex dynamical processes such climate change, the spread of an epidemic or in designing aircrafts. To derive such models, conservation laws, phys- ical principles and phenomenological behaviors are key. However, some systems are too complex to model with purely bottom up approach. In such situations, and when observational data is present, automated model discovery tools are becoming increasingly more useful to derive partial differential equations (PDEs) directly from the data. For example, to model the ocean dynamics in Sanchez-PP) jet al. (2020) and for embryo patterning in Maddu et al. (2020). From the mathematical point of view, model discovery of PDEs consists in finding FF such that,
U/ = F(1 (1,10 Mpy Mzz....),
where is the temporal derivative of the field u and Ug Mezz are higher order spatial deriva- tives. Usually is identified based on an experiment consisting of samples of the field u, see Brunton et al.] (2016); Rudy et al. (2177;; Schaeffer (2017); Raissi et al. (2017). Some recent ap- proaches use symbolic regression to find J. see Maslyaev al. (2019), but so far the most popular approach to perform model discovery is by linear regression which was first introduced in Rudy cct al. (2017) and consists considering F as a linear combination of some candidate terms,
Ut +.
where each column in Ô is a candidate term for the underlying equation, typically combination of polynomial and spatial derivative functions (e.g. u, uuz)) In order to obtain parsimonious PDE, many of the coefficients & must be zero, motivating the use of sparse regression method to infer the equation,
= arg min llue - -.111
The best subset selection is obtained for 0. Hastie et al. (015). However requires solving a nonconvex and combinatorial optimisation problem. The Lasso. Tibshirani (199).. is special in the sense that p is the smallest value which leads to a convex constraint region and hence a
Data, code and results shared on: https:://aoonyn 4open dopen.scis sci ence/r/spars
stert. istent.
model di sco- 56r87

---

convex optimisation problem for which very efficient solvers exist. Furthermore, there is large corpus of theoretical work available for the Lasso from which the model discovery community could benefit. it is anecdotaly known in the model discovery community that the Lasso does not perform very well compared to other relaxations of the original problem, see Rudy et al. (011)). [i et al. (2019), |Rudy et al. (20T9) and Maddu al.] (2019), however it has never been studied why. We trace back this lack of performance to the potential variable selection inconsistency of the Lasso.
Let us focus on variable selection consistency or sparsistency. a property defined as: when the number of samples n + oc, the estimated vector contains the same nonzero terms as the true vec- tor &. By first revisiting a geometric interpretation of the irrepresentability condition (IRC) of the Lasso in the context of model discovery, we provide some theoretical insights on Lasso's inconsis- tency but also on the design of the libraries generated for sparse regression based model discovery. In addition, these libraries have to be estimated using methods that introduce some deterministic noise into the sparse regression problem. We show when and why the adaptive Lasso Zou (2006) design matrices might have more chances of satisfying the IRC and introduce the randomised adap- tive Lasso to perform variable selection under violated IRC. The latter is tuned by stability selection Meinshausen & Bühhmann (2010), which has been used in the past for model discovery in pure sparse regression based approaches Li al. (2019); Maddu al. 2019) but without any variable selection error control.
Furthermore, purely relying on sparse regression such as Rudy al. (2017). Li al. (2019) and Maddu et al. (2019) heavily limits to low noise and dense data sets, due to the differentiation method used to build the library (typically numerical differentiation or splines). Purely relying on deep learn- ing, see Raissi et al. 2017). to discover F will result in hardly interpretable equations. In Both et al. (2021a) and Chen al. (2020), the two problems are tackled by concurrently learning a solution of the PDE using a physics informed neural network and inferring an explicit equation by performing sparse regression on a library built by automatic differentiation. However, deep learning model dis- covery frameworks typically require manually tuning many hyperparameters which are sensitive to the input data: (1) the original DeepMod Both et al. (202 requires to tune a threshold to prune coefficients with small magnitudes, (2) PiDL Chen et al. (2020) introduces couple of multipliers to parameter the amount of physics informed regularisation and the amount of regularisation in the sparsity estimator. Our results show how once the randomised adaptive Lasso with stability selec- tion is integrated within the deep learning model discovery framework DeepMod, a single set of hyperparameters can be used to recover wide variety of PDEs.
Contributions
We show the design matrices used to infer the equations by sparse regression violate the irrepresentability condition (IRC) of the Lasso, even when derived from analytical PDE solutions, i.e. without additional noise. This implies any sparse regression based model discovery framework needs to deal with highly correlated irrelevant variables with relevant ones, no matter the differentiation method used to compute the library.
We introduce a randomised adaptive Lasso (rAdaLasso) with stability selection and error control algorithm, to recover the true underlying PDE in the presence of design matrices that are highly correlated and violate the IRC.
By integrating rAdaLasso within the deep learning model discovery framework DeepMod, we show wide variety of nonlinear and chaotic canonical PDEs can be recovered: (1) at higher noise-to-sample ratios than state-of-the-art algorithms, (2) with a single set of hyperparameters, paving the road towards truly automated model discovery.
2 THEORY
Sparse regression based model discovery sometimes fails to discover the correct underlying PDE from data set, even when the model is present in the library and contains little noise. To illustrate this, we present a two-soliton solution of the Korteweg-de Vries (KdV) equation in figure aa). Considering a library θ of =12 terms, the Lasso fails to select the correct terms of the underlying
Obtained from analytical solution, details in Appendix
2

---

Lasso
AdaLzzeo
10
100p
10
10.5
10-3
10-7
10"
10"
10-7
10"5
10-2
A
noiseless library
(b) coefficient paths
Figure 1: Example PDE term selection inconsistency using the Lasso from noiseless library of a two-soliton analytical solution the Korteweg-de-Vi Vries (KdV) equation: úe = -uuu, - Ugaz- In (a) the terms U₂ and w are highly correlated. In (b) matter the regularisation, the Lasso selects the spurious term Ug. On the right hand side, with a proper choice of x, the adaptive Lasso might select the true model.
PDE, figure [b), in the absence of noise. In this section, explain why it occurs and introduce the irrepresentable condition to identify the cause.
2.1 ON LASSO's INCONSISTENCY
The Lasso is known to be variable selection consistent under the ir- representable condition (IRC), see Hastie et al. (2015). which re- quires the existence of 1) > 0 such that,
unit sphere
max [1(##rr) -040 JEN
77.
where O7 the subset of the design matrix θ that contains the true model and Oz column of the subset of the design matrix that con- tains the rest of vectors. Conversely, the Lasso will not consistently select the true variables for any design matrix, even if they are present the library.
101
Geometric interpretation can be remarked that (0)0r) is the least squares solution of.
(a)
relevant
OF Or + c,
(2)
vectors
with Gaussian noise, figure a)) The IRC will be violated if any of the projections OF,, onto the column space of O is larger than I, meaning they lie outside the unit sphere see figure bb... The IRC will be trivially satisfied when F is orthogonal to the column space of Or, in which case 7) 1: meaning irrelevant vectors are orthogonal to the relevant vectors Hastie et al. (2015). Furthermore, becomes clear from the example on figure that if the relevant vectors are orthogonal to each other then the projections of irrelevant terms onto the space relevant vectors will be exactly the corre- lations and the IRC will be less likely to be violated. While it is obvious the IRC will more likely be violated the case the data highly correlated: correlations among irrelevant vectors will not lead to violation of the IRC.
Bygs
4. projection
Van,
(b)
Figure Example of projec- tion that lies outside the unit sphere: the IRC is violated.
Many formulations of the IRC can be found in literature such as the seminal work from Zhao & Yu Yu 2006 and Meinshausen & Bühlmann (2010) we do not pretend to be exhaustive the matter but rather choose the that better suits our purpose.

---

A diagnostics metric A based on the IRC we introduce,
A(0,7) = max JEN
where T are the vectors of the support set S and F on its complementary set S.. In practice, the true support S is unknown, Its estimation $ can be obtained for example by taking for granted the result from first variable selection. A(O, I") can help us determining if our library is sufficiently well designed: if
we know a Lasso can distinguish S from Sc. otherwise the variable selection result should taken with caution. It is worth pointing out, that by no means if A(O, 7) < 1 we can claim f = T.
2.2
ON THE ADAPTIVE LASSO'S SPARSISTENCY
We have seen in the previous section that the Lasso might not be variable selection consistent when A 7) > 1, meaning that even if the true model is present in the library it might not be selected. Instead, we propose to use the adaptive Lasso which known to preserve the consistency, see Zou (206). In figure I we illustrate that applying it, results in the correct underlying KdV equation to be recovered. Let us give insights why it might perform better on model discovery problems. The adaptive Lasso is two-step estimation procedure where first initial estimation of the coefficients obtained derive weight vector w; 1/lÊ,/". We fix preferably obtained using a Ridge regression to handle multicollinearity. In the second step, a Lasso applied on the weighted coefficient, dū penalising the terms with their respective weights, i.e.
2 throughout this work. & is
& = arg min
- 1 2n
1
6E12 +
16(11)
where θ: = O./ú; and = w.st. Let us consider the impact of the transformation on the design matrix by the adaptive weights.
Proposition 1. by assuming all relevant coefficients are larger than the irrelevant ones in magnitude and 7 2 then as result of the transformation, the projection of the irreleevant vector onto the 1tth relevant vector will shrink by a factor / and,
(( S ((0,T)-
(5)
The design matrix Ô will therefore have more chances to verify the IRC than θ.
See proof on appendix A By coming back to the example presented on figure ! A = 1.69
while A (Ô, 7) = 2e which illustrates inequality 5 is worth insisting the correlation matrices of O and é are identical. However, the projection of the irrelevant vector ua into the space of relevant vectors (wigg Mzx) becomes very small, and small enough in this case for the IRC not be violated anymore (by replacing & with θ). This gives insights why the adaptive Lasso manages to identify the true underlying model when the Lasso would not.
2.3 WHAT HAPPENS WITH MORE REALISTIC LIBRARIES
In practice, the design matrix O cannot be observed, nor measured, nor derived from analytical solutions,
did figure 1 and has to be estimated. In classic model discovery the library θ is typically built using splines and/or numerical differentiation. In neural network based model discovery, there will also a non-random misfit between the output of the neural network and the data. As a result, these methods introduce non-random approximations of the higher order derivatives which can be seen as additional deterministic noise on top of the ground truth,
U1 ( + (0+ 8
(6)
We see in practice that if the correlations introduced by 5 are not too large, the true PDE can still be discovered by choosing the correct amount of regularisation for the randomised adaptive Lasso. To see the effect the interpolation and differentiation methods, we compute A from 2 libraries based on numerical solution of the chaotic Kuramoto-Sive Sivashiiki (KS) equation (see|Rud et a (2017)
4

---

Adaaesso
rondomised AdaLasso
rAdaLasso + ortor control
12
1.0
selection prob. I
0.8
selection prob. IIA
0.88
0.6
0.6
3
0.4
0.4
0.2
frrelevant
0.2
irrelevant
relevant
relevami
0.0
0.0
10-8
10-8
10-0
10-9
10-7
10-5
A
A
(a) stability plots
(b) selection error control
Figure 3: Randomising the adaptive Lasso to perform model selection under violated IRC. The library comes from Rudy al. (2017), and obtained by polynomial interpolation and numerical
differentiation from a numerical solution of the Kuramoto-Sivashinksy equation (u+ = - -W0 Maz Marzz) with additional 1% Gaussian white noise. Even violates the IRC: A(ê, 7) = 1.77. In (a) only the randomised adaptive Lasso allows disentangle relevant from irrelevant PDE terms. In (b), contours represent the upper bound on the selection error. The relevant PDE terms can be found by the rAdaLasso by proper choice of the maximum number of expected false positives EV 2 (at fixed minimum probability of being selected Tthr = 0.9).
with varying noise levels. Both libraries contain with p 36 potential terms and are derived by poly- nomial interpolations and numerical differentiation. When noise is added, A(O0%: 7) 1.33 and A ((0%. T) 7 meaning the adaptive weights used by the adaptive Lasso help cast- ing better the design matrix. However as soon as 1% noise is added, A(0 T") = 1.38 and (
1%, 7) = 1.77: the Lasso nor the adaptive Lasso would be able to select the true model. In
this case, the interpolation and differentiation methods introduce enough correlations for any design matrix to violate the IRC. On figure 3 the stability plots show the adaptive Lasso would not be able to select the true model even present in the library, no matter the amount of regularisation; this motivates the introduction of supplementary ingredient.
2.4 RANDOMISED ADAPTIVE LASSO (RADALASSO)
To work under violated IRC due to correlations that are due to the underlying physical process itself and/or interpolation and/or the differentiation method, we introduce randomised adaptive Lasso. It is inspired by the random Lasso presented in Meinshausen & Bühlmann (2010).
T = arg min
- + 2n
-
|& W,
where W, is randomly selected from a beta distribution, Such randomisation the regularisation is equivalent to a random rescaling of each column of the design matrix followed by an adaptive Lasso, it is therefore straightforward to solve Empirically, we can see the randomisation breaks the correlations and allows to disentangle the group of relevant from the group of irrelevant variables in stability selection loop: the stability plots without and with randomisation on figure 3
8(1, to promote weights close to 0.
Controlling the selection error The determination of can be done using stability selection, see Meinshausen & Bühlmann (2010), which has been used in the past for model discovery Li et al. (2019); Maddu et al. (2019). However, error control has not been used and is in our opinion under- utilised: as we will see in the experimental results its hyperparameters can become data insensitive. For the sake of completeness we revisit the details of stability selection on Appendix and recall
'It is (slightly) more computationally expensive than the adaptive Lasso with stability selection the ran- domness prevents from the benefit of warm starts.
5

---

main results in the following lines. The set of variables to be selected S is determined using two hyperparameters: Tthr is the minimum probability of being selected and an upper bound on the expected number of false positives EV
SA {i such that max f, 2 Mehr for A E }
(8)
where the regularisation path is restricted by an upper bound,
R.
A = A E A such that, E(V) - (2 1)p
S EV EV max
(9)
where GA is the average of selected variables. On figure 3b,,, the upper bounds on the selec- tion errors are represented using contours on top the stability plot. The set variables to be selected stable given by the terms inside given contour line while being above the threshold Mthr-
To conclude this section, in the context of model discovery. sparse regression is usually performed on highly correlated data, due to the data itself and/or to the differentiation method used to estimate the library, which will tend to violate the IRC. This means that even if the true model present the library it might not be selected by the Lasso. As a mitigation, introduce a randomised adaptive Lasso and show once in a stability selection loop with error control, the underlying true model can still be recovered.
2.5
DEEPMOD INTEGRATION
Neural network based model discovery improves the quality of the library with respect numerical differentiation based methods, see Both et al.] (20216) We can therefore expect the deterministic noise 8 be much smaller. To leverage such capability, we implement the randomised adaptive Lasso with stability selection and eгror control in the deep learning model discovery framework DeepMod |Both & Kusters (2020). The framework combines function approximator of u, typi- cally deep neural network which is trained with the following loss,
=
1 - O(E M)IB
(10)
Exwe
Luv
The first term Lmse learns the data mapping (x, - û, while the second term Ling constrains the function approximator to solutions of the partial differential equation given by 0zu, θ and (É M). The terms be selected in the PDEs are determined using mask M derived from the result of the randomised adaptive Lasso with stability selection and error control,
M₁ {
I
if f E SA otherwise
stable
(11)
where i E [1, is the index of a potential term and S: is determined by equation 8 The coefficients { in front of the potential terms are computed using Ridge regression on the masked library (e -M). During training, if Emse on the test set does not vary anymore or if it increases, the sparsity estimator is triggered periodically. As a result, the PDE terms are selected iteratively by the dynamic udpate of the mask M during the training. In practice, this promotes the discovery of parsimonious PDEs.
3 EXPERIMENTS
In this section, we first show how the randomised adaptive Lasso compares with state-of-the-art sparsity estimators. Second, once within DeepMod, we compare the original DeepMod frame- work.
'The randomised adaptive Lasso promoted here, uses the Ridge and Lasso inplementations from scikit- learn, Pedregosa (20TT). DeepMod is implemented in JAX, Bradhury et al. (018))

---

Known challenging cases from literature. When polynomial interpolation is used com- pute higher order derivatives from noisy data, is known that the quality of the library is going to be poor making it challenging to discover the underlying PDE by sparse regression. For both libraries A > 1 revealing the Lasso would not be able to recover the true support. *KS: Kuramoto-Sivashing
Table 1:
#
PDE
Noise
Terms
Deriv. Order
source
A
1
KS*
1%
36
250k
Rudy al. (2017)
1.38
2
Burgers
4%
19
4
20k
Maddu et al. (2019)
1.23
Table 2: Success in recovering the ground truth PDE terms for table cases. Here we reproduced the results from Rudy al. (2017). Maddu et al. (2019) (h stands for heuristic) and report addi- tional results using the Lasso, adaptive Lasso and randomised adaptive Lasso. In case I, PDE-FIND does find the correct terms, while does not in case In the latter, PDE-STRIDE and a randomised adaptive Lasso do, see figure 5
regularisation
Case
Case
Lasso
x
x
randomised Lasso
4
x
PDE-FIND (STRidge)
h
X
PDE-STRIDE (IHT) adaptive Lasso randomised adaptive Lasso
lo
V
1
x
X
/
V
Comparing with state-of-the art sparsity estimators In order to get an idea of the performance of the randomised adaptive Lasso with stability selection and erгоr control, we compare it to two pure sparse regression based model discovery approaches: PDE-FIND Rudy al. (2017) and PDE- STRIDE Maddu et al. (2019). While the first is heuristic, the latter solves relaxation of the best subset selection (4 regularisation) using an Iterative Hard Thresholding algorithm. To make sure the comparison is fair, we compare our approach with the ones from literature using the data from the original authors of those approaches. Furthermore, we restrict ourselves to cases where the original authors have tuned their algorithms and present the cases being hard ones, see table [ In these cases, ( 1. meaning they violate the IRC, see table [ The results from the benchmark are presented in table 2 For case 1, ( T) ~ 1.77 and for case 2, ( T) se 19 explaining why the adaptive Lasso alone will not work in those cases. The result for case is presented
figure 3 From figure 5 with proper tuning both the randomised adaptive Lasso as well as the Iterative Hard Thresholding (IHT) algorithm can recover the true underlying PDE of case 2. However, the computational cost of the IHT is much higher (x 100) than the one of the randomised adaptive Lasso (rAdaLasso), which solves convex optimisation problem.
Impact of rAdaLasso in DeepMod To quantify the impact of the proposed sparsity estimator within DeepMod we compare DeepMod with rAdaLasso and a baseline (the original DeepMod). The latter leverages thresholded Lasso with a preset threshold of 0.1 (to cut-off small terms) and found by cross validation on folds. We simulate model discoveries for the Burgers, Kuramoto- Sivashinsky (KS) and two additional PDEs that introduce different nonlinearities and derivative orders: Kortweg-de- Vries (KdV), u, = -6uu, Ugra and Whwell-Whttheadd (NW), Uy = 10uggx + u(1 u²) 0.4. A single set of hyperparameters used in all cases Appendix C The results are reported on figure Our approach allows to recover all PDEs without overfitting while the original DeepMod would for all, except for the KdV equation. The stability plot obtained figure bbb) for the KS equation can be compared to the one presented on figure b)::: the combination
obtained by running code with eters from the authors original work. In terms of computational cost, epoch takes in average around 0.04s (with 2 samples) on GeForce RTX 2070 GPU from NVIDIA: discovering the KS equation takes around 90k epochs and around hour.
'The computational cost reported
data and hyperparam-
7

---

Bargens (6 26.0 40%)
KdV. (a 2 1836)
NW. (i 28.0 2096)
KS, (n - 2 3006)
10-1
MSE
10
n 300.
thresholded
DeepMod
DeepMod
30
selec. emor
20
10
0
10"
10a
10*
10'
10
10'
10a
10a
10a
10a
epoch
epoch
epoch
spoch
(a) MSE test set, recovered PDEs and term selection error during leamning
Burgers
KdV
NW
K3
solection pzob. 11
0.75
0.50
-3
33
3
-2
2
0.25
10
10
10
10
10
(b) rAdaLasso stability plots after DeepMod converged
Figure 4: With and without rAdaLasso within DeepMod. In (a), true underlying equations are recovered when using the proposed rAdaLasso within DeepMod, from 2k samples for varying a% Gaussian white noise levels. The original DeepMod leverages a thresholded LassoCV sparsity estimator which selects spurious terms (except for the KdV example) and results in poorly general- isable PDEs (the MSE on the test set increases). In (b), the stability plots show the selected terms for all the examples have become independent (after DeepMod has converged) from the hyperpa- rameters the and EV mea-
of rAdal.sse Lasso and DeepMod allow to recover the chaotic equation with greater confidence as the probability of selecting irrelevant terms is null.
Benchmarking noise-to-sample ratios We compare the ratios a/n where a is the Gaussian white noise expressed in percent of successful model discoveries of additional frameworks for the
two investigated cases literature: Burgers and KS equations. The compared frameworks are PiDl_(deep learning based thats uses the sparsity estimator of PDE-FIND, Chen et al. (2020))
S (sparse bayesian learning. Yuan et al. (2019). SNAPE (basis function approximations based, Bhowmick & Nagarajaiahl (2021)) and R-DLGA (symbolic regression based, Xu & Zhang (221 )
The results can be found on table 3 Symbolic regression based approaches such as R-DLGA are more general than sparse regression based approaches in the sense they do need predefined library, but do not perform as well. For the Burgers equation, the approach proposed in this work can perform the discovery the same noise-to-sample ratio than PiDL but with larger library (x2) and much higher noise-to-sample ratio (x 100) for the KS equation.
Some limitations of our approach presented Appendix
8

---

Table 3:
Noise-to-sample ratios (V) of successful PDE discoveries from state-of-the-dr frame- works. In parenthesis, the sparsity estimator and the library size (p) are specified when applicable.
framework (spar. est.)
Burgers (p)
KS (p)
source
PDE-FIND (STRidge)
4e (16)
4e (36)
Rudy et al.
2017)
PDE-STRIDE (IHTT)
2e 2c-5 (19)
Maddu (2019)
S (SBL)
2e (36)
Yuan et al.
2019
PiDL (STRidge)
2e- (16) 3e (NA))
3e (36)
Chen et al.
20200
SNAPE (NA)
Bhowmick & Nagarajaiah 2021
R-DLGA (NA)
8e (NA)
Xu & Zhang (2021)
DeepMod (rAdaLasso)
2e (36)
2.5e (36)
this work
CONCLUSION
In this paper, we show that no matter the method used to compute the derivatives (numerical or auto- matic differentiation), the design matrices used for PDE discovery violate the imeeresentabiiltty conditions (IRC) of the Lasso. This means irrelevant variables might be highly correlated with rel- evant To perform model variable selection under violated IRC, introduce randomised adaptive Lasso (r (rAdalasso), In addition, it allows to preserve a convex optimization problem and experimental results show can select the true model challenging cases at much lower computa- tional cost than state-of-the afrt approaches. Furthermore, once integrated in the deep learning model discovery framework DeepMod, the rAdaLasso allows to recover a wide variety of nonlinear and chaotic canonical PDEs up to (22) higher voise-to-sample ratios than state-of-the-ee algorithms. Finally, the hyperparameters used to perform the model discoveries are identical across experiments. These contributions pave the road towards truly automated model discovery.
Future work will focus on (1) performing discoveries by leveraging the data from several datasets using multitask learning and (2) including evolutionary approach in the proposed framework to build the library O automatically.
ACKNOWLEDGMENTS
Acknowledgements will be included in the final version.
REPRODUCIBILITY STATEMENT
As reported in the abstract of the paper, we share the data and code public repository to repro- duce our results. In addition,
for figure Appendix D.1 recalls the expression of the analytical solution.
for figure the repository allows to reproduce it.
for figure Appendix D.3 gives more details about the input data for DeepMod. The hyperparameters are identical across experiment and shared in Appendix[C] Our repository also contains files with the requirements to reproduce the experiments using python.
The notebooks shared the repository contain supplementary figures where or not the modified version DeepMod interpolates the data (especially for the chaotic Kuramoto- Sivashinsky equation).
how well
REFERENCES
Sutanu Bhowmick and Satish Nagarajaiah. Data-driven theory-guided learning of partial differential equations using simultaneous basis function approximation and parameter estimation (snape). arXiv preprint arXiv:2109.07471 2021.

---

Gert-Jan Both and Remy Kusters. Sparsely constrained neural networks for model discovery of pdes. arXiv preprint arXiv:2011.04336 2020.
Gert-Jan Both, Subham Choudhury, Pierre Sens, and Remy Kusters. Deepmod: Deep learning for model discovery in noisy data. Journal of Computational Physics, 428: 1099909,, 2021a.
Gert-Jan Both, Georges Tod, and Remy Kusters. Model discovery in the sparse sampling regime. arXiv preprint arXiv:2105.0044 2021b.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnn and Qiao
Zhang. JAX: composable transformations Python+NumPP programs, 2018. URL http:
//github.cc com
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932-39)) 3937, 2016.
Zhao Chen, Yang Liu, and Hao Sun. Deep learning physical laws from scarce data. arXiv preprint arXiv:2005.03448, 2020.
Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical learning with sparsity: the lasso and generalizations. CRC press, 2015.
Xiuting Li, Liang Li, Zuogong Yue, Xiaoquan Tang, Henning U Voss, Jürgen Kurths, and Ye Yuan. Sparse learning of partial differential equations with structured dictionary matrix. Chaos: An Interdisciplinary Journal of Nonlinear Science, 29(4):043130 2019.
W.X. and B. Fuchssteiner. Explicit and exact solutions to a kommgoooo-ptrrottiiisssssss equation. International Journal of Non-Linear Mechanics, 31(3):329 338. May 1996. ISSN 0020-7462. doi: 10. 10.100166020774222550006644 0020-7462 (95) 00064-X
URL http: dx. doi. 10.
Suryanarayana Maddu, Bevan L. Cheeseman, Ivo Sbalzarini, and Christian L. Müller. Stability selection enables robust learning of partial differential equations from limited noisy data. arXiv preprint arXiv: 000 2019.
Suryanarayana Maddu, Bevan I.. Cheeseman, Christian L. Müller, and Ivo Sbalzarini. Learn- ing physically consistent mathematical models from data using group sparsity. arXiv preprint arXiv:2012.06391, 2020.
Mikhail Maslyaev, Alexander Hvatov, and Anna Kalyuzhnaya. Data-driven partial derivative equa- tions discovery with evolutionary approach. Computational Science - ICCS 2019, pp. 635-641, 2019. ISSN 1611-3349. doi: 10.
61. URL http: dx doi. org/
10.7/079 300099888---002770000000 22750-2220000 61
Nicolai Meinshausen and Peter Bühlmann. Stability selection. Journal of the Royal Statistical Society: Series (Statistical Methodology). 72(4):417-43,, 2010.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg. et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
M Raissi, P Perdikaris, and GE Karniadakis. Physics informed deep learning (part ii): Data- driven, discovery of nonlinear partial differential equations,", arxiv e-prints, p. arXiv preprint 17X1v:1711.1056., 2017.
Samuel Rudy, Alessandro Alla, Steven Brunton, and Nathan Kutz. Data-driven identification of parametric partial differential equations. SIAM Journal on Applied Dynamical Systems, 18(2): 643-660, 2019.
Samuel H Rudy, Steven L Brunton, Joshua Proctor, and Nathan Kutz. Data-driven discovery of partial differential equations. Science Advances, 364)ee6602114, 2017.
10

---

Nayat Sanchez-Pi. Luis Marti, André Abreu, Olivier Bernard, Colomban de Vargas, Damien Eveil- lard, Alejandro Maass, Pablo Marquet, Jacques Marie, Julien Salomon, et al. Artificial intelligence, machine learning and modeling for understanding the oceans and climate change. In NewrlPS 2020 warkshop-Tuckling Climate Change with Machine Learning, 2020.
Hayden Schaeffer.
Learning partial differential equations via data discovery and sparse optimiza- tion. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473 (2197):20160446, 2017.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288, 1996.
Hao Xu and Dongxiao Zhang. Robust discovery of partial differential equations in complex situa- tions. arXiv preprint arXiv:2106.0000 2021.
Ye Yuan, Junlin Li, Liang Li, Frank Jiang, Xiuchuan Tang. Fumin Zhang, Sheng Liu, Jorge Goncalves, Henning Voss, Xiuting Li, al. Machine discovery of partial differential equa- tions from spatiotemporal data. arXiv preprint
2019.
Peng Zhao and Bin Yu, On model selection consistency of lasso. The Journal of Machine Learning Research. 7:2541-2563, 2006.
Hui Zou. The adaptive lasso and its oracle properties. Journal of the American statistical associa- tion, 101(476):141--422) 2006.
A PROOF OF PROPOSITION 1
Proof. by definition for p). vi 1/l We denote ÔT. vector of the subset of the design matrix θ that contains the relevant vectors and θF.5 a vector of the subset the design matrix that contains the irelevvnt vectors, then, θτ θr.//ux = 1 θr and θF, some column index of θr and j of θş. The projection of
Op,/i úu K OF, where an irrelevant vector onto the space of relevant vectors in the least-squares sense O Orद + Or. can be decomposed into the space of relevant vectors, Or,- = Ee Or.j + 6, where Er,j are the projections of a given irrelevant vector onto the relevant space. Using such decomposition the irrepresentability condition becomes, max, En |EFjAl < 77. By applying the transformation
due to the adaptive weights we get, θF., = Ex EFJ.A 16
θr + é IÊPu!?, which
can be reduced to θ.. En Eri
ÉF.i
&F θr + E lÊp,l" Now if we assume that since
ÊY is a relevant term and Ery is irrelevant one: |Er,a1 > 1ErØl- If in addition, 2 1, then, 0 ( which leads to,
ÊFU E
Σ
|FraM S Σ EFaul
(12)
which is equivalent the inequality
B STABILITY SELECTION WITH ERROR CONTROL.
With stability selection, variables chosen according to their probabilities of being selected with warranty on the selection error, Meinshausen Bühlmann (201)). The first step consists in finding the probability of variable of being selected under data perturbation: let 7 be one of B random

---

sub-samples of half the size of the training data drawn without replacement. For given À an estimation of the probability of & being selected is given by,
if (1)) >0
(13)
0
otherwise
where (I,) = (1))l1 for the randomised adaptive Lasso. Second, by computing the proba- bilities of being selected over given range of X's the stability paths be obtained, figure 3 That range is denoted A = Amazz; Amax], where is the path length and Amar is the regularisation parameter where all coefficients & are null. In Meinshausen & Bühlmann (2010) derive upper bound on the expected number of false positives VV) that can help determining a smaller re- gion where control on the selection error can be warrantied. By fixing this bound to EV the regularisation region becomes, see figure
- x A such that, RVV) S
A ===
S
F.V max
(14)
(2
1)p
where Tehr is the minimum probability threshold to be selected and qa is the average of selected variables. We propose here to approximate Q^ by Ộ사 Σ. |Sa| = Σn # Hgg Finally, the set of stable variables with an upper bound on the expected number of false positives is,
B
{Rx such that max f 2 Tthr for X E A-}
(15)
stable
C A SINGLE SET OF HYPERPARAMETER.
Stability selection expected number of false positives upper bound E 3, number of samples B 40 and the minimum probability to be selected Tehr = 0.9.
Library consists of polynomials and partial derivatives up to the fifth order leading to library size of =36 potential terms: {1, Ugz Uex, Ugrzzz: U, Utll:: ...y Migrzzz,.... ub, u uu...., u d'ugnaee)
Neural network architecture & optimiser NNs are 4 layers deep with 65 neurons per layer and sinus activation functions with a specific initialisation strategy, see Sitzmann al. (2020). The NNs are trained by Adam optimiser with learning rate of 10-5 and (0.99, 0.99).
Randomness Seeds are identical across datasets: (1) to initialise the NNs and (2) generate the noise vectors.
D ABOUT THE DATA
This appendix provides data source details to reproduce the examples of this paper.
D. ANALYTICAL LIBRARY OF THE KDV EQUATION
The Kortweg-de-Write Vries (KdV) u, -6uu, Ugna PDE analytical solution for 2 travelling solitons
is,
cosh(v/2t2/2) cinih((//555111
t(xxt) t) 2(c1-C2)
((V (02) cosh ( ( /2 (Vc + (o2) cosh ([ V
where C1 > C2
> 0, a = x Cit and & = x czt. 40 points equally dis-
tributed such that and ej 5, C2
12], 50 points equally distributed such that t E [-1 2. By automatic differentiation we obtain the 12 terms library: (1, Uny Maz; Muary U, WUg, Wurzz Wilzaa 2² u² u²uz, uZuza, u² Uxaz).
12

---

D.2 LIBRARIES FROM SPLINES/NUMERICA DIFFERENTIATION
Burgers, Ue Vulzr utg. shared on the github repository mentionned in Maddu et al. (2019). The solution here very similar to the one obtained using the analytical expression below using Dirac delta initial conditions.
Kuramoto-Sivan SSiaasik--Sv (KS), tionned in Rudy et al. (2017).
Ugza
Ugzzn
shared on the github repository men-
D.3 INPUT DATA FOR DEEP LEARNING EXPERIMENTS
We generate numerical solutions from several equations, on top of which add a Gaussian white noise,
Unoisy + a auu) Z
(16)
where Z ((0 1). The following PDEs are considered:
Burgers, initial condition: Dirac delta, analytical solution,
V (e uxx = nt
1) e aut
+ (e 1)erfe
Vavi
where A is constant and is the viscosity, 0.1, A and 40 points equally distributed such that x E [-,,3], 50 points equally distributed such that E [0.5,5] 5).
Voriie (KdV), see subsection D.I
Newell-Whiteheal (NW), 10uza + u(1 w²) 0.4, numerical solution using finite differences solver and the following initial condition:
u(x, 0) = Σ or sin((:
where o; and 3; are constants. points equally distributed such that x [0,39]. 50 points equally
distributed such that / E [0, 1.96) and a1 = 0.2, 02 0.8, 0.4, 31 12, B2 5,33 10.
Kuramoto-Sivashinkk (KS), see subsection D.2 2000 samples are randomly drawn from a subset of the dataset, details can be found our github repository, note 1
E ADDITIONAL RESULTS
Stability plots for case 2 comparison In this case the performance of PDE-STRIDE and rAdaLasso are compared on figure
DeepMod interpolations for the experiments see figure
F LIMITATIONS OF THE APPROACH
Incomplete library An obvious limitation of the approach is that the library of potential PDE terms must contain the true underlying PDE terms. Typically this be diagnosed report (in TensorboardX) the mean square error on a test set which allows to verify if the discovered PDE generalizes well not.
13

---

PDE-STRIDE
rendomised AdaLasso
1.0
1.0
selection probability 11
selection probubility IIA
0.8
0.8
0.6
EV mar
0.6
VVanr
0.4
0.2
0.2
0.0
0.0
10-3
10-2
10-5
À
(a)
(b)
Figure 5: Comparing PDE-STRIDE and the randomised adaptive Lasso selection performance challenging case: recovering Burgers' equation from a library built using polynomial interpola- tion from dataset with 4% noise Maddu et al. (2019). In (a), PDE-STRIDE solves relaxation of the best subset selection (lo regularisation) using an Iterative Hard Thresholding algorithm. (b), the stability plot for the randomised adaptive Lasso. The true underlying PDE be recovered by both methods by a proper tuning the error selection: EVmaz = 2. However, the computational cost PDE-STRIDE couple orders of magnitude higher (+r 122s) compared to the of for the randomised adaptive Lasso ( 1.30s).
Burgess
K&V
NW
Ripert date
origind DeepMMi
Addul rAdul..com Joon DeppMed
Figure DeepMod interpolations for the experiments described in the main text.
14

---

Non-unique solutions If the discovery problem has non unique solutions, approach will pro- pose one that might not be the one we were looking for. So far, we have devised two cases which the solutions not unique and present them in the next paragraphs. is known that the solution of single soliton from the KdV equation is also a solution of the simpler travelling wave equation: 04 -ctty, see Rudy al. (2017). So trying discover the underlying equation from analytical of KdV with a single soliton like,
e
u(x,t) t) = 2 coshlvccz
(17)
where с > O, will result in the discovery of u, = -Cd. We obtained a similar result with our approach while trying to find the underlying equation from data generated by solution of Fishers equation (2t- Uen + u(1 - u)) from Ma & Fuchssteiner 1996).
u(2 t) = (1 (V2 1)e -2 -
(18)
2/6' For both analytical solutions x and & play symmetric role explaining why Ut = Ug, where k is some constant.
where - Vx= and x
5
Coefficient bias in chaotic systems for the successful discoveries of chaotic PDEs, the efficient mean errors are typically below (< 2%). For the chaotic Kuramoto-Sivashinkk PDE, we show that with 50% noise and fraction of the data the mean eπror on the coefficients magnitudes with respect to PDE-FIND. However, chaotic systems sensitive their initial conditions and the coefficients errors (20%) of the discovered PDE are very large with respect to the ground truth. As result, the discovered PDE cannot be used for predictions but can be used identify the underlying physical processes.
recover the terms of the PDE with half
15


Reviews:
[{"review_id": "ntgDC753Isc", "reviewer": "Reviewer_pXwU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:** \n\nI appreciate the detailed explanation of the IRC, geometric intuition, and defining a diagnostics metric that they then applied to their examples. I knew that a challenge for this class of methods is having correlated columns and that this was a downside of having large libraries, but I didn't know what to do about it or a more thorough explanation. It's also nice when papers include motivation/intuition about their algorithmic improvements.\n\nThe experiments are able to demonstrate improvements while comparing to quite a few previous methods, including 2021 papers. I also appreciate this: \"To make sure the comparison is fair, we compare our approach with the ones from literature using the data from the original authors of those approaches. Furthermore, we restrict ourselves to cases where the original authors have tuned their algorithms and present the cases as being hard ones, see table 1.\"\n\n**Weaknesses:**\n\n*Rigor of Experiments*\n1. Why are there missing entries in Tables 2 & 3?\n\n2. \"During training, if Lmse on the test set does not vary anymore or if it increases, the sparsity estimator is triggered periodically.\" They shouldn't be using test error for this! A validation set could be used for this, though. Test sets should be held out until the very end if they are going to legitimately check for generalizability and overfitting. This raises questions about these claims:\n    - \"Our approach allows to recover all 4 PDEs without overfitting while the original DeepMod would for all, except for the KdV equation.\"\n    - From the appendix: \"Typically this can be diagnosed as we report (in TensorboardX) the mean square error on a test set which allows to verify if the discovered PDE generalizes well or not.\" \n\n3. A major point in this paper is being able to handle higher noise-to-sample ratios (and larger libraries) than previous methods, which is a great goal. It's not clear to me how these results (Table 3) were established, though. Did you run each algorithm on the same data and see how much the noise could be increased until the wrong terms are selected? Similarly, did you gradually increase the library size? Or was this table constructed by checking those papers and seeing what level of noise and library size they reported for these particular PDEs? If the latter, that could explain why there are some missing entries. However, that weakens this claim in a few ways: (1) These six papers might have varied the details of how they set up these datasets, (2) they might have reported results with some particular level of noise and library size without trying to push their method to the limits, and (3) if these papers were trying to push their method to the limits, they might have had differing criteria about what the limit is. For example, are they satisfied with selecting the right terms, or do they also want the coefficients to be correct within some tolerance?\n\n\n4. Related to the previous point, I'm concerned about this paper not reporting how accurate the coefficients are, either by reporting the learned coefficients or reporting their error. This is briefly touched on at the very end of the appendix without giving a lot of specifics. In contrast, the related papers that they cite typically report this information. Some also show how accurate the solution is if you solve the PDE with the estimated coefficients. Perhaps this algorithm chooses the correct coefficients more often than other papers but the coefficients are much less accurate, which would be a valuable trade-off to know about. \n\n\n*Correctness of other claims/statements:*\n\n5. \"It is anecdotaly known in the model discovery community that the Lasso does not perform very well compared to other relaxations of the original problem, see Rudy et al. (2017), Li et al. (2019), Rudy et al. (2019) and Maddu et al. (2019), however it has never been studied why. We trace back this lack of performance to the potential variable selection inconsistency of the Lasso.\" I don't think it's fair to say that this has never been studied. For example, Maddu et al. (2019) says \"For example, the studies in [43, 44] provide sufficient and necessary conditions, called the irrepresentable conditions, for consistent variable selection using LASSO, essentially excluding too strong correlations of the predictors in the design matrix.\" They then use this as motivation to use Randomized Lasso. \n\n6. The implication of meeting the IRC (or having this paper's diagnostic metric be < 1) seems to be inconsistently described, which confused me. I think that if IRC is violated, the Lasso could still be correct, right? Then these two sentences are a little too strong.\n    - \"For both libraries [the diagnostics metric] > 1 revealing the Lasso would not be able to recover the true support.\" \n    - [The diagnostic metric after adaptive Lasso is greater than 1] \"explaining why the adaptive Lasso alone will not work in those cases.\"\n\n7. \"In addition, it allows to preserve a convex optimization problem and experimental results show it can select the true model in challenging cases at much lower computational cost than state-of-the-art approaches.\" This is overstated - they only show an improvement in computational cost for one example & one method.\n\n8. This part of the proposition seems imprecise \"The design matrix Θ ̃ will therefore have more chances to verify the IRC than Θ.\" \n\n\n*Novelty:*\n \n9. While this paper definitely contains novelty and should (with some revision) be published somewhere, I'm not sure it's enough novelty for this venue. In particular, although I definitely appreciate the deeper discussion of IRC, the Maddu et al. (2019) (PDE-STRIDE) paper also discussed it and used it for motivation to use Randomized Lasso and stability selection. This paper then adds variable selection error control and changes from Randomized Lasso to Randomized Adaptive Lasso. They then show improved results compared to PDE-STRIDE. Similarly, they show that incorporating rAdaLasso with stability selection and error control into DeepMod gives improved results over DeepMod. So, this seems worth publishing but doesn't seem like a big step in novelty to me. \n\n\n\n**Minor feedback**, no need to address in discussion phase:\n- I found it a little confusing to compare Tables 2 & 3. I'm guessing the noise level changed. Did anything else change about these test cases? \n- I thought that \"up to O(2) higher noise-to-sample ratios\" was confusing notation. I think you want to say \"two orders of magnitude,\" but I'm not familiar with using O(2) to say that. \n\n- \"Conversely, the Lasso will not consistently select the true variables for any design matrix, even if they are present in the library.\" This sentence is unclear to me. By \"conversely\", do they mean that without IRC, the Lasso will not consistently...?\n- This sentence is hard to read: \"Our approach allows to recover all 4 PDEs without overfitting while the original DeepMod would for all, except for the KdV equation.\" \n- Figure 6 in the Appendix doesn't have much of a description, so I'm not sure what's happening. \n\n~~~~\n\nI should note that I did not check the proof, since it's in the Appendix. ", "summary_of_the_paper": "This paper describes an improvement to previous methods for discovering PDEs from data. They show that even for clean data, the design matrices for discovering PDEs using sparse regression can violate irrepresentability condition (IRC), which motivates their approach. They introduce randomised adaptive Lasso (rAdaLasso) and incorporate stability selection and error control. They demonstrate this approach for discovering PDEs from data that was numerically differentiated, comparing to two previous methods on two tricky PDE test cases. They also incorporate rAdaLasso with stability selection and error control into DeepMod (an existing model discovery framework that uses deep learning and avoids numerical differentiation) and show with four PDEs that this is an improvement over the original DeepMod method. Then they compare their version of DeepMod to six previous papers on two PDEs, showing the ability to tolerate more noise and larger libraries. ", "summary_of_the_review": "My primary concern is the rigor of the experiments, and my secondary concern is that some claims are overstated. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iGdQx3KEztr", "reviewer": "Reviewer_kL72", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is generally well written and the problem of variable selection when modelling dynamical systems is well motivated. \nI have following reservations about this: \n- Diagnostic tool developed in the paper needs $\\Theta_T$. It seems to me that It does not have much application beyond the PDEs where we know true variables.\n- In proposition 1, it's not clear why the assumption that relevant coefficients are larger than irrelevant coefficients. Since this assumption is the basis of the proof, it's important to clarify when this assumption will be met.\n- The onus on the choice of randomisation used in rAdaLasso ($\\beta (1, 2)$.) is not very clear to me. How doe the performance change if I change this distribution and how does that affect the performance of the rAdaLasso?\n- In Figure 4, authors claim MSE on test in has poor generalisation. I don't understand why the blue curve (corresponding to the rAdaLasso) stops after certain number of epochs in 3 examples. If we only compare the performance of rAdaLasso when they are trained for same number of epochs, their performance is comparable.\n- Finally empirical evaluation in a more realistic example where one doesn't know the true support of the design matrix. I understand that computing the selection error on such examples is difficult, one can still look at the MSE loss for comparison.  \n\n", "summary_of_the_paper": "This paper deals with the problem of model discovery based on sparse regression. Authors propose a new randomised adaptive lasso for that selects the terms to be used in the PDEs selection that is used in conjunction with Deep learning framework. Authors then experimentally verify on 4 PDEs that their method perform better for model selection and at least as well when looking at MSE loss. \n\nFor a detailed summary, authors first verify the violation of IRC which is defined as in eq (1) for Lasso on an analytic PDE (Korteweg-de-Vries (KdV) equation). As their first contribution, authors develop a diagnostic metric that indicates the violation of Lasso when one know the true support of the design matrix.\nIn proposition 1, authors show that under certain assumptions on weights, the adaptive Lasso, where one reweighs the design matrix has better diagnostic score than Lasso. \nSince the design matrix is typically approximated, if we incur a deterministic error term, authors empirically show that adaptive Lasso is also susceptible to the violation of the IRC. In lieu of this experiment, authors propose randomised adaptive Lasso, where each column of the design matrix of adaptive Lasso is randomly scaled with $\\beta(1,2)$. Author integrate it in deep learning model discovery framework and empirically verify the performance of of rAdaLasso for 1) MSE loss and 2) model selection error.   ", "summary_of_the_review": "Authors proposed a heuristic algorithm, randomised adaptive Lasso, that is evaluated empirically on 4 analytic PDEs. In those specific cases authors show success of their algorithm in model selection problem. Having said that I still recommend to reject this submission because I think the empirical evaluation is limited to PDEs where we know the True coefficients and generalisation on more difficult problems in unknown. And finally, the limited theoretical justification behind the work has some strong assumptions which are not sufficiently discussed in the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5Ho_9WZLer_", "reviewer": "Reviewer_dRBM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1) A drawback of sparse regression based equation inference is exposed: the dictionaries can violate the irrepresentability condition (IRC) of Lasso, even when derived from analytical PDE solutions.\n2) To alleviate the drawback, a randomised adaptive Lasso (rAdaLasso) is proposed and intergated within the DeepMod framework with promising empirical performance. \n\nWeakness:\n\n1) (Technical novelty). The technical novelty of the proposed rAdaLasso seems incremental to random Lasso. It is suggested to tell the readers what is really new in the proposal. \n2) (Theoretical depth). The proposed rAdaLasso is shown to perform better when IRC is violated. However, the theoretical analysis of the underlying machenism is insufficient. It is possible to theoretically characterize how better the proposed rAdaLasso can perform? ", "summary_of_the_paper": "This manuscript studies the variable selection consistency in model discovery of PDEs. A randomised adaptive Lasso (rAdaLasso) with stability selection and error control algorithm is proposed to recover the true underlying PDE in the presence of design matrices\nthat are highly correlated and violate the IRC. Better model recovery performance is obtained by integrating rAdaLasso within the deep learning model discovery framework DeepMod. ", "summary_of_the_review": "The problem of variable seclection consistency of model discovery in PDEs is studied by first pointing out that sparse regression may fail with violated IRC and then proposing a randomised adaptive Lasso to tackle it. To me, this work is interesting. However, the proposed rAdaLasso seems not quite novel, and its relevant theroetical analysis seems a bit weak. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1WeMFbS5kN", "reviewer": "Reviewer_1UJP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Post-response update. The author response clarified my technical comments. My opinion has not changed: the paper still has insufficient empirical exposition and experiments.\n\n-----------\n\nThe paper is well written, and presents both the problem and its solution relatively clearly. The IRC problem is well presented and motivated, but I would have liked to see more exposition on the context. How does IRC connect with collinearity (or other types of identifiability issues) and identifiability in general? Giving more context would clarify the scope of the paper. \n\nThere is considerable literature on SINDy style methods, and the field is already mature and well-developed. The SINDy papers usually have extensive experimental sections: the original SINDy paper did 6 systems with and without noise, and later papers have expanded on this.\n\nSince the new lasso method should be universally applicable to any SINDy type method, one then naturally expects to see experiments on applying the new radalasso to most systems already considered in the literature, and showing improvement. One would expect to see ablation studies comparing the performance as a function of data size, noise levels or system complexity/stifness to explain in which situations or regimes the old/new lasso method fails/succeeds. \n\nUnfortunately, the paper does not consider ODEs at all; limits the experiments to few PDEs; and doesn’t do any ablation studies. This limits the impact of this paper significantly. (If one only considers PDEs, the title is then overly broad as well!). The DeepMod experiment is interesting, but requires explaining the resulting difference by eg. varying the noise or data size. The results are also a bit ambiguous, it seems that selection error is similar for the two methods, yet the identified systems seem quite different. The experiment of table 3 shows good results, especially with KS.\n\nThe two or four systems considered are not sufficient to show that the new method consistently outperforms SOTA.\n \nThe paper needs to consider more previous systems, and show that (i) it performs effortlessly in “easy” systems, (ii) it performs well for many “difficult” systems, and (iii) it works for both ODEs and PDEs and within neural methods, (iv) how it performs on various levels of noise and data size. Also studying how the lasso behaves under varying system \"complexities\" would be useful exposition to the problem.\n\nTechnical comments\n* Fig 1: This is excellent figure, and shows the adalasso performs well. However, the introduction stated the goal of the paper to show that lasso fails even if one takes n -> infinity. This figure should then also present the lasso/adalasso behavior under different data counts: does the problem persist also in high-data regimes?\n* Sec 2.2. claims that ridge regression “handles” multicollinearity. It is unclear what multicollinearity means in this context, what it means to “handle” something, and why ridge handles it. Ridge regression finds dense solutions, and I fail to see how this handles collinearity. Please clarify.\n* Eq 4 is vague. We optimise \\tilde\\xi, yet \\tilde\\xi is defined to be w\\xi. So then one needs to either optimise w or \\xi instead (or both). I assume that one ultimately is optimising \\xi, since w’s are found as preprocessing. Please clarify\n* Can you clarify if the adaptive lasso is off-the-shelf, or a some new variant of it.\n* Prop 1: What does “relevant coefficient” mean? Does this refer to \\xi's or something else? \n* Sec 2.3. claims the design matrix has to be estimated. I’m confused: isn’t the point of SINDy type models that one enumerates “all\" combinations of the covariates and derivatives, and the \\xi then selects from them. What does the “estimation” here refer to? What’s the role of the splines here? Is this referring to non-stationary bases?\n* Eq 6: I don’t understand why one corrupts the underlying terms, instead of corrupting the coefficients \\xi. Generally this is confusingly written, and its unclear what the neural networks are doing here. They were not discussed previously, and NNs are not present in eq 4. Please this section, and show how the neural network plays into equation 4.\n* Fig4: what is the x-axis of top row? What do the equations in top row mean? What is k? What is select. error (please define). ", "summary_of_the_paper": "The paper discusses the unidentifiability of lasso-based differential equation discovery, and proposes to use a more stable lasso variant (randomized adaptive lasso), which can more accurately identify PDE coefficients. The method is well-motivated and adresses a major bottleneck in differential discovery. The method is correctly presented, and the initial results are excellent. ", "summary_of_the_review": "In summary the method is a great contribution to PDE discovery methods, but requires a much more comprehensive experimental section to convincingly demonstrate its benefits under different settings and demonstrate insight into its behavior. Currently the paper feels premature for publication, but no doubt will become an excellent paper in future.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers and AC all agree that the paper considers an important problem but that several concerns remain which makes the present submission of limited novelty. 
We strongly encourage the authors to revise their manuscript to incorporate the reviewers comments as this will significantly strengthen the significance of their work. 
In particular it will be important to strengthen the theoretical analysis and expand the empirical evaluation, including incorporating an ablation study and considering settings of various difficulty, noise level, etc.
Decision: Reject