Fully differentiable model discovery
Authors: Gert-Jan Both
Remy Kusters
Venue: ICLR 2022 Submitted (2022)
Keywords: Model discovery
Sparse Bayesian Learning
Normalizing Flows

Abstract: Model discovery aims at autonomously discovering differential equations underlying a dataset. Approaches based on Physics Informed Neural Networks (PINNs) have shown great promise, but a fully-differentiable model which explicitly learns the equation has remained elusive. In this paper we propose such an approach by integrating neural network-based surrogates with Sparse Bayesian Learning (SBL). This combination yields a robust model discovery algorithm, which we showcase on various datasets. We then identify a connection with multitask learning, and build on it to construct a Physics Informed Normalizing Flows (PINFs). We present a proof-of-concept using a PINF to directly learn a density model from single particle data. Our work expands PINNs to various types of neural network architectures, and connects neural network-based surrogates to the rich field of Bayesian parameter inference.

Under review as a conference paper at ICLR 2022
FULLY DIFFERENTIABLE MODEL DISCOVERY
Anonymous authors
Paper under double-blind review
ABSTRACT
Model discovery aims at autonomously discovering differential equations under- lying dataset. Approaches based on Physics Informed Neural Networks (PINNs) have shown great promise, but a fully-differentiable model which explicitly learns the equation has remained elusive. In this paper we propose such an approach by integrating neural network-based surrogates with Sparse Bayesian Learning (SBL). This combination yields robust model discovery algorithm. which we showcase on various datasets. We then identify a connection with multitask learn- ing, and build on it to construct a Physics Informed Normalizing Flow (PINF). We present a proof-of-concept using a PINF to directly learn density model from single particle data, Our work expands PINNs to various types of neural network architectures, and connects neural network-based surrogates to the rich field of Bayesian parameter inference.
INTRODUCTION
Many physical, chemical and biological systems be modelled with (partial) differential equa- tions. They capture a system's interactions, scales and conserved quantities in an interpretive man- ner. Unfortunately, manually deriving these equations from first principles process and requires expert knowledge of the underlying dynamics. The last few years have seen a rising interest in automating this process, also known as model discovery. As the model space is exponentially large, a popular approach is select a large set of candidate terms (features) and perform sparse regression on these features, effectively turning model discovery into variable selec- tion (Brunton et al. 2016 Rudy et al.] (017). A uniquely challenging aspect of discovering PDEs is that many of the candidate features contain higher-order derivatives, which are challenging to calcu- late accurately using numerical differentiation. This essentially limited model discovery to densely sampled datasets with low noise levels.
time-consuming
Various works have explored the use of neural networks to generate surrogate of the data (Rasheed ct al. 2020), and perform model discovery on this surrogate instead (Berg & Nyström 2019] Both al. 2019). By using neural network denoises the data, while automatic differentiation can be used to accurately calculate the (higher-) order derivatives of used in the candidate features. These approaches show significant improve- ments when the neural network is constrained to solutions allowed by the candidate features. This essentially yields a Physies Informed Neural Network (PINN) (Raissi al. 2017). with the impor- tant distinction that the form of the constraint, i.e. which terms of the candidate features are active and make up the underlying equation, is also learned. As it is composed of all candidate features, the constraint itself is prone to overfitting, and the discovered equation will contain more terms than re- quired. Applying & regularisation can alleviate this problem, but raises the question of how strongly to apply it. Alternatively, inactive terms can be pruned from the constraint by applying mask, but this is a non non-differentiable operation, and training the network does not take sparsity into account. The open question then is how optimally apply constraint, which itself is learned and sensitive to overfitting, all the while remaining fully differentiable.
learn the data such that g(x. L), the network
In this work introduce a fully differentiable model discovery algorithm consisting of neural- network based surrogate with a constraint based on Sparse Bayesian Learning (SBL). We summarise contributions follows:

---

Under review as a conference paper at ICLR 2022
We show how Bayesian parameter inference methods can be used constraint in PINN. Specifically, we use Sparse Bayesian Learning (SBL) to create a dillleeifferet,ibbee, robust model discovery algorithm and showcase this on various datasets.
We identify connection with multitask learning using uncertainty and exploit this to gen- eralise PINNs to probabilistic surrogates. We introduce conditional normalizing flow constrained by SBL, called Physies Informed Normalizing Flow (PINF).
We present proof-of-conee where PINF learns time-dependent density from unla- belled Linle-particle data, allowing the constraint discover density model directly from single particle data.
2 BACKGROUND
Model discovery with sparse regression Model discovery aims to discover the PDE from large set of M candidate features {u, Uzz, u. Assuming the underlying equation can be written as a linear combination of the candidate features, model discovery can be approached as regression problem (Brunton al. 2016) by solving
= min min |0
AEll XR(E),
where @ NNX contains all candidate features, & E RMM the unknown coefficient vector and R(E) some pransity-promoting penalty; the number of candidate features is typically much larger than the number of terms in the underlying equation. The main challenge of discovering the under- lying equation using this approach is dealing with large, possible correlated errors in the features containing derivatives; using numerical differentiation to calculate these higher-order derivatives accurately from noisy and sparse data is extremely challenging, even after denoising. One line of work has focused on constructing more robust and sparser approaches to solving eq. [ for example SR3 (Zheng et al. 2019) or using stability criteria (Maddu et al., (020). Alternatively, several works (Both et al. 2019 Berg & Nyström 2019) have explored the use of neural networks to create a sur- rogate of the data and perform model discovery on this surrogate instead. Automatic differentiation can then be used to calculate the derivatives, yielding much more accurate features.
PINNs Physics Informed Neural Networks (PINNs) (Raissi et al. 2017) have become a very pop- ular method either solve differential equation perform parameter inference with neural net- works. Here we focus on parameter inference and consider a (noisy) dataset {u, 0; 4 [ gov- erned by differential equation of form 0.u X u with X the terms of the equation and ws the unknown parameters. PINNs infer w by using neural network to approximate u, and constrain the network to the given differential equation by minimising
1 CPINN(O, w) = N Σ l
u,
Σ
10:0; X,w||
Here ù; ge v0((,, (r, t2) is the prediction of the neural network and sets the strength of the regu- larisation. The constraint ensures the network approximates the data consistently with the given differential equation, and terms containing derivatives can be calculated using automatic differenti- ation. These two features make PINNs especially useful in noisy and sparse datasets.
Model discovery with PINNs PINNs can easily be adapted to perform model discovery by re- placing the given differential equation X with a larger set of candidate features O Additionally, mask m is applied to the coefficients, yielding as loss function.
N
LMD (0.E) = Σ
1
L
u1l1
Jcù - O.(m O 2
==1
( + XCreg E)
The mask deseribes which terms feature in the equation, and hence the form of the constraint; this approach can be interpreted as a PINN in which the constraint also learned. The mask is updated periodically by some sparse regression technique, and terms are pruned, the constraint becomes
2

---

Under review as a conference paper at ICLR 2022
stricter, preventing overfitting of the constraint itself and improving the approximation of the net- work, boosting performance significantly (Both et al. 2021). However, the con-differentiability of the mask lead to issues during training, for example when it is updated at the wrong time, or when the wrong terms are accidentally pruned. Our goal here is thus to construct an approach where the mask M is learned together with the networks parameters, while still maintaining the benefits of iteratively refining the approximation: a fully-differeentabl model discovery algorithm.
Removing free variables Training with eq. 3 optimises two sets of parameters: the network parameters and the coefficients &. Typically both are minimised together using gradient descent, but the optimisation of & can be performed analytically (Both et al. 2021). Given configuration of the network parameters 0*, the minimisation over
regression problem given by eq. and can be solved exactly. Referring to this solution as the maximum likelihood estimate ÉMLE. we define a loss function LMD (0) = LMD(e, ÉMLE), which optimises only the network parameters using gradient descent. This significantly speeds up convergence and reduces the variance of the discovered coefficients across initialisations. We shall adopt this approach in the rest of the paper, and define the convention of L denoting the loss function L with the independent variables calculated analytically.
3 FULLY DIFFERENTIABLE MODEL DISCOVERY
Our goal is to create fully-eifferemtiable model discovery algorithm, which, considering eq. requires making the mask m differentiable. Differentiable masking is challenging due the binary nature of the problem, and instead we relax the application of the mask to regularisation prob- lem. Specifically, we propose to use Sparse Bayesian Learning (Tipping) (001) select the active features and act constraint. We start this section by reviewing SBL and how it can be used for differentiable variable selection, next show to integrate it in PINNs and finally introduce Physics Informed Normalizing Flows.
3.1 DIFFERENTIABLE MASKING WITH SBL
Sparse Bayesian Learning Sparse Bayesian Learning (SBL) (Tipping, 2001 is Bayesian ap- proach to regression yielding sparse results. SBL defines a hierarchical model, starting with a Gaus- sian likelihood with noise precision 3 = a-2 and zero-mean Gaussian with precision oy on each component Ey as prior.
N p(O,ú: O.E.3) II N( O
(4)
M
p A) = II N(G; 0,a5))
(5)
with o. E RN, θ E RNXM & E RMM, and we have defined A = diag(a). The posterior distribution & is Gaussian with mean μa and covariance E.
Σ (3e - # ToO 0,ú.
Many of the terms in A will go to infinity when optimised, and correspondingly the prior for term j becomes a delta peak. We are thus centain that that specific term is inactive and can be pruned from the model. This makes SBL very suitable choice for model discovery, as it gives a rigorous criterion for deciding whether term is active or not. Additionally defines hyper-priors over a and B,
M [aa; a. 1=1
p(a) II
p(3) = 1(3; c.
3

---

Under review as a conference paper at ICLR 2022
The inference of A and 3 cannot be performed exactly, and SBL uses type-1l maximum likelihood find the most likely values of A and B by minimising the negative log marginal likelihood
LSBL (A, = 2 [ 1lue - 0ull + 1²" Au log Σ
log A
N Nlog |3]
(a log aj box)) clog B + d3, (9)
using an iterative method (see Tipping (2001)
Continuous relaxation The marginal likelihood also offers insight how the SBL provides differ- entiable masking. Considering only the first two terms of eq. 9
3 I|u 0ul 4² Ap
(10)
we note that the SBL essentially applies coefficient-specine &2 penalty to the posterior mean μa. If Aj + oc, the corresponding coefficient μı - 0, pruning the variable from the model. Effectively, the SBL replaces the discrete mask m, E {0, by a continuous regularisation Aj E (0,00), and approach continuous relaxation.
thus refer
3.2 SBL-CONSTRAINED PINNS
Model To integrate SBL as a constraint in PINNs (similar to eq. on the output the neural network,
2 we place Gaussian likelihood
N à p(u: ú, = II N M(u; ú; T-1),
(11)
i=1
and define a Gamma hyper prior on t, p(T) = [(r; e, f), yielding the loss function,
Canta (0, 1 uu ûll N Nog 7 log T + fT.
(12)
Assuming the likelihoods factorise, i.c. p(u, uti u, O {) = p(u; u) p(u:: O E). SBL can be integrated constraint in PINN by simply adding the two losses given by eq. 9 gladd eq. 12
LSBL-PINN O, A, B) = Laaa (0, + (0,
(13)
Our approach does not rely on any specific property of the SBL, and thus generalises to other Bayesian regression approaches.
Training The loss function for the SBL-constranedd PINN contains three variables which can be exactly minimised, and denote these as A, f and B. With these values, we introduce LSBL- = ESBL-PINN (0, Á, f, () and note that can further simplify this expression the gradient of the loss with respect these variables is zero. For example, VoL(A) v VoAl as V A O. Thus, keeping only terms directly depending on the neural network parameters yields,
7 ESBL-INN (0) 1|u á||
+ 114-111p -pul + 1 log
N+
B
(14)
(i)
& (0
O H1)
+H² Ap log Σ
2
=Cmox(0.1)
"Neglecting the hyper-prion this loss function can also written more compactly as LSBL (A, 3) = log |C| t C=
C=8-11+44'00
(8)
but the format provides more insight how SBI provides differentiahle masking.

---

Under review as a conference paper at ICLR 2022
where in the second line have rewritten the loss function in terms of classical PINN with relative regularisation strength À = 3/f f and coefficients ç = Contrary to a PINN however, the regularisation strength is inferred from the data, and the coefficients μ inherently sparse.
An additional consequence of VoC(A, gating through the solver. While such an operation could be efficiently performed using implicit differentiation (Bai al. 2019), our method requires solving an iterative problem only the for- ward pass. During the backwards pass the values obtained during the forward pass can be considered constant.
0 is that our method does not require backpropa-
Connection with multitask learning Considering eq. 14 note the resemblance to multitask learning using uncertainty, introduced by Cipolla et al. 2018). Given a set of objectives, the au- thors propose placing Gaussian likelihood on each objective so that each task gets weighed by its uncertainty. The similarity implies that we are essentially reinterpreting PINNs as Bayesian or hierarchical multi-task models.
3.3 PHYSICS INFORMED NORMALIZING FLOWS
Having redefined the PINN loss function (eq. in terms of likelihoods (i.e. eq. 14 allows intro- duce a PINN-type constraint to any architecture with probabilistic loss function. In this section we introduce an approach with normalizing flows, called Physies Informed Normalizing Flows (PINFs). As most physical equations involve time, we first shortly discuss how to construct a time-dependent normalizing flow. We show in the experiments section how PINFs can be used directly infer density model from single particle observations.
Conditional Normalizing Flows Normalizing flows construct arbitrary probability distributions by applying series of K invertible transformations f to a known probability distribution Tz),
z = fK .... .... fo(æ)) go
Σ log det 0fx(z)
(15)
logp(x) = log (z) +
Odz
and are trained by minimising the negative log likelihood, LNF = - log p(z). Most phys- ical processes yield time-dependent densities, meaning that the spatial axis is proper probabil- ity distribution with J p(x, t)dx 1. Contrarily, this not valid along the temporal axis, as ((xx,t))t t)dt = fx) To construct PINFs, we first require a Conditional Normalizing Flow ca- pable of modelling such time-dependent densities. Instead of following the method of Both & Kusters (2019). which modifies the Jacobian, we employ time-dependent nyper-network. This hyper-network outputs the flow parameters 0, and only dependent time, i.e. h(t), thus defining a time-dependent normalizing flow as z 9h(e)(2:)
PINFs Conditional normalizing flows yield a continuous spatio-temporal density, and the loss function of a PINF is defined simply adding the SBL-loss to that of the normalizing flow, yielding
NO Lan (0, p1) + nTT Åμ log Σ
ÉPINF (0) = (NF(() +
(16)
2
EXPERIMENTS
We now show several experiments illustrating our approach. We start this section by discussing choice of hyperprior, followed by benchmark several datasets and finally proof-of-concept with physics-informed normalizing flows.
4.1 CHOOSING PRIOR
The loss function for the SBL constrained approach contains several hyper- parameters, all defining the (hyper-) priors respectively A, and r. We set uninformed priors A and B, b

---

equation from Under review as a conference paper at ICLR 2022
e= f=f= 1e but those B. the precision of the constraint, must be chosen more carefully. Figure iillustrates the learning dynamics on a dataset of the Korteweg-de Vries equation 2 when the 3 hyperprior is uninformed, i.c. d = 1e 6 Observe that the model fails to learn the data, while almost immediately optimising the constraint. We explain this behaviour as a consequence of our assumption that the likelihoods factorise, which implies the two tasks of learning the data and applying the constraint are independent. Since the constraint contains much more terms than required, can fit a model with high precision any output the neural network produces. The two tasks then are not independent but conditional: high precision on the constraint is warranted only if the data reasonably approximated by the neural network. To escape the local minimum observed in figure [] we couple the two tasks by making the hyper-prior on B dependent on the performance of the fitting task.
Dynamic prior Our starting point is the up- date equation for 3 (see Tipping 2001) for de- tails),
10-3
N M M M an 2c
8
(17)
10-5
Nrog 2d
Ceg
We typically observe good convergence of nor- mal PINNS with \ 1, and following this implies 3 i, and similarly Leg + as the model converges. Assuming N >> M + E; ozZ we have
10-7
Uninformed Dynamic
10-9
N + 2c
10-3
10 Cnt
10-1
(18)
2d
which can be satisfied with c= N/2, dd-n n/F. Figure[] that with this dynamic prior the SBL constrained PINN does not get trapped in local minimum and learns the underly- ing data. We hope to exploit multitask learn- ing techniques optimize this choice in future work.
Figure 1: Regression loss function of fitting loss during training, comparing an uninformed prior with dynamic prior.
4.2 EXPERIMENTS
We present three experiments benchmark our approach. We first study the learning dynamics in- depth on a solution of the Korteweg- de Vries equation, followed by a robustness study of the Burgers equation, and finally show the ability to discover the chaotic Kuramoto-Shivasiee highly noisy data. Reproducibility details can be found in the appendix.
Korteweg-de Vries The Korteweg-de Vries equation describes waves in shallow water and given by u, Uxzx Wig. Figure 2 shows the dataset: 2000 samples with 20% noise from two-soliton solution. We compare our approach with I) Sparse Bayesian Learning with features calculated with numerical differentiaaton. II) a model discovery algorithm with PINNs, but non- differentiable variable selection called DeepMoD (Both & Kusters 2021) and III) PDE-find (Rudy et al.) 2017), popular model discovery method for PDEs based on SINDY (Brunton et al. 2016)- The first two benchmarks also act as an ablation study: method uses the same regression algorithm but does not use neural network interpolate, while method II uses a neural network to interpo- late but does not implement differentiable variable selection. In figure 2 and we show that the differentiable approach recovers the correct equation after approximately 3000 epochs. Contrarily, DeepMoD recovers the wrong equation. Performing the inference 10 times with different seeds shows that the dallydiiferrentiale approach manages to recover the Kortweg-de Vries equation nine times, while DeepMol) recovers the correct equation only twice- worse, it recovers the same wrong equation the other times. Neither PDE-find nor SBL with numerical differentiation is able to discover the Korteweg-de Vries equation from this dataset, even at 0% noise due to the data sparsity.
We choose to plot the losses of the original PINN loss Las and Lrez because these are more easily inter- preted than the likelihood-based losses we have introduced.

---

Under review as a conference paper at ICLR 2022
a)
b)
10
-
10
100
au
0
2
4
2
4
Epoch (103)
Epoch (103)
Figure 2: Comparison of differentiable SBL-constrained model and an non-differentiable OLS- constrained model on Korteweg-de Vries dataset (panel a) with library consisting of 4th order derivatives and 3rd order polynomials, for a total of 20 candidate features. In panel and e we respectively plot the inferred prior A and the posterior coefficients μ. In panel we show the non- differentiable DeePyMod approach. In panels and line: Uga, bold orange line: wlz)) discovered early on, while the non-differentiable model (panel d) selects the wrong terms.
that the correct equation (bold blue
a)
0.0
0.0
w -0.5
0.5
-1.0
-1.0
50
100
0
50
100
noise (%)
noise (%)
Figure 3: Exploration of robustness SBL-constrained model for model discovery for the Burgers equation (panel a). We show the discovered equation over range of noise for DeepMoD (panel b) and the approach presented this paper (panel c). The bold orange and blue lines denotes uy and Wug, and black dashed line their true value.
Burgers We now explore how robust the SBL-constrained PINN with respect to noise on dataset of the Burgers equation, u4 Pizz
(figure 3)). We add noise varying from 1% to 100% and compare the equation discovered by benchmark method (DeepMoD, panel b) and approach (panel c) the bold orange and blue lines denote uzz and uu, respectively, and the black dashed line their true value. Observe that DeepMoD discovers small additional terms for 50% noise, which become significant when noise > 80%. Contrarily, approach discovers the same equation with nearly the same coefficients across the entire range of noise, with only very small additional terms (0(10-1) Neither PDE-find nor SBL with numerical differentiation able to find the correct equation on this dataset at 10% noise or higher.
fully differentiable
Curamoto-Shivashinske The Kuramoto-Shivvshiine Shivashinksy equation describes flame propagation and given by 14 = Wtlly Mux Urrzzz The fourth order derivative makes it challenging learn with numerical duffereentiation-asee methods, while its periodic and chaotic nature makes it challenging to learn with neural network based methods (Both al. 202I). We show here that using the SBL- constrained approach we discover the KS-equation from only small slice of the chaotic data (256 in space, 25 time steps), with 20% additive noise. We use a tanh-activated network with layers of 60 neurons each, and the library consists of derivatives up 5th order and polynomials up fourth order for total of thirty terms. Additionally, we precondition the network by training without the constraint for 10k epochs.

---

Under review as a conference paper at ICLR 2022
a)
10
10
b)
wwmmmm
10a
10 15
10
Epoch (10*)
Epoch (10*)
equation. We show the chaotic data and a cross section in panels a and b. The periodicity makes this a challenging dataset to learn, requiring 200k iterations to fully converge before it be recovered (panel c). Panels d and show that the posterior and MILE of the coefficients yield nearly the same coefficients, indicating that the network was able construct extremely accurate approximation of the data.
Figure 4: Recovering the
Training this dataset to convergence takes significantly longer than previous examples, as the net- work struggles with the data's periodicity (panel b). After roughly 70k epochs, clear separation between active and inactive terms is visible in panel c, but it takes another 30k epochs before all in- active terms completely pruned from the model. Panels d and show the corresponding posterior and the maximum likelihood estimate of the coefficients using the whole library. Remarkably, the MLE estimate recovers the correct coefficients for the active terms, while the inactive terms all nearly zero. In other words, the accuracy of the approximation is so high, that least squares identifies the correct equation.
4.3 MODEL DISCOVERY WITH NORMALIZING FLOWS
Consider a of particles whose movement is described by micro-scale stochastic process. In the limit of many of such particles, such processes can often be described with deterministic macro-scale density model, determining the evolution of the density of the particles over time. For example, a biased random walk can be mapped to an advection-diffusse scale density models are typically more insightful than the corresponding microscopic model, but many (biological) experiments yield single-particle data, rather than densities. Discovering the un- derlying equation thus requires first reconstructing the density profile from the particles' locations. Classical approaches such as binning or kernel density estimation are either oon-iffeerntiable. non- continuous or computationally expensive. Normalizing Flows (NFs) have emerged in recent years as flexible and powerful method of constructing probability distribution, which is similar to density estimation up to a multiplicative factor. In this section we use physics informed normalizing flows learn PDE describing the evolution of the density directly from unlabelled single particle data.
equation. The macro-
Since the conditional normalizing flow is used to construct the density, a precision denoting the noise level does not exist, and instead a flow consisting of ten planar transforms (Rezende & Mohamed, (013)) and a hyper-network of two layers with thirty neurons each. The dataset consists of 200 walkers on biased random walk for 50 steps, corresponding to an addvetion-difffsion diffusion model, with an initial condition consisting of two Gaussians, leading to the density profile shown in figure 5p. The two smallest terms in panel e correspond to the advection (bold green line) and diffusion (bold red line) term, but not all terms are pruned. Panels b, с and compare the inferred density (red line) to the true density (dashed black line) and the result obtained by binning. In all three panels the constrained NF is able to infer fairly accurate density from only 200 walkers. We hypothesise that the extra terms are mainly due to the small deviations, and that properly tuning the prior parameters and using more expressive transformation would prune the remaining terms completely. Nonetheless, this shows that NF flows can be integrated in this fully differentiable model discovery framework.
as prior for (a N, N 10-9). We consider
8

---

Under review as a conference paper at ICLR 2022
a)
b)
10a
10a
10
d)
10-1
100 Epoch (10?)
200
Figure 5: Using a tempo-spatial Normalizing Flow constrained by Sparse Bayesian Learning to discover the advection-diffusin equation directly from single particle data. Panel a shows the true density profile, and in panels b and d we show the density inferred by binning (blue bars), inferred by NF (red) and the ground truth (black, dashed) at 0.1, 2.5, 4.5. Note that although the estimate of the density is very good, we see in panel that we recover two additional terms (bold blue line: Uz, bold orange line Myz-
OUTLOOK
Our experiments show strong improvement over non-differentiable comparable methods, and opens up several new avenues to explore. One direction is the choice of the prior parameters for the precision. We presented reasonable choice of prior parameters, but future work could find better estimates, for example 'prior-scheduler, similar to a learning rate scheduler, or explore approaches multitask learning. A different direction is exploring different Bayesian regression methods. For example, using Laplacian (Helgøy & Li 2020) spike-and-slab prior can improve sparsity (Nayek et al.) 2020). Alternatively, the prior can used to introduce more structure into the problem. For example, the group-SBL could be used combine data from several experiments (Babacan et al. 2014).
REFERENCES
S. Derin Babacan, Shinichi Nakajima, and Minh N. Do. Bayesian Stoup-Sparse Modeling and Variational Inference. IEEE Transactions on Signal Processing, 62(11):2906 2921, June 2014. ISSN 1053-587X, 1941-0476. doi: 10. 00.1109/TSP20114233999 leee. http://ccummttt8844000
URL http::tpp///tt httphttt///eeeeelooreee
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep Equilibrium Models. arXiv:1909.01377 [cs,
stat], October 2019. URL
arXiv: 1909.01377.
Jens Berg and Kaj Nyström. Data-driven discovery of PDEs in complex datasets. Journal of Com- putational Physics, 084:2399225333 May 2019. ISSN 00219991. doi: 10. n0.1016/j.ep..0199.0 ...066.
URL
arXiv: 1008.10788.
Gert-Jan Both and Remy Kusters. Temporal Normalizing Flows. 1912. [physics, stat), 09092
December 2019. URL http:://axxxv..rrraaaass///
arXiv: 1912.09092.
Gert-Jan Both and Remy Kusters. Fully diffeerentiabllleccc...ccc mtw.:nttiiiillelllllllleooo model discovery. arXiv:2106.048880 04886 [cs, stat), June 2021. URL htt.://pp//arriii.oo//aa///////s
04886
arXiv: 2106.04886.
Gert-Jan Both, Subham Choudhury, Pierre Sens, and Remy Kusters. DeepMoD: Deep learning for Model Discovery in noisy data. 1eXXiv:9044.9940 [physics, q-bio, stat], April 2019. URL
http::/aaae www.o///aaa..... abs////////// /1////9444 09406
arXiv: 1904.09406.
Gert-Jan Both, Gijs Vermarien, and Remy Kusters. Sparsely constrained neural networks for model discovery of PDEs. +XXi:2011.043:: [physics]. May 2021. URL attp: //t//va.vv.///////////..../.awa. wiw.ivvoooo org/ arXiv: 2011 201130.
04336

---

Under review as a conference paper at ICLR 2022
Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the Na- tional Academy of Sciences, (L3(15):3932-3937 April 2016. ISSN 0027-8424, 1091-6490.
doi: 10.1073/paas.15173411... 1517384113. URL hhttp://www... pnas. org/ wttp:/oooog///6/111010000 pnas. 151738411
Roberto Cipolla, Yarin Gal, and Alex Kendall. Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7482-7491, Salt Lake City, UT, USA, June 2018. IEEE. ISBN URL wttpp::///eeepppaaappaaaa
978- 978-1-5386-6420- doi: 10..119/CVP..2018... #Tc//dccumet//8578877
ieee.
Ingvild M. Helgøy and Yushu Li. A Noise-Robust Fast Sparse Bayesian Learning Model. wwt.o/ggga////.o/e/ 07220
arXiv:I908. 07220 [cs, stat], May 2020. URL arXiv: 1908.07220.
Suryanarayana Maddu, Bevan L. Cheeseman, Christian Müller, and Ivo Sbalzarini. Learning physically consistent mathematical models from data using group sparsity, (rxxi:2012.0639) [es, q-bio, stat], December 2020. URL http:://ark www.a/twwrro/ooooooooooooooossea abs /2012. 06391 2012.06391.
arXiv:
Rajdip Nayek, Ramon Fuentes, Keith Worden, and Elizabeth Cross. On spike-and-slab priors for Bayesian equation discovery of nonlinear dynamical systems via sparse linear regression. arXiv:2012.000 2.1997 (cs, eess, stal), December 2020. URL /Rtttttp://///// httt::://axxi..or//bbb...
01937 arXiv: 2012.01937.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics Informed Deep Learning (Part If): Data-driven Discovery of Nonlinear Partial Differential Equations. 0711. [cs, math, stat], November 2017. URL http://arxlv.co wttp://wwwvvg//b//////obso/ss/// 1 arXiv: 1711.10566.
Adil Rasheed, Oner San, and Trond Kvamsdal. Digital Twin: Values, Challenges and Enablers From Modeling Perspective. IEEE Access, 8:21980-22012, 2020. ISSN 2169-3536. doi: ALL10//ACCESS.2220.270000 8972429/
URL htttp::/ http:///wwpppoeeeee.eee.eo.////e wtt.:/eeeeee..ooeoceeeeeeeeeee
Danilo Jimenez Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows.
arXiv: 5555505.777 fes, stat], May 2015. URL. http:://arxx...rrg//////5555..00
arXiv: 1505.05770.
Samuel H. Rudy, Steven L. Brunton, Joshua L. Proctor, and Nathan Kutz. Data-driven dis- covery of partial differential equations. Science Advances, 364)x166 April 2017. ISSN 2375-2548. doi: 20.11166sciadv.1602200
URL URLaatt:://avaacesss. www.:aancc....ccccccmea....a
1tt.1122//ccaa.11122..... 1602614
Michael Tipping. tippingOla.pdf. Journal of Machine Learning Research, I, 2001. URL https:
7/ /www.
pdf
Peng Zheng, Travis Askham, Steven L. Brunton, J. Nathan Kutz, and Aleksandr Y. Aravkin. A Unified Framework for Sparse Relaxed Regularized Regression: SR3. IEEE Access, 7: 1404-1423, 2019. ISSN 2169-3536. doi: 10.1109/ACCESS.2005 20REESSS..2...22. 2886228858.. URL https:
//ieeexplore. ieee. org ottee/ooo//ccmome///5333355
A REPRODUCIBILITY
Optimizer We use the Adam optimizer with learning rate 10-3 and 3 (0.99, 0.99) for all experiments.
10

---

Under review as a conference paper at ICLR 2022
Figure We two soliton solution of the Korteweg-de Vries equation, starting 20 (-5.0, 0.0) with speeds с = 2) We regularly sample 50 points on the spatial domain [-6. 7] and 40 along the temporal domain [0.1, 3.0 and add 10% (in terms of the std. of the data) white noise. The network is composed of 3 hidden layers of 30 neurons. As library we derivatives up to fourth order, polynomials up third and all combinations of these.
Figure 2 We use two soliton solution of the Korteweg-de Vries equation, starting at xo (-5.0, 0.0) with speeds c (5, 2) We regularly sample 50 points on the spatial domain [-6, 7] and 40 along the temporal domain 3.0), and add 20% (in terms of the std. of the data) white noise. The network is composed of 3 hidden layers of neurons. As library we derivatives up to fourth order, polynomials up third and all combinations of these.
We solved the Burgers equation for a delta peak initial condition, regularly sample 50 points the spatial domain [-3, 4] and 20 along the temporal domain [0.5, 5.0], and add 1 10: 100% (in terms of the std. of the data) white noise. The network is composed of hidden layers of 30 neurons, As library derivatives up to fourth order, polynomials up third and all combinations of these.
Figure 3
Figure 4 We numerically solve the Kuramoto with spatial resolution of 1024 and spatial resolution of 256 frames. We take slice of data between 45 < t55 (25 frames) and subsample every 4th point along the spatial axis (giving 256 points/ frame) and add 20% (in terms of the std. of the data) white noise. The network is composed of 5 hidden layers 60 neurons. As library we derivatives up to fifth order, polynomials up to fourth and all combinations of these.
Kuramoto-Shivaahhnnky Shivashinsky equation for random initial condition
Figure 5
We release 200 random particles on a biased random walk with D = 1.5, == 0.5 from (1.5, 0.5) and
an initial distribution of two Gaussians centred at Tp (-5, -1) with widths take 50 steps with At = 0.05. The hyper network is composed of 2 hidden layers of 30 neurons and we use NF of 10 layers. As library we derivatives up third order, polynomials up to second and all combinations of these.
11


Reviews:
[{"review_id": "yllfgAJCXM", "reviewer": "Reviewer_MdGD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work proposes to denoise noisy data with PINNs, and use SBL on (fixed basis expansions of) the denoised data to provide differentiable basis selection.  The broad idea of combining SBL-like procedures with deep models is natural and has been examined in the context of generative modeling (e.g. Dai and Wipf, 2019), but application to PDE learning appears new.  Numerical experiments seem to demonstrate some improvements, although I don't work on PDE learning so am not completely sure.\n\nMy main concern is about the optimization landscape.  For SBL to extract truly sparse coefficients we need its objective to go to infinity; but the PINN loss is a normal likelihood, and will be bounded by the noise level of the data, which is assumed to be finite in this setting (this is different from typical sparse learning scenarios where a low level of noise is assumed).  Therefore, optimizing the sum of the two objectives seems a bad idea.  In particular, one \"global optima\" would be for the PINN to estimate $\\hat u \\equiv 0$, incurring a finite reconstruction error, so that the SBL loss tends to negative infinity.  The authors seem to work around this issue by imposing proper hyperpriors, and pretrain the PINN so that the joint optimization starts at an informative initial value, these fixes seem somewhat hackish: proper hyperpriors will restrict the algorithm's ability to obtain truly sparse solutions in practice, or the behavior of the algorithm may depend on the value of the hyperprior parameters $a,b,c,d$ which are difficult to determine a priori.\n\nAlso, while I don't work on PINNs so cannot judge the originality, a quick search leads to this work which should be discussed:\n\nAmuthan A. Ramabathiran and Prabhu Ramachandran^, \"SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEs\", Journal of Computational Physics, Volume 445, pages 110600, 2021.\n\nMinor questions/comments:\n\n* In the first experiment, why does SBL without PINN fail in the noiseless setting? Is this due to numerical errors in computing higher-order derivatives?\n* $\\mathbf{u}_t$(=$\\partial_t \\hat{\\mathbf{u}}$?) isn't defined in the text. The choices of notation might also be optimized.\n* Formatting issues: better to use `\\eqref` for equations, `Eq.~` as opposed to `Eq. `, etc.\n\nReferences:\n\nDai and Wipf (2019), Diagnosing and Enhancing VAE Models, in ICLR.", "summary_of_the_paper": "This work proposes to combine sparse-Bayesian learning (SBL) with physics informed neural networks (PINNs) to achieve feature/basis selection in learning PDEs.", "summary_of_the_review": "Pros:\n+ The problem of differentiable model discovery on noisy data seems important, and the proposed method appears promising and easy to implement\n\nCons:\n- Optimization landscape appears insensible\n\n**Post-rebuttal update**: while the authors clarified on the experiments and related work, my core concern on the optimization landscape remains.  Therefore I am keeping my score unchanged.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oH4o1aFOgQG", "reviewer": "Reviewer_rgZy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper provides a nice introduction and motivation to the problem of identifying simple governing dynamical equations from observed data.\n\nIt seems like there are two main contributions to the method:\nIncorporating a soft relaxation of a hard constraint (the discrete mask) used in previous work.\nUsing sparse Bayesian learning to formulate the sparse regression problem in a fully Bayesian way.\nThe paper seems to focus on (a), and does not focus on the benefits of (b) in the text.\n\nFirst, I have a really basic question: in eq (3), why not add a sparsity penalty on the coefficients of $\\Theta$ and then get rid of the mask? Would this not also be a soft relaxation of the hard mask constraint? This also seems more in line with prior work such as SINDy. I’m sure there’s some reason why this isn’t feasible, but I couldn’t figure out why.\n\nThe text describing the first experimental results (4.2) suggest that two baseline methods were tried, but Figure 2 only compares against one baseline method? What am I missing here?\n\nIn addition, more details could be provided about certain choices in the experiments. How sensitive are the results to different hyperparameter choices (choice of the network architecture, number of library terms, regularization hyperparameters, etc).\n\nSome suggestions regarding the figures:\n- Add axes labels to all figures. For example, in Figure 2a or 3a, what are the axes? In Figure 2b and 2c or 3b and 3c, it would be helpful if each panel had a short descriptive title.\n- It would be helpful to see the recovered equations and reconstruction for at least one of the example applications\n- Consider plotting the relative error of each parameter fit, instead of (or in addition to) the actual values (e.g. Fig 3b and 3c).", "summary_of_the_paper": "This paper studies the problem of identifying sets of equations that govern the dynamics of observed natural systems. Building off of work that uses sparse regression to identify a small set of active terms from a large dictionary of possible terms, this paper proposes a soft relaxation of a hard constraint used for selecting out possible terms in the optimization problem. The paper applies this technique to recover the underlying equations from data generated by a few canonical PDEs.", "summary_of_the_review": "Overall, I had a bit of a hard time understanding just how significant the improvements were when using this method. Is the differentiable method more robust to hyperparameters? In principle, with careful tuning, it seems like the non-differentiable version should also be to recover the same equations, given that they both use the same model for approximating the dynamics. In addition to significance, the clarity of the paper could also be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dHPiCOehmRv", "reviewer": "Reviewer_F5HG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThe approach presented in this work seems sample efficient (requires more experiments for empirical validation) and can identify equations from noisy datasets as compared to the baselines.\n\n**Weaknesses**\n\n* The authors need to compare their approach against PDE-NET 2.0 [1]. I'd be surprised if PDE-NET 2.0 doesn't perform competitively with the proposed method.\n\n* The authors need to present the exact equations retrieved by all the 3 baselines for each experiment. Also, they need to show which coefficients the colored lines correspond to in the legend (Fig. 2(c), 2(d), 2(e), 3(e)) to improve the readability of the plots.\n\n* I'm not sure if I understand the concerns raised by the authors regarding the regularisation constant in $l_1$ regularization in the following statement in the Introduction - *Applying $l_1$ regularisation can alleviate this problem, but raises the question of how strongly\nto apply it*. $\\lambda$, the strength of the regularization constant, is a hyperparameter that is varied uniformly. And the model/equation with the least MSE (or any other relevant metric) is picked.\n\n* The authors need to elaborate on the training strategy they used for DeepMod. DeepMod follows a similar learning strategy as [1] (i.e. use L1 regularization to induce sparsity into the Neural Network) to identify concise equations. In [2] the authors have demonstrated that such a learning scheme is susceptible to random restarts. This is also consistent with the observation made by the authors in sec.4.2. For that reason, a grid search has to be performed for different values of $\\lambda$ (strength of $l_1$ regularization) and $seeds$. Then the correct model/equation needs to be selected from these candidates using some metric, for instance in [2] the authors used the extrapolation error. Thus model selection is really important and if there doesn't exist a robust model selection strategy that can find the right equation from the pool of candidate equations then finding the right equation just once isn't at all useful. Thus, I'd like for the authors to shed some light on the training curriculum they followed for DeepMod. I'm assuming they didn't run DeepMod with a single set of hyperparameters and thus elaborate on the model selection strategy they used. \n\n* The authors have shown that their approach is able to identify the correct equations for a very limited number of cases like - Korteweg-de Vries, Burgers, Kuramoto-Shivashinsky, and Normalizing flows. They need to demonstrate that their algorithm can identify a variety of equations. Thus I strongly suggest that they generate PDEs with random terms and coefficients and report the number of cases for which the network succeeds or fails to identify the correct equation as they did in [2].\n\n* (Optional) For the above experiment it'd really insightful to see how training data size affects the quality of the learned equation.\n\n**References**\n\n[1] Z. Long, Y. Lu, X. Ma, and B. Dong. PDE-Net: Learning PDEs from Data. ArXiv e-prints, 2017\n\n[2] Sahoo S, Lampert C, Martius G. 2018 Learning equations for extrapolation and control. In Proc. of the 35th Int. Conf. on Machine Learning, Stockholm, Sweden, 10–15 July 2018 (eds J Dy, A Krause), vol. 80, pp. 4442–4450. Proceedings of Machine Learning Research\n", "summary_of_the_paper": "becauseThis work presents a model discovery algorithm to learn the PDEs from data. Their method uses Sparse Bayesian Learning to identify a minimal set of terms (along with their coefficients) from a set of predefined terms that constitute the PDE. Finally, they demonstrate that their algorithm can identify the PDE equations precisely for the following problems -  Korteweg-de Vries, Burgers, Kuramoto-Shivashinsky, and Normalizing flows.", "summary_of_the_review": "The only reason I feel the paper is marginally below the acceptance threshold is that -\n* The authors haven't compared their method against one of the most popular methods - PDE-NET 2.0\n* Lack of extensive experiments (see above) to demonstrate that their approach can identify a variety of equations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YZzlxIpgEet", "reviewer": "Reviewer_VxhG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n  - The task of learning a PDE is interesting.\n\nCons:\n  - But the technical aspect ultimately becomes sparse regression.\n  - There are many other differentiable alternatives that are simpler, or newer, that are not explored.\n\nComments: \n  - This is an arguably straightforward combination of prior works. As such, I'd liked to have seen more analysis on why the chosen approach (SBL) is good. For instance, there are many better sparsity-inducing priors than a Gaussian, such as the horseshoe prior, log-uniform, and other \"spike-and-slab\" style distributions. See for instance, [1, 2]. A comparison of such approaches would provide more insight. \n\n  - The baselines warrant more description, as I'm not sure why they fail. For instance, the main alternative method seems to be a greedy search over the mask (after Eq 3: \"The mask is updated periodically by some sparse regression technique, and as terms are pruned, the constraint becomes stricter,...\"). But more sophisticated search methods should fare better. \n\n  - Stronger baselines. For instance, before relaxing the mask, what about using REINFORCE gradient to learn binary masks? What about using the Gumbel-softmax or other types of relaxations first? (I imagine a sparsity-inducing approach should still fare better overall, but a comparison to alternative/simpler approaches should be warranted.) There are many other differentiable approaches to sparse regression that the paper does not mention. For a paper titled \"Fully differentiable model discovery\", I'd expect more exploration among these methods and ultimately a convincing conclusion on why the chosen approach seems the most worthwhile. (Is SBL with the chosen prior particularly suited for PINNs in any way?)\n\n  - PINF: I'm quite lost. (i) Is there's a vector field u(t, x) that is in the PINF, for which the SBL loss is applied on? (ii) I didn't understand the motivation behind a PINF, is it because the Gaussian observation model in PINN may be insufficient? (iii) Perhaps an example would help; what is the underlying PDE for the random walk experiment?\n\n  - Why optimize A and beta all the way, and not jointly with the PINN parameters?\n\nClarity issues:\n\n  - The paper mentions that this approach can provide \"a rigorous criterion for deciding whether a term is active or not\". What is the criterion? And why is it \"rigorous\". Such approach should still requires a threshold for deciding whether a term is active if you ultimately want a deterministic value for the mask.\n\n  - At the top of page 5, \"our method does not require backpropagating through the solver\" -> What solver is this referring to? Is this referring to optimizing A and beta? (By the way, since there's no closed form expression for A and beta, why not optimize for them at the same time as theta, which should be faster overall?)\n\n  - How does the DeepMoD method differ from what is written in/after Eq 3?\n\n  - When noise is added, is this being added to u only? What if you add noise to x or t? \n\n  - What exactly is 1% or 100% noise? What is this a percentage of? (i.e. does it correspond to some maximum standard deviation for some gaussian noise distribution?)\n\nMinor:\n\n  - Under Eq 18: what is little \"n\"? Should that be N?\n\n  - Figure 2: what is OLS? ordinary least squares? Is this referring to DeepMoD?\n\n  - Figure 2b: what is the dashed line?\n\n  - Is it DeePyMod or DeepMoD? Both names seem to be used in the paper.\n\n  - Maybe consider providing in-line labels (i.e. display u_x, u_xx, etc on top of the line), especially for Figure 5e. \n\n[1] \"Handling Sparsity via the Horseshoe\" Carvalho et al. (2009)\n\n[2] \"Bayesian Compression for Deep Learning\" Louizos et al. (2017)", "summary_of_the_paper": "This paper proposes learning an underlying PDE through the use of a physics-informed neural network and sparsity regularization on the partial derivatives. A neural network representing the PDE solution is trained simultaneously with a sparse regression on the partial derivatives of this neural network.\n\nThe main novelty seems to be combining the model discovery loss of Both et al. 2021 with a sparse prior, relaxing the objective and allowing gradient-based optimization. \n\nThe paper also discusses physics-informed normalizing flows but I feel like it's a bit disconnected / orthogonal to the rest of the paper and not enough information is given; I did not understand the motivation, how it connects to a PDE solution, and how it's applied in the experiments.", "summary_of_the_review": "This paper explores only one of many differentiable approaches to learning binary variables. I think a more comprehensive comparison against alternative differentiable approaches, from simple REINFORCE to horseshoe priors, would provide more insight and ultimately a convincing conclusion on why the chosen approach seems the most worthwhile. I do not understand the relevance of the PINF section; perhaps it could be its own standalone paper instead. There are also some clarity issues.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are in consensus that this manuscript falls just short of the bar. I recommend that the authors take their recommendations into consideration in revising their manuscript, with a particular focus on comparison to the state of the art.
Decision: Reject