SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Step 3. D-CIPHER consists of two optimization loops. The outer optimization is performed by a symbolic regression algorithm (in our case genetic programming). The inner optimization is performed by CoLLie (Section 7). Searching through a space of closed-form expression requires testing many candidate equations. D-CIPHER is designed to work with many different algorithms for symbolic regression and it is advised to choose an algorithm that can search through this space most efficiently.
Figure 9: Computation time of the experiments in Appendix F.3 The plot shows how the size of the dictionary  $\mathcal{Q}$  influences the computation time
CoLLie was specifically designed to solve the optimization problem as quickly as possible with a minor accuracy trade-off (see Figure 2). It is based on LARS which has time complexity  $\mathcal{O}(mn^2)$  [17] where  $m$  is the number of samples and  $n$  is the number of features. In our case,  $n = P$  is usually small as it corresponds to the size of the dictionary and  $m = SD$ . In our experiments,  $S$  was set up to 100 and  $D$  up to 10. Overall LARS is performed very quickly. The additional steps in CoLLie require only a few arithmetical operations (linear in  $n$ ) and a possible root searching of a single variable function that is efficiently implemented using Brentq algorithm [9].
Other important parts of the algorithm are the numerical integrations. One such integration is performed at the beginning of the algorithm to compute the matrix  $Z$  (see Algorithm 1). It does not contribute much to the computation time as it is performed only once. The other integration is performed for each candidate equation to compute vector  $\mathbf{w}$ . Also, substantial time is spent on computing the values of the candidate function  $g$  used in the integration. Fortunately, both of these operations can be implemented as vectorized operations which are designed to run very efficiently on modern hardware. We also discourage overly long equations for  $g$  (check the discussion in Appendix E.3) to limit the number of operations performed.
F.5 Challenges of derivative estimation
One of the advantages of D-CIPHER compared to other methods is the use of the variational formulation of PDEs that allows it to circumvent derivative estimation. This is important as derivative estimation is challenging, especially in noisy settings with infrequent sampling. The problem becomes more pronounced the higher the order of the derivative. To demonstrate these issues, we perform a series of synthetic experiments.
Qualitative study. First we qualitatively show how challenging the task of derivative estimation is. We generate an observed trajectory for the damped and forced harmonic oscillator. Then we estimate this trajectory using both Gaussian Process regression and Spline regression. As shown in Figure 10 (Panel A), the estimated trajectories are very close to the true trajectory. Then we estimate the first derivative (Panel B) and the second derivative (Panel C). We show the standard finite difference methods as well as derivative estimation techniques using Spline regression and Gaussian Process regression. In both cases we see that the estimated derivatives do not match the ground truth (calculated analytically) as closely as in Panel A. Moreover the mismatch for the second derivative seems to be bigger than for the first derivative.
Quantitative study. We investigate this relation quantitatively for the damped and forced harmonic oscillator and the wave equation. For the oscillator we generate an observed trajectory and then we estimate its derivatives, up to the fourth order using both finite difference and Gaussian Processes. We then compare the derivatives with the analytically calculated ground truths and measure root
Figure 10: Panel A shows that estimation of the true trajectory can be performed successfully by both Spline regression (Spline) and Gaussian Process regression (GP). Panel B shows the estimated first derivative and Panel C shows the estimated second derivative. We observe that the higher the order the less accurate is the estimate.
mean squared error. The results are shown in Figure 11 (Panel A). We can see that the error increases the higher the order of the derivative. For the wave equation, we perform a similar experiment but this time we estimate different mixed derivatives. We consider any mixed derivative  $\partial_t^i\partial_x^j$ , where  $i,j\in \{0,1,2\}$ . We demonstrate the results in Figure 11 (Panel B). We observe that the error increases the higher the order of the derivative  $(i + j)$ .
(B)
Figure 11: Panel A demonstrates the error between the estimated derivative and the ground truth increases with the order of the derivative (performed for the damped and forced harmonic oscillator). Panel B shows that the same happens for the wave equation. The  $(i,j)$  entry of the heatmap should be interpreted as the RMSE between the estimated derivative  $\partial_t^i\partial_x^j$  and the ground truth.
F.6 Challenges of PDE discovery and how we address them
PDE discovery is a very difficult task with many challenges. In Table 9 we summarize some of them and describe how our work addresses them.
F.7 Significance of the new notions
In this section, we want to justify that the new notions we introduce (evolution assumption, linear combination form, derivative-bound part, derivative-fee part, Variational-Ready PDEs) are important theoretical contributions that help us understand the landscape of different PDEs from the machine learning perspective.
The definitions we introduce let us characterize different classes of PDEs. These new notions complement the standard recognized PDE classes such as semi-linear, quasilinear, hyperbolic, etc. These standard classes were introduced predominantly to characterize the solving techniques or the properties of the solutions, whereas the notions we introduce relate to the difficulty of discovering such equations from data.
Table 9: Some challenges of PDE discovery and how we address them
Challenge	Novel contribution (how we address the challenge)
Noisy measurements	We use the variational loss function (Equation 11)
Large diversity of PDEs	We allow any closed-form \(\partial\)-free part and require no evolution assumption (Section 3)
Learning compact equations	We penalize long \(\partial\)-free parts and use L1 normalization for the \(\partial\)-bound part (Section 6, Appendix E.3)
Efficient search	We develop a quick algorithm, CoLLie, for the inner optimization (Section 7)
Chaotic systems	We do not estimate the initial conditions [11] or perform forward time stepping [29] which are computationally unstable for chaotic systems
Significance of linear combination form and the evolution assumption. In Table 10 we demonstrate how the presence of the two assumptions, the linear combination form (LC) and the evolution assumption (EA), influences the optimization problem. We see that with both assumptions, the problem is relatively straightforward and reduces to sparse linear regression. With only one of the assumptions present the problem becomes more difficult. With neither of these assumptions, the problem becomes very difficult and requires some other assumptions. D-CIPHER does not make either of these assumptions but it assumes the PDE to be of the form described by Equation 13.
Significance of  $\partial$ -bound part,  $\partial$ -free part and Variational-Ready PDEs. The difficulty of derivative estimation has been one of the main challenges of PDE discovery. The variational formulation allows to circumvent derivative estimation and thus is more robust to noisy data. Previously, the variational formulation has been applied only to a subset of equations in a linear combination form and with the evolution assumption. We observed that any restrictions that the variational formulation might put on the equation come from the terms containing the derivatives. Thus we define the derivative-bound part and the derivative-free part of the PDE due to their significance for the variational formulation. That allows us to define Variational-Ready PDEs as currently the broadest class of PDEs that admit the variational formulation. We believe it is an important contribution as methods requiring derivative estimation underperform in settings with high noise. This definition outlines the current limits of any method that circumvents derivative estimation in that way.
Table 10: This table demonstrates how the presence of the two assumptions, the linear combination form (LC) and the evolution assumption (EA), influences the optimization problem.
LC	EA	Equation form	Optimization problem	Examples
Yes	Yes	∂tuj = ∑p=1Pθpfp(x, u(x), ∂[K]u(x))	Relatively easy. Can be formulated as finding sparse solution to linear least squares (similar to ridge regression)	[10, 47]
Yes	No	∑p=1Pθpfp(x, u(x), ∂[K]u(x)) = 0	Medium difficulty. Can be formulated as P separate linear least squares problems as above	[23]
No	Yes	∂tuj = g(x, u(x))	Medium difficulty. Find g using symbolic regression	[42]
No	No	f(x, u(x), ∂[K]u(x)) = 0	Very difficult. Requires other assumptions (in this work, Equation 13)	D-CIPHER
F.8 Comparison with D-CODE
To clarify our contributions, we compare D-CIPHER to D-CODE [42] which we think is closest in spirit to the algorithm we developed as both of them allow any closed-form derivative-free part and
use variational formulation to circumvent derivative estimation. We also note how the new notions we introduce help us compare the two works.
Algorithm presented in D-CODE [42] can discover any first-order explicit closed-form ODE, i.e., an equation of the form.
$$
\partial_ {t} u _ {j} (t) = g (\pmb {u} (t), t)
$$
where  $g$  is a closed-form function. It uses the variational formulation of ODEs to circumvent derivative estimation.
Table 11: Comparison between D-CIPHER and D-CODE [42]
Property	D-CODE [42]	D-CIPHER
Applicable to PDEs	No	Yes
Higher-order derivatives	No (only first-order)	Yes
Requires evolution assumption	Yes	No
Derivative-bound part	Fixed: ∂tuj	Learned: ∑p=1Pβpˆp[uj]
Derivative-free part	Any closed-form function	Any closed-form function
Derivative estimation	No	No
D-CODE [42] can be considered a special case of D-CIPHER. We can recover it from D-CIPHER by choosing the dictionary to contain only one element, i.e.,  $\mathcal{Q} = \partial_t u_j$ .
As the derivative-bound part is fixed, every ODE of that form admits the variational formulation. This is not true for PDEs as there are derivative-bound parts that might prohibit the variational formulation. Thus D-CIPHER required careful consideration of the appropriate class of equations to search over.
As D-CIPHER needs to find both the  $\partial$ -free part (function  $g$ ) and the  $\partial$ -bound part, the optimization problem is much more complicated. That is why we restrict the derivative-bound part of the PDE to be spanned by terms from the pre-specified dictionary and develop an efficient optimization algorithm, CoLLie. We emphasize that, as is the case for [42], we do not put any constraints on the derivative-free part of the PDE, apart from it being closed-form.
F.9 Error bounds
While we would like to have error bounds for the discovered PDE, we note that the problem we solve is significantly more difficult than the one considered in other works. The space of PDEs we consider is much more complex than the space of PDEs in a linear combination form. In other works (e.g., [47], [35]), the PDE is basically a vector in  $\mathbb{R}^P$  and the discovery task is mostly reduced to finding a sparse enough vector that approximately solves a certain linear equation. Of course, there is a lot of literature that aids in establishing error bounds in such problem settings. However, D-CIPHER searches over a space  $\mathbb{R}^P\times CFE(M + N)$ , where  $CFE(M + N)$  is a space of closed-form expressions in  $M + N$  variables. This space is combinatorial in the functional form and continuous in real constants. This makes it very challenging to derive any error bounds.
Figure 12: Comparison of different estimation algorithms that can be used in D-CIPHER. GP - Gaussian Process regression, Nearest - Nearest point interpolation, Linear - Linear interpolation, Cubic - Cubic interpolation.
F.10 Comparison between different estimation algorithms
Step 2 of D-CIPHER requires estimating the fields. We emphasize that any choice of reconstruction algorithm can be used, and it should be chosen based on the application and domain knowledge. In our experiments, D-CIPHER is implemented using Gaussian Process regression [60]. In this section, we investigate other common interpolation algorithms. We implement D-CIPHER with different estimation algorithms. In particular, we compare Gaussian Process (GP) against: Nearest point interpolation (Nearest), Linear interpolation, and Cubic interpolation (Cubic). The implementation details of these three algorithms can be found in the scipy [56] documentation. The results are presented in Figure 12. We see that the estimation algorithms that produce smoother functions (GP, Cubic) tend to give better results.
F.11 How comprehensive does the dictionary  $\mathcal{Q}$  need to be?
Table 12 shows 17 different differential equations and what terms they need in a dictionary. All of them can be discovered with a dictionary of size of 10,  $\mathcal{Q} = \{\partial_t u, \partial_x u, \partial_t \partial_x u, \partial_t^2 u, \partial_x^2 u, \partial_t \partial_x^2, \partial_x^3 u, \partial_x^4 u, \partial_x(u^2), \partial_x^2(u^2)\}$ , and 9 of those equations can be described with a dictionary containing just 4 terms,  $\mathcal{Q} = \{\partial_t u, \partial_x u, \partial_t^2 u, \partial_x^2 u\}$ . It is thus likely that such small dictionaries are sufficient to discover most of the well-known equations. Many differential equations have similar derivative-bound parts and differ by derivative-free parts. Being able to discover any closed-form derivative-free part is what makes D-CIPHER stand out.
Table 12: This table lists various Variational-Ready PDEs (some of which are discussed in the paper) and shows which extended derivative terms they require. All of them can be discovered with a dictionary of size of 10, and 9 of them can be described with a dictionary containing just 4 terms. It is thus likely that such small dictionaries are sufficient to discover most of the well-known equations.
Equation	∂t u	∂x u	∂t ∂x u	∂t2 u	∂x2 u	∂t ∂x2 u	∂x3 u	∂x4 u	∂x(u2)	∂x2(u2)
Heat equation	✓	✓
Wave equation	✓	✓
Burger's equation	✓	✓	✓
SLM model	✓	✓
Damped harmonic oscillator	✓	✓
Kuramoto-Sivashinsky equation	✓	✓	✓	✓
Benjamin-Bona-Mahony equation	✓	✓	✓	✓
Boussinesq equation	✓	✓	✓	✓
Chafee-Infante equation	✓	✓
Damped wave equation	✓	✓	✓
Fisher's equation	✓	✓
Hunter-Saxton equation	✓	✓	✓
Klein-Gordon equation	✓
Korteweg-De Vries equation	✓	✓	✓
Liouville's equation	✓	✓
Sine-Gordon equation	✓	✓
Sinh-Gordon equation	✓
F.12 How to take advantage of the observed derivatives?
D-CIPHER can make use of the observed derivatives (if they are available) by adapting the dictionary. Consider a setting with a dictionary  $\mathcal{Q} = \{\partial_t u, \partial_x u, \partial_t \partial_x u, \partial_t^2 u, \partial_x^2 u\}$ . If we happen to have the measurements of  $\partial_t u$  then we can introduce a new variable  $v = \partial_t u$  and change the dictionary to  $\mathcal{Q} = \{v, \partial_x u, \partial_x v, \partial_t v, \partial_x^2 u\}$ . Note, we have performed experiments where the dictionary contains more than one dependent variable in Appendix F.2. With observed derivatives, we can also enlarge the space of Variational-Ready PDEs by allowing  $g$  ( $\partial$ -free part) to depend on  $v$  as well.
F.13 Impact on real-world problems
D-CIPHER is especially useful in discovering governing equations for systems with more than one independent variable. For instance, spatiotemporal data or temporal data structured by age or size.
In particular, we envision D-CIPHER to be useful in modeling spatiotemporal physical systems, population models, and epidemiological models.
Spatiotemporal physical systems. D-CIPHER may prove useful in discovering equations governing the oceans or the atmosphere. For instance, some places actively add or remove CO2 from the atmosphere. These "sources" and "sinks" are likely to be described by a  $\partial$ -free part which D-CIPHER is specially equipped to discover. Similarly, with the ocean temperature where  $\partial$ -free part can describe a heat source. Another area of application can be modeling seismic waves across the earth's crust. Here the  $\partial$ -free part can describe the vibration source (e.g., an earthquake).
Population models. Population models can be used in agriculture to determine the harvest or for pest control to predict their impact on the crop. They have also been used in environmental conservation to model the population of endangered species. Population models have also been used in modeling the growth of cells to better understand tumor growth. Moreover, understanding the evolution of a population pyramid for a specific country may prove invaluable in ensuring its economic stability. As in all these scenarios, the rates of growth and mortality are likely to be described by  $\partial$ -free part, D-CIPHER is uniquely positioned to discover such equations as an aid to human experts.
Epidemiological models. Epidemiological models are crucial during a pandemic for better planning and interventions. For many diseases, the rates of mortality and infection are age-dependent. Thus, modeling the spread of disease using PDEs (rather than ODEs) might provide superior results.
F.14 D-CIPHER in practice
Below we discuss the things to consider while using D-CIPHER.
The order of the differential equation. One of the first considerations should be to choose the order of the differential equation  $K$ . For many dynamical systems,  $K = 2$  is sufficient unless we expect very complicated behavior. Then, considering  $K = 3$  or even  $K = 4$  may be warranted. Note, that we show that D-CIPHER can discover a fourth-order PDE (Kuramoto-Sivashinsky equation) in Section 8.
Homogeneous equations. Before searching through the whole space of closed-form  $g$  (derivative-free parts), we can consider whether the equation we want to discover may be homogeneous. These experiments on the restricted search space can provide quick insights before searching through all closed-form derivative-free parts.
Terms in a dictionary. For a given order of a differential equation  $K$ , it is a good idea to include all standard differential operators up to order  $K$  acting on all the variables. For instance, for  $K = 2$  and  $M = 1 + 1$  we could choose  $\mathcal{Q} = \{\partial_t u, \partial_x u, \partial_t \partial_x u, \partial_t^2 u, \partial_x^2 u\}$ . That allows to cover all linear PDEs with constant coefficients up to that order. To allow for non-linear PDEs we can include a term like  $\partial_t(u^2) = 2u \partial_t u$  that often describes advection (as in Burger's equation).
Dictionary steering when dealing with many dependent variables. When we deal with a system of PDEs rather than a single PDE choosing a dictionary is increasingly important. As we explain in Appendix F.2, discovering whole systems of PDEs is very challenging and D-CIPHER is not designed to do so out of the box. However, we show how that can be done in certain situations. We can steer what kind of equations are discovered by choosing the terms in the dictionary.
Estimation algorithm. Estimation algorithms make different assumptions on the data-generating process and should be chosen based on domain expertise. As we show in Appendix F.10, algorithms that produce smoother functions, such as Gaussian Process regression and cubic spline interpolation, tend to have good results. We can consider the advantages and disadvantages of these methods. For instance, Gaussian Process regression works very well for smooth signals. However, it is computationally intensive and might not perform well if the signal is not smooth enough (it has abrupt changes). Spline interpolation, on the other hand, is faster and more appropriate for less smooth signals, but it might introduce certain unwanted artifacts because of using cubic polynomials to interpolate the data.