SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qianli Shen
shenqianli@u.nus.edu
Wai Hoh Tang
waihoh.tang@nus.edu.sg
Apostolos Psaros
a.psaros@brown.edu
Kenji Kawaguchi
kenji@nus.edu.sg
Standard approaches for uncertainty quantification in deep learning and physics-informed learning have persistent limitations. Indicatively, strong assumptions regarding the data likelihood are required, the performance highly depends on the selection of priors, and the posterior can be sampled only approximately, which leads to poor approximations because of the associated computational cost. This paper introduces and studies confidence interval (CI) estimation for deterministic partial differential equations as a novel problem. That is, to propagate confidence, in the form of CIs, from data locations to the entire domain with probabilistic guarantees. We propose a method, termed Physics-Informed Confidence Propagation (PICProp), based on bi-level optimization to compute a valid CI without making heavy assumptions. We provide a theorem regarding the validity of our method, and computational experiments, where the focus is on physics-informed learning. Code is available at https://github.com/ShenQianli/PICProp.
Combining data and physics rules using deep learning for solving differential equations [25] and learning operator mappings [20] has recently attracted increased research interest. In this context, physics-informed machine learning is transforming the computational science field in an unprecedented manner, by solving ill-posed problems, involving noisy and multi-fidelity data as well as missing functional terms, which could not be tackled before [16]. Nevertheless, uncertainty – mainly due to data noise and neural network over-parametrization – must be considered for using such models safely in critical applications [24]. Despite extensive work on uncertainty quantification (UQ) for deep learning [1], standard UQ has persistent limitations. Indicatively, although for most practical applications where only confidence intervals (CIs) are required, the harder problem of UQ is often solved first using simplifying assumptions, and the CIs are subsequently computed. Further, strong assumptions regarding the data likelihood are required, the performance highly depends on
the selection of priors, and the posterior can be sampled only approximately, which leads to poor approximations because of the associated computational cost.
In this paper, we introduce and study CI estimation for addressing deterministic problems involving partial differential equations (PDEs) as a novel problem. That is, to propagate confidence, in the form of CIs, from data locations to the entire domain with probabilistic guarantees. We name our framework Physics-Informed Confidence Propagation (PICProp) and propose a practical method based on bi-level optimization to identify valid CIs in physics-informed deep learning. Our method can be applied in diverse practical scenarios, such as propagating data CIs, either estimated or given by experts, to the entire problem domain without distributional assumptions. Indicative applications are provided in Appendix A.
PICProp provides distinct advantages over traditional UQ approaches in the context of physics-informed deep learning. Specifically, PICProp does not rely on strong assumptions regarding the data likelihood and constructs theoretically valid CIs. Further, PICProp offers a unique capability to construct joint CIs for continuous spatio-temporal domains. In contrast, UQ methods typically construct marginal intervals for discrete locations. Furthermore, due to the conservative propagation of uncertainty, PICProp produces conservative CIs, which is preferred in critical and risk-sensitive applications [7, 13], considering the significant harm of non-valid confidence intervals. This sets it apart from commonly used UQ methods like deep ensembles [18] and Monte-Carlo dropout [28, 10], which are known to be overconfident in practice [9, 7].
The paper is organized as follows: In Section 2 (and its Appendices B & C), we introduce the PDE problem setup and discuss related work. In Section 3 (and its Appendices D-F), we introduce the problem of CI estimation for PDEs and extensively describe joint solution predictions. Next, we formulate this novel problem as a bi-level optimization problem in Section 4 and Appendices G & H, and adopt various algorithms for computing the required implicit gradients in Appendix I. Finally, we evaluate our method on various forward deterministic PDE problems in Section 5 and Appendices J-L, and discuss our findings in Section 6.
Consider the following general PDE describing a physical or biological system:
$$
\mathcal {F} [ u (x), \lambda (x) ] = 0, x \in \Omega , \quad \mathcal {B} [ u (x), \lambda (x) ] = b (x), x \in \Gamma , \tag {1}
$$
where  $x$  is the  $D_x$ -dimensional space-time coordinate,  $\Omega$  is a bounded domain with boundary  $\Gamma$ , and  $\mathcal{F}$  as well as  $\mathcal{B}$  are general differential operators acting on  $\Omega$  and  $\Gamma$ , respectively. The PDE coefficients are denoted by  $\lambda(x)$ , the boundary term by  $b(x)$ , and the solution by  $u(x)$ .
In this paper, we focus on the data-driven forward deterministic PDE problem, where (i) operators  $\mathcal{F},\mathcal{B}$  and coefficients  $\lambda$  are deterministic and known; (ii) datasets  $\mathcal{D}_f = \{x_f^i\}_{i = 1}^{N_f}$  for the force condition and  $\mathcal{D}_b = \{(x_b^j,b^j)\}_{j = 1}^{N_b}$  for the boundary condition are given; (iii) uncertainty only occurs in the boundary term due to data noise, i.e.,  $b^{j} = b(x_{b}^{j}) + \epsilon^{j}$  for  $j = 1,\dots,N_{b}$ , and the rest of the dataset is considered to be deterministic without loss of generality. We denote the function obtained by utilizing the dataset  $z = \{\mathcal{D}_f,\mathcal{D}_b\}$  for approximating the solution  $u(x)$  of Equation (1) with a data-driven solver  $\mathcal{A}$  as  $u_{\mathcal{A}}(x;z)$ .
Clearly, different datasets  $z$  contaminated by random noise lead to different solutions  $u(x;z)$  and thus, to a distribution of solutions at each  $x$ . It follows that the solution  $u(x;Z)$  can be construed as stochastic, where  $Z$  is used to denote the dataset as a random variable and  $z$  to denote an instance of  $Z$ . In this regard, this paper can be viewed as an effort towards a universal framework, with minimal assumptions, for constructing CIs for solutions of nonlinear PDEs.
Physics-Informed Neural Networks The physics-informed neural network (PINN) method, developed by [25], addresses the forward deterministic PDE problem of Equation (1) by constructing a neural network (NN) approximator  $u_{\theta}(x)$ , parameterized by  $\theta$ , for modeling the approximate solution
$u(x;z)$ . The approximator  $u_{\theta}(x)$  is substituted into Equation (1) via automatic differentiation for producing  $f_{\theta} = \mathcal{F}[u_{\theta},\lambda]$  and  $b_{\theta} = \mathcal{B}[u_{\theta},\lambda]$ . Finally, the optimization problem for obtaining  $\theta$  in the forward problem scenario given a dataset  $z$  is cast as
$$
\hat {\theta} = \underset {\theta} {\operatorname {a r g m i n}} \mathcal {L} _ {p i n n} (\theta , z), \quad \text {w h e r e} \quad L _ {p i n n} := \frac {w _ {f}}{N _ {f}} \sum_ {i = 1} ^ {N _ {f}} | | f _ {\theta} \left(x _ {f} ^ {i}\right) | | _ {2} ^ {2} + \frac {w _ {b}}{N _ {b}} \sum_ {i = 1} ^ {N _ {b}} | | b _ {\theta} \left(x _ {b} ^ {j}\right) - b ^ {j} | | _ {2} ^ {2}, \tag {2}
$$
and  $\{w_{f}, w_{b}\}$  are objective function weights for balancing the various terms in Equation (2); see [23]. Following the determination of  $\hat{\theta}$ ,  $u_{\hat{\theta}}$  can be evaluated at any  $x$  in  $\Omega$ . PINN can be scalled up for high-dimensional PDEs [14].
Uncertainty Quantification for Physics-Informed Learning Although UQ for physics-informed learning is not extensively studied, the field is gaining research attention in recent years. Indicatively, [37] employed dropout to quantify the uncertainty of NNs in approximating the modal functions of stochastic differential equations; [36] presented a model based on generative adversarial networks (GANs) for quantifying and propagating uncertainty in systems governed by PDEs; [35] developed a new class of physics-informed GANs to solve forward, inverse, and mixed stochastic problems; [11] further studied Wasserstein GANs for UQ in solutions of PDEs; [34] proposed Bayesian PINNs for solving forward and inverse problems described by PDEs with noisy data. Although this field is always evolving, we refer to it in this paper as "standard" UQ in order to distinguish it from our proposed method in Section 4. In Appendices B-C, we summarize the limitations of standard UQ, and we direct interested readers to the review paper [24] for more information.
Interval Predictor Model (IPM) The Interval Predictor Model (IPM) in regression analysis has a similar motivation as our approach. A function of interest is approximated by an IPM and the bounds on the function are obtained. [26] proposed a technique using a double-loop Monte Carlo algorithm to propagate a CI using the IPM. The technique requires sampling epistemic and aleatory parameters from uniform distributions to estimate the upper and lower bounds of the IPM. Our approach is different for two reasons. First, we fit a PINN model directly and thus, sampling epistemic parameters may not be feasible. Second, we do not require a sampling procedure. Instead, we cast the CI estimation as a bi-level optimization problem and solve it using hyper-gradient methods.
3 Confidence Intervals for Solutions of Nonlinear Partial Differential Equations
In this section, we provide the definition of CI for the exact PDE solution and introduce our main assumption.
Definition 3.1 (Joint CI for PDE Solution). A pair of functions  $(L_Z, U_Z) \in \mathbb{R}^\Omega \times \mathbb{R}^\Omega$  is a  $p$ -CI for the exact PDE solution  $u(x)$  if the random event  $\mathcal{E} := \{\{L_Z(x) \leq u(x) \leq U_Z(x), \forall x \in \Omega\}\}$  holds with probability at least  $p$ .
Note that this interval refers to the exact solution  $u(x)$ . Therefore, a novel problem is posed in Definition 3.1, pertaining to propagating randomness, in the form of CIs, from the data locations, typically on the boundary, to the rest of the domain. The subscript  $Z$  in  $(L_Z, U_Z)$  signifies that the CI is random as it depends on the random dataset  $Z$ . The available data can be an instance of  $Z$ , denoted as  $z$ , or a collection of instances; see Appendix D for the considered problem scenarios. Such instances can, for example, be drawn from repeating independent experiments. The subscript  $Z$  in  $(L_Z, U_Z)$  is dropped if there is no ambiguity.
We also note that the joint CI (also referred to as confidence sequence [17, 32]) in Definition 3.1 corresponds to joint predictions  $u(\cdot)$  over the whole domain  $\Omega$ , rather than to marginal predictions  $u(x)$  for a given  $x$ . The importance of joint predictions as an essential component in UQ has been emphasized in recent works for active learning [30], sequential decision making [33], and supervised learning [22]. Nevertheless, the present paper is the first work to consider joint predictions, and general CI estimation, in the context of physics-informed learning. In Appendix E, we provide several theoretical remarks delineating the differences between marginal and joint predictions, as well as unions of bounds, which are typically used in standard UQ. In Appendix F, we highlight these differences with a toy example.
Next, to relate the PINN solution with the exact PDE solution  $u(x)$  of Definition 3.1, we make the following assumption regarding the noise at the boundary.
Assumption 3.2 (Zero-mean Noise). The noise on the boundary is zero-mean, i.e.,  $\mathbb{E}[\epsilon^j] = 0$ ,  $\forall j \in \{1, \dots, N_b\}$ .
Next, a clean dataset (without noise), denoted as  $\bar{z}$ , can be considered as the expectation of  $Z$ , i.e.,  $\bar{z} = \mathbb{E}[Z]$ . In this regard, consider a data-driven PDE solver  $\mathcal{A}$ , which yields an approximate solution  $u_{\mathcal{A}}(x;z)$  given a dataset  $z$ . We define the properness of  $\mathcal{A}$  based on the concept of a clean dataset.
Definition 3.3 (Proper PDE Solver). A PDE solver  $\mathcal{A}$  is  $\eta$ -proper with respect to a clean dataset  $\bar{z}$ , if  $\max_{x \in \Omega} |u_{\mathcal{A}}(x; \bar{z}) - u(x)| \leq \eta$ .
Henceforth, we only consider proper PDE solvers. That is, a powerful solver is available such that the absolute error of the solution obtained by the solver with a clean dataset is bounded. In all cases, we assume that such an informative clean dataset exists, but is not available. Further, the constant  $\eta$ , which is only related to  $\bar{z}$  and  $\mathcal{A}$ , is considered known and fixed in the computational experiments. Note that no assumptions are made about  $u(x;z)$  for  $z \neq \bar{z}$ , i.e., there are no assumptions about the PINN solver robustness against noisy data. Finally, in all NN training sessions we consider no NN parameter uncertainty, although it is straightforward to account for it by combining our method with Deep Ensembles [18], for instance. The aim of this paper is to propose a principled and theoretically guaranteed solution approach and thus, we refrain from applying such ad hoc solutions in the computational experiments and plan to consider parameter uncertainty rigorously in a future work. Nevertheless, we provide in Appendix L a comparison with Bayesian PINNs for completeness.
In this section, we introduce our PICProp method and provide the accompanying theorem, as well as practical implementation algorithms. Further, we discuss the computational cost of our method and propose a modification for increasing efficiency. Specifically, based on Definition 3.1, we address the problem of propagating the uncertainty from the boundary to arbitrary locations  $x \in \Omega$  of the domain. Equivalently, this relates to obtaining the corresponding  $p$ -CIs, i.e., obtaining the functions  $(L, U) \in \mathbb{R}^{\Omega} \times \mathbb{R}^{\Omega}$  in Definition 3.1.
4.1 Physics-Informed Confidence Propagation (PICProp)
Our method comprises two steps: (i) a  $p$ -CI  $\tilde{\mathcal{Z}}_p$  for the clean data  $\bar{z}$  is constructed. (ii)  $\tilde{\mathcal{Z}}_p$  is propagated to any point  $x \in \Omega$  of the domain by solving the following problem,
$$
L (x) = \min  _ {z \in \tilde {\mathcal {Z}} _ {p}} u _ {\mathcal {A}} (x; z) - \eta , \quad U (x) = \max  _ {z \in \tilde {\mathcal {Z}} _ {p}} u _ {\mathcal {A}} (x; z) + \eta , \tag {3}
$$
which searches for configurations of points  $z$  within the interval  $\tilde{\mathcal{Z}}_p$  such that the predicted values for  $u(x)$  given by the  $\eta$ -proper solver  $\mathcal{A}$  are minimized or maximized, respectively.
The CI in the first step can either be given or constructed from data. The former case relates to various practical scenarios (e.g., the error ranges of measurement sensors with a certain confidence degree are known or given by experts), whereas the latter serves as a data-driven scheme, where less prior knowledge is required. Techniques to construct the CI in the first step depend on the data (noise) distribution. For example, if the noise is assumed to be Gaussian, the corresponding CI can be identified by chi-squared or Hotelling's  $T$ -squared statistics. If the noise distribution is not known, a concentration inequality can be used to identify the CI. Relevant details are provided in Appendix D. See also Figure 1 for an illustration.
The interval constructed via Equation (3) is a joint  $p$ -CI for the whole domain  $\Omega$  of  $u$ . This fact is formalized in the next theorem (see Appendix G for the corresponding proof).
Theorem 4.1. Consider a  $p$ -CI for  $\bar{z}$  denoted as  $\tilde{\mathcal{Z}}_p$ , constructed such that  $\operatorname{Pr}(\bar{z} \in \tilde{\mathcal{Z}}_p) \geq p$ . Then the functions  $(L, U)$  obtained by solving Equation (3) define a  $p$ -CI for  $u$ .
Although the robustness of the PDE solver does not affect the validity of Theorem 4.1, it nevertheless affects the quality of intervals constructed via Equation (3). Specifically, although valid CIs can always be constructed with arbitrary proper PDE solvers by solving Equation (3), a robust PDE solver less sensitive to data noise is expected to yield tighter CIs compared to a less robust solver.
Note that the confidence guarantee of the constructed CIs corresponds to a probability that is greater than, rather than equal to,  $p$ , which means that the constructed CIs are conservative. From a design perspective, conservative uncertainty estimation is the preferred route for critical and risk-sensitive applications [7, 13], although it can be more costly.
Next, a brute force and computationally expensive approach to solve Equation (3) pertains to traversing  $z \in \tilde{\mathcal{Z}}_p$  in an exhaustive search (ES) manner and invoking a PDE solver for each candidate  $z$ . Alternatively, we propose to first employ a NN for obtaining  $u_{\hat{\theta}(z)}$  by minimizing the physics-informed loss of Equation (2) and approximating  $u(x;z)$  for each  $z \in \tilde{\mathcal{Z}}_p$ . In this regard, if a PINN is assumed to be an  $\eta$ -proper PDE solver, Equation (3) can be further extended to a bi-level optimization problem cast as
$$
\begin{array}{l} L (x) = \min  _ {z \in \tilde {\mathcal {Z}} _ {p}} u _ {\hat {\theta} (z)} (x) - \eta , \quad U (x) = \max  _ {z \in \tilde {\mathcal {Z}} _ {p}} u _ {\hat {\theta} (z)} (x) + \eta , \\ s. t. \quad \hat {\theta} (z) = \arg \min  _ {\theta} \mathcal {L} _ {p i n n} (\theta , z). \\ \end{array}
$$
Nevertheless, explicit gradient  $\nabla_{z}u$  calculation is intractable, as multiple steps of gradient descent for the NN are required for a reliable approximation. Despite this fact, bi-level optimization involving large-scale parameterized models such as NNs, with main applications in meta-learning, has been extensively studied in recent years and has resulted in numerous implicit gradient approximation methods (see Appendix I). In this regard, Algorithm 1 in Appendix H summarizes our method for solving efficiently Equation (3) by using such approximations. In passing, note that  $\eta = 0$  is considered for all implementations in this work.
4.2 Efficient Physics-Informed Confidence Propagation (EffiPICProp)
If joint CIs are required for multiple query points, denoted by the set  $\mathcal{X}_q$ , Algorithm 1 has to be used multiple times to solve Equation (4) for each and every point  $x$  in the set. This clearly incurs a high computational cost. To address this issue, a straightforward approach pertains to training an auxiliary model parameterized by  $\psi$  that models the CI as  $(L_{\psi}(x_q), U_{\psi}(x_q))$  for all query points  $x_q \in \mathcal{X}_q$ , or more generally of the entire domain  $\Omega$ . However, this naive regression using solutions of the bi-level optimization at a set of locations, which we refer to as SimPICProp, does not provide accurate results when the size of the query set is small; see computational results in Section 5.1.
For improved generalization properties with limited query points, we propose to train a two-input parameterized meta-model  $u_{\psi}(x_q,x)$  that models the resulting PDE solution  $u(x)$ , via Equation (4), for each different query point. For example, note that  $L(x)$  in Equation (4) evaluates the NN  $u_{\hat{\theta}(z)}$  at the location of the query point and  $z$  minimizes the corresponding value. As a result, if a trained model  $u_{\psi}(x_q,x)$  minimizes the value of  $u$  at  $x_q$ , the sought  $p$ -CI can be simply obtained by evaluating  $u_{\psi}$  at  $(x_q,x_q)$ . More generally, given a query set consisting of  $K$  randomly selected query points  $\mathcal{X}_q = \{x_q^k\}_{k=1}^K$ , the model is trained such that it solves the problem
$$
\begin{array}{l} \min  _ {\psi} \frac {1}{2 K} \sum_ {k = 1} ^ {K} \mathcal {L} _ {p i n n} \left(u _ {\psi} ^ {L} \left(x _ {q} ^ {k}, \cdot\right), z ^ {L} \left(x _ {q} ^ {k}\right)\right) + \mathcal {L} _ {p i n n} \left(u _ {\psi} ^ {U} \left(x _ {q} ^ {k}, \cdot\right), z ^ {U} \left(x _ {q} ^ {k}\right)\right), \\ \text {w h e r e} \quad z ^ {L} \left(x _ {q} ^ {k}\right) = \underset {z \in \tilde {\mathcal {Z}} _ {p}} {\arg \min } u _ {\hat {\theta} (z)} \left(x _ {q} ^ {k}\right), z ^ {U} \left(x _ {q} ^ {k}\right) = \underset {z \in \tilde {\mathcal {Z}} _ {p}} {\arg \max } u _ {\hat {\theta} (z)} \left(x _ {q} ^ {k}\right), \tag {5} \\ s. t. \quad \hat {\theta} (z) = \arg \min  _ {\theta} \mathcal {L} _ {p i n n} (u _ {\theta} (\cdot), z). \\ \end{array}
$$
The estimated  $p$ -CI is, subsequently, given by
$$
L (x) = u _ {\psi} ^ {L} (x, x), \quad U (x) = u _ {\psi} ^ {U} (x, x). \tag {6}
$$
In fact, we can train a single model for both upper- and lower-bound predictions because the parameters are shared as in Equation (5). This is achieved by encoding an indicator into the input vector, so that  $u_{\psi}^{L}(x_{q},\cdot) = u_{\psi}(x_{q},\cdot , - 1)$  and  $u_{\psi}^{U}(x_{q},\cdot) = u_{\psi}(x_{q},\cdot ,1)$ . Furthermore, to balance the weights of the terms in the objective of Equation (5) more effectively, PINNs are separately trained with boundary conditions  $z^{L}(x_{q}^{1:K})$  and  $z^{U}(x_{q}^{1:K})$ . In this regard,  $u_{\psi}$  is obtained by
$$
\min  _ {\psi} \mathcal {L} _ {\text {E f f i}} \left(x _ {q} ^ {1: K}; \psi\right), \text {w h e r e} \quad \mathcal {L} _ {\text {E f f i}} := \frac {1}{2 K} \sum_ {k = 1} ^ {K} (1 - \lambda) \mathcal {L} \left(x _ {q} ^ {k}, x _ {q} ^ {k}; \psi\right) + \lambda \underset {x \in \Omega} {\mathbb {E}} \mathcal {L} \left(x _ {q} ^ {k}, x; \psi\right), \tag {7}
$$
$$
s. t. \mathcal {L} (x _ {q} ^ {k}, x; \psi) := (u _ {\psi} (x _ {q} ^ {k}, x, - 1) - u _ {\hat {\theta} (z ^ {L} (x _ {q} ^ {k}))} (x)) ^ {2} + (u _ {\psi} (x _ {q} ^ {k}, x, 1) - u _ {\hat {\theta} (z ^ {U} (x _ {q} ^ {k}))} (x)) ^ {2},
$$
and  $\lambda \in [0,1]$  is a weight to balance the two loss terms, of which the first aims to enforce close predictions of CIs at query points, and the second to utilize extra information in PDE solutions for better generalization. We refer to this model as EffiPICProp and provide the corresponding implementation in Algorithm 2 in Appendix H. Specifically, the EffiPICProp version with  $\lambda = 0$  is called SimPICProp, as it simply focuses on close predictions of confidence intervals at query points, while disregarding additional information at other locations.
5 Computational experiments
We provide three computational experiments that are relevant to physics-informed machine learning. We will start with a 1D pedagogical example to illustrate our proposed approaches. Next, we will move to more complicated problems involving standard representative equations met in many real-world problems including heat transfer, fluid dynamics, flows through porous media, and even epidemiology [21, 38]. Details regarding implementation, query points, and computational times can be found in Appendix J.
5.1 A Pedagogical Example
We consider the following equation, which has been adopted from the pedagogical example of [11],
$$
u _ {x x} - u ^ {2} u _ {x} = f (x), \quad x \in [ - 1, 1 ], \quad (\mathrm {P D E}), \tag {8}
$$
where  $f(x) = -\pi^2 \sin(\pi x) - \pi \cos(\pi x) \sin^2(\pi x)$ . For the standard BCs  $u(-1) = u(1) = 0$ , the exact solution is  $u(x) = \sin(\pi x)$ . The purpose of this one-dimensional PDE with only two boundary points, i.e., with a two-dimensional search space in Equation (4), is to demonstrate the applicability of our method in different data scenarios (see Appendix D). Further, the small size of the problem is ideal for comparing computationally expensive methods, such as exhaustive search (ES) and employing PICProp separately at all domain locations, with the more efficient implementation of Algorithm 2.