SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1: Pedagogical example: Boundary  $95\%$  CIs to be propagated to the rest of the domain. (a-b) Random samples follow  $\mathcal{N}(\mathbf{0},\sigma^2\mathbf{I})$ , where  $\sigma = 0.05$ . The black crosses correspond to 5,000 random samples from the distribution. A  $\chi^2$  CI is constructed using the only one available sample and a  $T^2$  CI is constructed using the five available samples. (c) Random samples follow  $\mathcal{U}([-1.5,1.5]\times [-1.5,1.5])$ . The black crosses correspond to 2,000 random samples from the distribution. A Hoeffding CI is constructed using the five available samples. All available samples are shown with red crosses.
We consider two types of noisy BCs (Figure 1). First, we consider BCs with Gaussian noise given as
$$
u (- 1) \sim \mathcal {N} \left(0, \sigma_ {1} ^ {2}\right), \quad u (1) \sim \mathcal {N} \left(0, \sigma_ {2} ^ {2}\right), \tag {9}
$$
where the standard deviations are  $\sigma_{1} = \sigma_{2} = 0.05$ , i.e., the data is assumed to be Gaussian and the standard deviations are known (or estimated if unknown). We sample one or five datapoint(s) for each side of the boundary, for each case, respectively. Given the dataset, a  $\chi^2$  CI and a  $T^2$  CI are constructed according to the corresponding problem scenarios discussed in Appendix D; see also Figure 1(a),(b).
Second, we consider BCs with uniform noise given as
$$
u (- 1) \sim \mathcal {U} (- D _ {1}, D _ {1}), \quad u (1) \sim \mathcal {U} (- D _ {2}, D _ {2}), \tag {10}
$$
where  $D_{1} = D_{2} = 1.5$  are assumed to be known. We sample five datapoints for each boundary side. Given the dataset, a Hoeffding CI is constructed; see also Figure 1(c).
For the query set, six points are used for implementing SimPICProp and EffiPICProp in Section 4.2, whereas 41 points are used for implementing the brute-force version of PICProp, which amounts to solving the bi-level optimization problem of Equation (4) for each point. The implementation details of the PICProp-based methods are summarized in Table J.1. Further, 5,000 BCs are sampled for solving Equation (4) with ES, which can be considered as a reference solution.
Figure 2: Pedagogical example: CI predictions for the entire domain  $x \in [-1,1]$ . Left: Noise is Gaussian with known standard deviation, and a  $\chi^2$  CI is constructed at the boundary. Middle: Noise is Gaussian with unknown standard deviation, and a  $T^2$  CI is constructed at the boundary. Right: Uniform noise with known bounds, and a Hoeffding CI is constructed at the boundary.
The CI predictions for the three cases are visualized in Figure 2, whereas the meta-learning curves can be found in Appendix K.1. Note that the sizes of the CIs are approximately equal to  $0.1 - 0.2$  near the boundaries, which are comparable with the propagated CI sizes of Figure 1, and become larger around  $x$  of  $\pm 0.5$ . In all cases, the PICProp results match those of ES, despite the fact that PICProp requires solving a challenging bi-level optimization method with approximate implicit gradient methods. Further, the efficient method EffiPICProp performs accurately as compared to its reference method PICProp, while retaining a significantly lower computational cost (see Figure 3 for  $\chi^2$  results and Figure K.4 K.5 for  $T^2$  and Hoeffding results). On the other hand, the predictions of SimPICProp match the brute-force PICProp results only on the query points, but generalize poorly on other locations. In contrast, EffiPICProp, which is based on a meta-model that fits the PDE solutions for multiple query points, generalizes well on unseen locations.
Some additional experimental results are summarized in Appendix K.1 for the completeness of our empirical study: Firstly, a comparison among PICProp and EffiPICProp with various  $\lambda$ s is provided in Figure K.3 to better demonstrate how the balance between two EffiPICProp loss terms in Equation (7) affects the CI prediction. Next, we conduct an additional experiment to empirically verify the validity of EffiPICProp CIs in Appendix K.1.4. Finally, a brief comparison with Bayesian PINNs is presented in Appendix L.
5.2 Two-dimensional Poisson equation
Next, we consider a two-dimensional Poisson equation given as
$$
\begin{array}{l} u _ {x x} + u _ {y y} = f (x, y), \quad (x, y) \in [ - 1, 1 ] \times [ - 1, 1 ], \quad (\text {P D E}), \tag {11} \\ u (x, y) = e ^ {x} + e ^ {y}, \quad x = \pm 1 \text {o r} y = \pm 1, \quad (\mathrm {B C s}). \\ \end{array}
$$
Figure 3: Pedagogical example,  $\chi^2$ : Although the results of both SimPICProp and EffiPICProp match PICProp results on the query points, EffiPICProp generalizes better on unseen locations.
The exact solution is  $u(x,y) = e^{x} + e^{y}$ , and the corresponding  $f(x,y)$  is obtained by substituting  $u(x,y)$  into Equation (11). This example serves to demonstrate the effectiveness of our method for treating cases with large numbers of noisy BC datapoints. In such cases, an exhaustive search approach for solving the bi-level optimization of Algorithm 1 generally fails to identify the global optimum, unless we increase the number of trials and the computational cost at prohibitive levels.
We propagate the following fixed CI from the boundary to the rest of the domain  $(x,y)\in (-1,1)\times (-1,1)$ :
$$
u (x, y) \in \left[ e ^ {x} + e ^ {y} - 0. 0 5, e ^ {x} + e ^ {y} + 0. 0 5 \right], \quad x = \pm 1 \text {o r} y = \pm 1. \tag {12}
$$
To solve the problem, 10 boundary points are sampled for each side of the domain, i.e., 40 boundary points in total are used for training and uncertainty is considered at all points. The search space is thus 40-dimensional. The implementation details of the PICProp-based methods are summarized in Table J.1, whereas 4,000 steps of standard PINN training are run for each BC for ES. For a fair comparison, 121 query points for PICProp are selected in the domain, see Figure J.1, and the computational time is approximately the same as of ES with 1,000 trials. Moreover, we include the ES result with 5,000 trials, which is considered to be exhaustive, as a reference.
Figure 4: 2-D Poisson equation: Slices of CI predictions for  $x \in \{-0.5, 0.0, 0.5, 1.0\}$ . EffiPICProp can identify the solution to the bi-level optimization problem more efficiently and accurately, as compared with the computationally expensive, search-based method ES.
Figure 4 plots the slices of CI predictions for  $x \in \{-0.5, 0.0, 0.5, 1.0\}$ , whereas Appendix K.2 includes slices along the  $y$  axis and the heatmaps over the entire domain. For reference, the size
of the CIs obtained with EffiPICProp is approximately 0.1 across the domain, which is equal to the size of the propagated CI of Equation (12). The predicted CI size is also larger in most cases than that of the ES-based CIs. We conclude that 1,000 trials for ES are clearly not sufficient, as the intervals of ES with 5,000 trials are significantly wider; thus, a global optimum is not reached with 1,000 trials. In contrast, EffiPICProp converges to similar or wider intervals compared to ES with 5,000 trials. Overall, the optimization-based method EffiPICProp can identify the solution to the bi-level optimization problem more efficiently and accurately, as compared with the computationally expensive, search-based method ES.
5.3 One-dimensional, time-dependent Burgers equation
In this example, we study the Burgers equation given as
$$
u _ {t} + u _ {x} - \nu u _ {x x} = 0, \quad (x, t) \in [ - 1, 1 ] \times [ 0, 1 ], \quad (\text {P D E}),
$$
$$
u (x, t = 0) = - \sin (\pi x), \quad u (x = - 1, t) = u (x = 1, t) = 0, \quad (\mathrm {B C s}), \tag {13}
$$
where  $\nu = \frac{0.01}{\pi}$  is the viscosity parameter. We consider clean data for the BCs and noisy data for the initial conditions (ICs). Specifically, for the ICs, we propagate the following fixed CI from  $t = 0$  to the rest of the time domain  $t \in (0,1]$  and for all  $x \in [-1,1]$ :
$$
u (x, t = 0) \in [ - \sin (\pi x) - 0. 2, - \sin (\pi x) + 0. 2 ]. \tag {14}
$$
This problem serves to demonstrate the reliability of our method even in challenging cases with steep gradients or discontinuities of the PDE solution. For solving the problem using PINNs, we consider 200 boundary points, 256 initial points, and 10,000 samples in the interior domain. The 256 initial points are evenly spaced along  $[-1,1]$ . The implementation details of the PICProp-based methods are summarized in Table J.1, whereas 2,500 and 5,000 trials are considered for ES. The 246 PICProp query points are visualized in Figure J.1 and the respective computational time is approximately equal to that of ES with 2,500 trials.
Slices of CI predictions for  $t \in \{0.0, 0.25, 0.5, 0.75\}$  are plotted in Figure 5, whereas Appendix K.3 includes the respective meta-learning curves. We conclude that EffiPICProp, although computationally cheaper, in most cases provides wider, i.e., more conservative, CIs, as compared to ES with varying numbers of trials. It also succeeds in capturing the expected increase in uncertainty as the query points move away from the source of uncertainty, which corresponds to larger  $t$  values. Specifically, the size of the CIs at  $t = 0$  is approximately 0.4, which is equal to the size of the propagated CI in Equation (14), and becomes approximately 2 at  $t = 0.75$  and near  $x = 0$ . On the other hand, since we consider 256 random initial points, the search space of ES is 256-dimensional. It requires a prohibitively large number of trials to identify the global optima of Equation (4). Finally, note that the results of EffiPICProp in Figure 5 correspond to  $\lambda = 0$  in Equation (7), because we found that the additional information provided by the second loss term in Section 4.2 adversely affects generalization. This is partly due to the large number of query points in this problem.
Physics-informed machine learning is transforming the computational science field in an unprecedented manner, by solving ill-posed problems, involving noisy and multi-fidelity data as well as missing functional terms, which could not be tackled before. However, uncertainty quantification is necessary for deploying such methods in critical applications. In this work, we considered a novel problem pertaining to propagating randomness, in the form of CIs, from data locations (at the boundary but can easily be extended to inner or collocation data points) to the entire domain, with probabilistic guarantees. We proposed PICProp, a method based on bi-level optimization, and its efficient version, with a theorem guaranteeing the validity of the obtained CIs (Algorithms 1-2 and Theorem 4.1). To the best of our knowledge, this is the first work on joint CI estimation with theoretical guarantees in the context of physics-informed learning, while our method can be applied in diverse practical scenarios without imposing any distributional assumptions.
In the computational experiments of Section 5, involving various partial differential equations, we demonstrated the effectiveness and efficiency of our proposed methods. Specifically, the predicted CIs are wider, i.e. more conservative, in most cases, as compared with the more computationally demanding exhaustive search for solving the bi-level optimization. Although not encountered in the
Figure 5: Burgers equation: Slices of CI predictions for  $x \in \{0.0, 0.25, 0.5, 0.75\}$ . EffiPICProp, although computationally cheaper, in most cases provides wider, i.e., more conservative CIs, as compared to ES with varying numbers of trials. To interpret the cost of ES, note that its search space is 256-dimensional and requires a prohibitively large number of trials to identify the global optima of Equation (4).
experiments, a potential limitation of our method is that the obtained CIs can be loose in some cases, even when propagating a tight CI from the boundaries, due to the max/min operations in the bi-level optimization of Equation (3). Because of the theoretical challenges involved, we plan to study this limitation and perform pertinent experiments in a future work.
This material is based upon work supported by the Google Cloud Research Credit program with the award (6NW8-CF7K-3AG4-1WH1). The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (https://www.nssc.sg).
[1] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243-297, 2021.
[2] Vidmantas Bentkus. An extension of the hoeffding inequality to unbounded random variables. Lithuanian Mathematical Journal, 48(2):137-157, 2008.
[3] BVSS Bharadwaja, Mohammad Amin Nabian, Bharatkumar Sharma, Sanjay Choudhry, and Alankar Alankar. Physics-informed machine learning and uncertainty quantification for mechanics of heterogeneous materials. Integrating Materials and Manufacturing Innovation, pages 1-21, 2022.
[4] Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng, editors. Handbook of Markov Chain Monte Carlo. Chapman and Hall/CRC, may 2011.
[5] Nicolas Brosse, Alain Durmus, and Eric Moulines. The promises and pitfalls of stochastic gradient Langevin dynamics. Advances in Neural Information Processing Systems, 31, 2018.
[6] Min Cai, George Em Karniadakis, and Changpin Li. Fractional heir model and data-driven predictions of Covid-19 dynamics of omicron variant. *Chaos: An Interdisciplinary Journal of Nonlinear Science*, 32(7):071101, 2022.
[7] Kamil Ciosek, Vincent Fortuin, Ryota Tomioka, Katja Hofmann, and Richard Turner. Conservative uncertainty estimation by fitting prior networks. In International Conference on Learning Representations, 2020.
[32] Ian Waudby-Smith and Aaditya Ramdas. Confidence sequences for sampling without replacement. Advances in Neural Information Processing Systems, 33:20204-20214, 2020.
[33] Zheng Wen, Ian Osband, Chao Qin, Xiuyuan Lu, Morteza Ibrahimi, Vikranth Dwaracherla, Mohammad Asghari, and Benjamin Van Roy. From predictions to decisions: The importance of joint predictive distributions. arXiv e-prints, pages arXiv-2107, 2021.
[34] Liu Yang, Xuhui Meng, and George Em Karniadakis. B-pinns: Bayesian physics-informed neural networks for forward and inverse pde problems with noisy data. Journal of Computational Physics, 425:109913, 2021.
[35] Liu Yang, Dongkun Zhang, and George Em Karniadakis. Physics-informed generative adversarial networks for stochastic differential equations. SIAM Journal on Scientific Computing, 42(1):A292-A317, 2020.
[36] Yibo Yang and Paris Perdikaris. Adversarial uncertainty quantification in physics-informed neural networks. Journal of Computational Physics, 394:136-152, 2019.
[37] Dongkun Zhang, Lu Lu, Ling Guo, and George Em Karniadakis. Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems. Journal of Computational Physics, 397:108850, 2019.
[38] Zongren Zou, Xuhui Meng, Apostolos F Psaros, and George Em Karniadakis. Neuraluq: A comprehensive library for uncertainty quantification in neural differential equations and operators. arXiv preprint arXiv:2208.11866, 2022.
A Indicative applications
Our proposed method, delineated in Algorithms 1 and 2 of Appendix H, provides a way to quantify how the data uncertainty affects the obtained solution. In addition, if the unseen clean data lies within the constructed/given data bounds, the ground-truth solution lies within our predicted CIs. Some indicative applications include:
1. In additive manufacturing problems, understanding, through our computed CIs, how the model responds to different material properties arising from the manufacturing process can help in model correction and thus quality control (see [31]).
2. Further, in material science, cracks affect the elasticity constants which together with geometric properties affect deformations (see [3]). The range of these deformations can be identified with our method. Accurately predicting the upper bound of such deformations can help designers to adjust cross-sectional properties of structural elements to better account for potential cracks.
3. More generally, our model can be extended to account for PDE model uncertainty. For example, PINNs have been used for short-term predictions of infected cases and deaths due to COVID-19 (see [6]). In such problems, our approach can be used to provide bounds of uncertainty for the predicted variables of interest. Accurately predicting the upper bound of infected cases can help the states better prepare for short-term hospitalization needs.
B Additional details regarding standard UQ
Standard UQ considers that the data is generated from a distribution  $p(u|x, \theta = \theta^{*})$ , where  $\theta^{*}$  is the NN parameters that are assumed to represent exactly the PDE solution  $u(x)$ . Given a dataset  $z$ , standard UQ approximates the solution  $u_{\theta^{*}}(x)$  with the distribution
$$
p (u | x; z) = \int p (u | x, \theta) p (\theta | z) d \theta , \tag {15}
$$
where  $p(\theta |z)\propto p(\theta)p(z|\theta)$  according to Bayes' theorem and  $p(z|\theta)$  is the data likelihood. However, as a closed-form  $p(\theta |z)$  is typically intractable, UQ draws approximate samples from  $p(\theta |z)$ , denoted as  $\{\hat{\theta}_j\}_{j = 1}^{N_\theta}$ . Next, it constructs  $\hat{p} (u|x;z)$  for each  $x$  based on the collected samples to approximate  $p(u|x;z)$ . Typically, a Gaussian distribution is fitted to the PINN solutions  $\{u_{\hat{\theta}_j}(x)\}_{j = 1}^{N_\theta}$  and its corresponding statistical intervals are used in lieu of CIs for the PDE solution. Loosely speaking, statistical intervals of  $\{\hat{\theta}_j\}_{j = 1}^{N_\theta}$  can be viewed as credible intervals from Bayesian statistics, and intervals of  $\{u_{\hat{\theta}_j}(x)\}_{j = 1}^{N_\theta}$  as the corresponding prediction intervals.
Alternatively, one could use a concentration inequality to construct directly a prediction interval using the samples  $\{u_{\hat{\theta}_j}(x)\}_{j=1}^{N_\theta}$ . Nevertheless, such intervals still correspond to the Bayesian estimate  $\mathbb{E}_{\theta \sim p(\theta | \mathcal{D})} u_\theta$ , rather than to the exact solution  $u(x)$ , unless further assumptions are made. Even if the assumption that the Bayes estimate corresponds to  $\theta^*$  is made, i.e.,  $\theta^* = \mathbb{E}_{\theta | \mathcal{D}} \theta$ , the typically considered  $\mathbb{E}_{\theta | \mathcal{D}} u_\theta(x)$  is not guaranteed to equal  $u_{\mathbb{E}_{\theta | \mathcal{D}} \theta}(x)$  as  $u_\theta$  can be arbitrary nonlinear (see also Appendix C).
Standard UQ approaches have several other limitations. For example, strong assumptions regarding the noise distribution and data likelihood are often required, and depending on the problem it is notoriously difficult to tune Bayesian NNs [5]. Furthermore, it is not possible to obtain/sample  $p(u|x,z)$  exactly in practical problems and thus, additional approximations are required, typically referred to as approximate posterior inference. In addition, UQ approaches for PINNs are not supported by theoretical guarantees regarding the quality of uncertainty propagation due to the physics-informed constraints [11]. Finally, for the case of multiple available datasets, see Appendices C and E for the limitations of Monte Carlo-based approaches.
C Additional details regarding Monte Carlo Simulation
Assume that we have  $m$  i.i.d. datasets denoted as  $\{z^{(j)}\}_{j = 1}^{m}$  and a 0-proper data-driven PDE solver (see Definition 3.3). It is natural to consider solving the PDE with each dataset in a Monte Carlo manner and then using a concentration inequality to construct CIs. However, a CI for the exact solution  $u(x)$  cannot be identified without further assumptions. The reason is that we only have access to  $\mathbb{E}[u(x;Z)]$ , rather than to  $u(x)$  given as  $u(x) = u(x;\bar{Z}) = u(x;\mathbb{E}[Z])$ , considering data-driven solutions from a 0-proper solver. Clearly, the difference between the two is in the form of  $\mathbb{E}[f(X)]\neq f(\mathbb{E}[X])$ , where  $f$  in our case is the solver. In Figure C.1 we provide an illustration of this point as applied to linear/convex/concave functions, as s straightforward consequence of Jensen's inequality.
Figure C.1: Difference between  $\mathbb{E}[f(X)]$  and  $f(\mathbb{E}[X])$ . (a) Linear function  $f(x) = x$ ,  $\mathbb{E}[f(X)] = f(\mathbb{E}[X])$ ; (b) Convex function  $f(x) = (e^x - 1)/(e - 1)$ ,  $\mathbb{E}[f(X)] > f(\mathbb{E}[X])$ ; (c) Concave function  $f(x) = \sin(\pi x/2)$ ,  $\mathbb{E}[f(X)] < f(\mathbb{E}[X])$ .
D Confidence Interval Construction with Data
Consider a dataset  $\{z^{(j)}\}_{j = 1}^{m}$ , which contains  $m$  independent samples of an  $n$ -dimensional random vector  $Z$ . Based on the data  $\{z^{(j)}\}_{j = 1}^{m}$ , we construct a CI for  $Z$ , denoted as  $\tilde{\mathcal{Z}}_p$ , where  $p$  is the confidence level. Equivalently,  $Pr(\bar{z}\in \tilde{\mathcal{Z}}_p)\geq p$ .
In this regard, we consider the following problem scenarios:
1. The random vector  $Z$  follows a multivariate Gaussian distribution with known covariance matrix  $\Sigma$  of rank  $n$ . Given one observation  $z$ , i.e.,  $m = 1$ , the chi-squared statistic  $\chi^2 = (z - \mu)^T\Sigma^{-1}(z - \mu)$  follows a chi-squared distribution with  $n$  degrees of freedom. The corresponding  $p$ -CI is  $\tilde{\mathcal{Z}}_p = \{z'|||z - z'||_{\Sigma^{-1}}\leq k_{n,p}^{\chi^2}\}$ , where  $||z - z'||_{\Sigma^{-1}} = \sqrt{(z - z')^{T\Sigma^{-1}}(z - z')}$  and  $k_{n,p}^{\chi^2}$  is the squared root of the upper  $(1 - p) / 2$  critical value of the chi-squared distribution with  $n$  degrees of freedom; see Figure 1 for an example corresponding to the case of  $n = 2$ .
2. The random vector  $Z$  follows a multivariate Gaussian distribution with unknown covariance matrix  $\Sigma$  and  $m$  observations are available. The sample mean and covariance are estimated as  $\bar{z} = \frac{1}{m}\sum_{i=1}^{m}z_i$  and  $\hat{\Sigma} = \frac{1}{m-1}\sum_{i=1}^{m}(z_i - \bar{z})(z_i - \bar{z})^T$ , respectively. The Hotelling's  $t$ -squared statistic  $t^2 = N(\bar{z} - \mu)^T\hat{\Sigma}^{-1}(\bar{z} - \mu)$  follows a Hotelling's  $T$ -squared distribution with parameters  $n, m-1$ . The corresponding  $p$ -CI is  $\tilde{\mathcal{Z}}_p = \{z'|||\bar{z} - z'||_{\hat{\Sigma}^{-1}} \leq m^{-1/2}k_{n,m-1,p}^{T^2}\}$ , where  $||\bar{z} - z'||_{\hat{\Sigma}^{-1}} = \sqrt{(\bar{z} - z')^T\hat{\Sigma}^{-1}(\bar{z} - z')}$  and  $k_{n,m-1,p}^{T^2}$  denotes the squared root of the upper  $(1-p)/2$  critical value of the aforementioned Hotelling's  $T$ -squared distribution; see Figure 1 for an example corresponding to the case of  $n = 2$  and  $m = 5$ .
3. The components of the random vector  $Z$  follow an unknown distribution, and  $m$  independent observations are available. For this case, a concentration inequality can be used for each component, i.e., for each data location, and the obtained bounds can be combined using a union of bounds for obtaining a  $p$ -CI (see also Appendix E). Note that many popular concentration inequalities require specific assumptions for the unknown distribution besides the fact that independent samples are available. For example, Hoeffding's inequality and its variants require the random variable to be bounded. [2] extends Hoeffding's inequality to unbounded random variables, but it is limited to Pareto-type tails. Other concentration inequalities without heavy assumptions for the distribution, such as Chebyshev's inequality and Chernoff's inequality, require various true population statistics (e.g., the population mean and variance) for computing the bounds, which are not available in the cases we consider. In this regard, [27] provides a Chebyshev-type bound with estimated mean and variance. [29] further extends the bound of [27] to the multivariate case.
4. The CI  $\tilde{\mathcal{Z}}_p$  can alternatively be given by domain experts.
E Joint Confidence Intervals for PDE Solutions