Title: PICProp: Physics-Informed Confidence Propagation for Uncertainty Quantification
Venue: NeurIPS 2023
Accepted: yes
Keywords: physics-informed learning

uncertainty quantification

deep learning
Abstract: Standard approaches for uncertainty quantification in deep learning and physics-informed learning have persistent limitations. 
Indicatively, strong assumptions regarding the data likelihood are required, the performance highly depends on the selection of priors, and the posterior can be sampled only approximately, which leads to poor approximations because of the associated computational cost.
This paper introduces and studies confidence interval (CI) estimation for deterministic partial differential equations as a novel problem.
That is, to propagate confidence, in the form of CIs, from data locations to the entire domain with probabilistic guarantees.
We propose a method, termed Physics-Informed Confidence Propagation (PICProp), based on bi-level optimization to compute a valid CI without making heavy assumptions.
We provide a theorem regarding the validity of our method, and computational experiments, where the focus is on physics-informed learning. Code is available at https://github.com/ShenQianli/PICProp.

PICProp: Physics-Informed Confidence Propagation for Uncertainty Quantification

Abstract

Standard approaches for uncertainty quantification in deep learning and physicsinformed learning have persistent limitations. Indicatively, strong assumptions regarding the data likelihood are required, the performance highly depends on the selection of priors, and the posterior can be sampled only approximately, which leads to poor approximations because of the associated computational cost. This paper introduces and studies confidence interval (CI) estimation for deterministic partial differential equations as a novel problem. That is, to propagate confidence, in the form of CIs, from data locations to the entire domain with probabilistic guarantees. We propose a method, termed Physics-Informed Confidence Propagation (PICProp), based on bi-level optimization to compute a valid CI without making heavy assumptions. We provide a theorem regarding the validity of our method, and computational experiments, where the focus is on physics-informed learning. Code is available at https://github.com/ShenQianli/PICProp.

1 Introduction

Combining data and physics rules using deep learning for solving differential equations [25] and learning operator mappings [20] has recently attracted increased research interest. In this context, physics-informed machine learning is transforming the computational science field in an unprecedented manner, by solving ill-posed problems, involving noisy and multi-fidelity data as well as missing functional terms, which could not be tackled before [16]. Nevertheless, uncertainty - mainly due to data noise and neural network over-parametrization - must be considered for using such models safely in critical applications [24]. Despite extensive work on uncertainty quantification (UQ) for deep learning [1], standard UQ has persistent limitations. Indicatively, although for most practical applications where only confidence intervals (CIs) are required, the harder problem of UQ is often solved first using simplifying assumptions, and the CIs are subsequently computed. Further, strong assumptions regarding the data likelihood are required, the performance highly depends on

the selection of priors, and the posterior can be sampled only approximately, which leads to poor approximations because of the associated computational cost.

In this paper, we introduce and study CI estimation for addressing deterministic problems involving partial differential equations (PDEs) as a novel problem. That is, to propagate confidence, in the form of CIs, from data locations to the entire domain with probabilistic guarantees. We name our framework Physics-Informed Confidence Propagation (PICProp) and propose a practical method based on bi-level optimization to identify valid CIs in physics-informed deep learning. Our method can be applied in diverse practical scenarios, such as propagating data CIs, either estimated or given by experts, to the entire problem domain without distributional assumptions. Indicative applications are provided in Appendix A.

PICProp provides distinct advantages over traditional UQ approaches in the context of physicsinformed deep learning. Specifically, PICProp does not rely on strong assumptions regarding the data likelihood and constructs theoretically valid CIs. Further, PICProp offers a unique capability to construct joint CIs for continuous spatio-temporal domains. In contrast, UQ methods typically construct marginal intervals for discrete locations. Furthermore, due to the conservative propagation of uncertainty, PICProp produces conservative CIs, which is preferred in critical and risk-sensitive applications [7, 13], considering the significant harm of non-valid confidence intervals. This sets it apart from commonly used UQ methods like deep ensembles [18] and Monte-Carlo dropout [28, 10], which are known to be overconfident in practice [9, 7].

The paper is organized as follows: In Section 2 (and its Appendices B & C), we introduce the PDE problem setup and discuss related work. In Section 3 (and its Appendices D-F), we introduce the problem of CI estimation for PDEs and extensively describe joint solution predictions. Next, we formulate this novel problem as a bi-level optimization problem in Section 4 and Appendices G & H, and adopt various algorithms for computing the required implicit gradients in Appendix I. Finally, we evaluate our method on various forward deterministic PDE problems in Section 5 and Appendices J-L, and discuss our findings in Section 6.

2 Background

2.1 Problem Setup

Consider the following general PDE describing a physical or biological system:

where x is the D x -dimensional space-time coordinate, Ω is a bounded domain with boundary Γ , and F as well as B are general differential operators acting on Ω and Γ , respectively. The PDE coefficients are denoted by λ ( x ) , the boundary term by b ( x ) , and the solution by u ( x ) .

In this paper, we focus on the data-driven forward deterministic PDE problem, where (i) operators F , B and coefficients λ are deterministic and known; (ii) datasets D f = { x i f } N f i =1 for the force condition and D b = { ( x j b , b j ) } N b j =1 for the boundary condition are given; (iii) uncertainty only occurs in the boundary term due to data noise, i.e., b j = b ( x j b ) + ϵ j for j = 1 , ..., N b , and the rest of the dataset is considered to be deterministic without loss of generality. We denote the function obtained by utilizing the dataset z = {D f , D b } for approximating the solution u ( x ) of Equation (1) with a data-driven solver A as u A ( x ; z ) .

Clearly, different datasets z contaminated by random noise lead to different solutions u ( x ; z ) and thus, to a distribution of solutions at each x . It follows that the solution u ( x ; Z ) can be construed as stochastic, where Z is used to denote the dataset as a random variable and z to denote an instance of Z . In this regard, this paper can be viewed as an effort towards a universal framework, with minimal assumptions, for constructing CIs for solutions of nonlinear PDEs.

2.2 Related Work

Physics-Informed Neural Networks The physics-informed neural network (PINN) method, developed by [25], addresses the forward deterministic PDE problem of Equation (1) by constructing a neural network (NN) approximator u θ ( x ) , parameterized by θ , for modeling the approximate solution

u ( x ; z ) . The approximator u θ ( x ) is substituted into Equation (1) via automatic differentiation for producing f θ = F [ u θ , λ ] and b θ = B [ u θ , λ ] . Finally, the optimization problem for obtaining θ in the forward problem scenario given a dataset z is cast as

and { w f , w b } are objective function weights for balancing the various terms in Equation (2); see [23]. Following the determination of ˆ θ , u ˆ θ can be evaluated at any x in Ω . PINN can be scalled up for high-dimensional PDEs [14].

Uncertainty Quantification for Physics-Informed Learning Although UQ for physics-informed learning is not extensively studied, the field is gaining research attention in recent years. Indicatively, [37] employed dropout to quantify the uncertainty of NNs in approximating the modal functions of stochastic differential equations; [36] presented a model based on generative adversarial networks (GANs) for quantifying and propagating uncertainty in systems governed by PDEs; [35] developed a new class of physics-informed GANs to solve forward, inverse, and mixed stochastic problems; [11] further studied Wasserstein GANs for UQ in solutions of PDEs; [34] proposed Bayesian PINNs for solving forward and inverse problems described by PDEs with noisy data. Although this field is always evolving, we refer to it in this paper as 'standard' UQ in order to distinguish it from our proposed method in Section 4. In Appendices B-C, we summarize the limitations of standard UQ, and we direct interested readers to the review paper [24] for more information.

Interval Predictor Model (IPM) The Interval Predictor Model (IPM) in regression analysis has a similar motivation as our approach. A function of interest is approximated by an IPM and the bounds on the function are obtained. [26] proposed a technique using a double-loop Monte Carlo algorithm to propagate a CI using the IPM. The technique requires sampling epistemic and aleatory parameters from uniform distributions to estimate the upper and lower bounds of the IPM. Our approach is different for two reasons. First, we fit a PINN model directly and thus, sampling epistemic parameters may not be feasible. Second, we do not require a sampling procedure. Instead, we cast the CI estimation as a bi-level optimization problem and solve it using hyper-gradient methods.

3 Confidence Intervals for Solutions of Nonlinear Partial Differential Equations

In this section, we provide the definition of CI for the exact PDE solution and introduce our main assumption.

Definition 3.1 (Joint CI for PDE Solution) . A pair of functions ( L Z , U Z ) ∈ R Ω × R Ω is a p -CI for the exact PDE solution u ( x ) if the random event E := {{ L Z ( x ) ≤ u ( x ) ≤ U Z ( x ) , ∀ x ∈ Ω } holds with probability at least p .

Note that this interval refers to the exact solution u ( x ) . Therefore, a novel problem is posed in Definition 3.1, pertaining to propagating randomness, in the form of CIs, from the data locations, typically on the boundary, to the rest of the domain. The subscript Z in ( L Z , U Z ) signifies that the CI is random as it depends on the random dataset Z . The available data can be an instance of Z , denoted as z , or a collection of instances; see Appendix D for the considered problem scenarios. Such instances can, for example, be drawn from repeating independent experiments. The subscript Z in ( L Z , U Z ) is dropped if there is no ambiguity.

We also note that the joint CI (also referred to as confidence sequence [17, 32]) in Definition 3.1 corresponds to joint predictions u ( · ) over the whole domain Ω , rather than to marginal predictions u ( x ) for a given x . The importance of joint predictions as an essential component in UQ has been emphasized in recent works for active learning [30], sequential decision making [33], and supervised learning [22]. Nevertheless, the present paper is the first work to consider joint predictions, and general CI estimation, in the context of physics-informed learning. In Appendix E, we provide several theoretical remarks delineating the differences between marginal and joint predictions, as well as unions of bounds, which are typically used in standard UQ. In Appendix F, we highlight these differences with a toy example.

Next, to relate the PINN solution with the exact PDE solution u ( x ) of Definition 3.1, we make the following assumption regarding the noise at the boundary.

Assumption 3.2 (Zero-mean Noise) . The noise on the boundary is zero-mean, i.e., E [ ϵ j ] = 0 , ∀ j ∈ { 1 , . . . , N b } .

Next, a clean dataset (without noise), denoted as ¯ z , can be considered as the expectation of Z , i.e., ¯ z = E [ Z ] . In this regard, consider a data-driven PDE solver A , which yields an approximate solution u A ( x ; z ) given a dataset z . We define the properness of A based on the concept of a clean dataset.

Definition 3.3 (Proper PDE Solver) . A PDE solver A is η -proper with respect to a clean dataset ¯ z , if max x ∈ Ω | u A ( x ; ¯ z ) -u ( x ) | ≤ η .

̸

Henceforth, we only consider proper PDE solvers. That is, a powerful solver is available such that the absolute error of the solution obtained by the solver with a clean dataset is bounded. In all cases, we assume that such an informative clean dataset exists, but is not available. Further, the constant η , which is only related to ¯ z and A , is considered known and fixed in the computational experiments. Note that no assumptions are made about u ( x ; z ) for z = ¯ z , i.e., there are no assumptions about the PINN solver robustness against noisy data. Finally, in all NN training sessions we consider no NN parameter uncertainty, although it is straightforward to account for it by combining our method with Deep Ensembles [18], for instance. The aim of this paper is to propose a principled and theoretically guaranteed solution approach and thus, we refrain from applying such ad hoc solutions in the computational experiments and plan to consider parameter uncertainty rigorously in a future work. Nevertheless, we provide in Appendix L a comparison with Bayesian PINNs for completeness.

4 Method

In this section, we introduce our PICProp method and provide the accompanying theorem, as well as practical implementation algorithms. Further, we discuss the computational cost of our method and propose a modification for increasing efficiency. Specifically, based on Definition 3.1, we address the problem of propagating the uncertainty from the boundary to arbitrary locations x ∈ Ω of the domain. Equivalently, this relates to obtaining the corresponding p -CIs, i.e., obtaining the functions ( L, U ) ∈ R Ω × R Ω in Definition 3.1.

4.1 Physics-Informed Confidence Propagation (PICProp)

Our method comprises two steps: (i) a p -CI ˜ Z p for the clean data ¯ z is constructed. (ii) ˜ Z p is propagated to any point x ∈ Ω of the domain by solving the following problem,

which searches for configurations of points z within the interval ˜ Z p such that the predicted values for u ( x ) given by the η -proper solver A are minimized or maximized, respectively.

The CI in the first step can either be given or constructed from data. The former case relates to various practical scenarios (e.g., the error ranges of measurement sensors with a certain confidence degree are known or given by experts), whereas the latter serves as a data-driven scheme, where less prior knowledge is required. Techniques to construct the CI in the first step depend on the data (noise) distribution. For example, if the noise is assumed to be Gaussian, the corresponding CI can be identified by chi-squared or Hotelling's T -squared statistics. If the noise distribution is not known, a concentration inequality can be used to identify the CI. Relevant details are provided in Appendix D. See also Figure 1 for an illustration.

The interval constructed via Equation (3) is a joint p -CI for the whole domain Ω of u . This fact is formalized in the next theorem (see Appendix G for the corresponding proof).

Theorem 4.1. Consider a p -CI for ¯ z denoted as ˜ Z p , constructed such that Pr (¯ z ∈ ˜ Z p ) ≥ p . Then the functions ( L, U ) obtained by solving Equation (3) define a p -CI for u .

Although the robustness of the PDE solver does not affect the validity of Theorem 4.1, it nevertheless affects the quality of intervals constructed via Equation (3). Specifically, although valid CIs can always be constructed with arbitrary proper PDE solvers by solving Equation (3), a robust PDE solver less sensitive to data noise is expected to yield tighter CIs compared to a less robust solver.

Note that the confidence guarantee of the constructed CIs corresponds to a probability that is greater than, rather than equal to, p , which means that the constructed CIs are conservative. From a design perspective, conservative uncertainty estimation is the preferred route for critical and risk-sensitive applications [7, 13], although it can be more costly.

Next, a brute force and computationally expensive approach to solve Equation (3) pertains to traversing z ∈ ˜ Z p in an exhaustive search (ES) manner and invoking a PDE solver for each candidate z . Alternatively, we propose to first employ a NN for obtaining u ˆ θ ( z ) by minimizing the physicsinformed loss of Equation (2) and approximating u ( x ; z ) for each z ∈ ˜ Z p . In this regard, if a PINN is assumed to be an η -proper PDE solver, Equation (3) can be further extended to a bi-level optimization problem cast as

Nevertheless, explicit gradient ∇ z u calculation is intractable, as multiple steps of gradient descent for the NN are required for a reliable approximation. Despite this fact, bi-level optimization involving large-scale parameterized models such as NNs, with main applications in meta-learning, has been extensively studied in recent years and has resulted to numerous implicit gradient approximation methods (see Appendix I). In this regard, Algorithm 1 in Appendix H summarizes our method for solving efficiently Equation (3) by using such approximations. In passing, note that η = 0 is considered for all implementations in this work.

4.2 Efficient Physics-Informed Confidence Propagation (EffiPICProp)

If joint CIs are required for multiple query points, denoted by the set X q , Algorithm 1 has to be used multiple times to solve Equation (4) for each and every point x in the set. This clearly incurs a high computational cost. To address this issue, a straightforward approach pertains to training an auxiliary model parameterized by ψ that models the CI as ( L ψ ( x q ) , U ψ ( x q )) for all query points x q ∈ X q , or more generally of the entire domain Ω . However, this naive regression using solutions of the bi-level optimization at a set of locations, which we refer to as SimPICProp, does not provide accurate results when the size of the query set is small; see computational results in Section 5.1.

For improved generalization properties with limited query points, we propose to train a two-input parameterized meta-model u ψ ( x q , x ) that models the resulting PDE solution u ( x ) , via Equation (4), for each different query point. For example, note that L ( x ) in Equation (4) evaluates the NN u ˆ θ ( z ) at the location of the query point and z minimizes the corresponding value. As a result, if a trained model u ψ ( x q , x ) minimizes the value of u at x q , the sought p -CI can be simply obtained by evaluating u ψ at ( x q , x q ) . More generally, given a query set consisting of K randomly selected query points X q = { x k q } K k =1 , the model is trained such that it solves the problem

The estimated p -CI is, subsequently, given by

In fact, we can train a single model for both upper- and lower-bound predictions because the parameters are shared as in Equation (5). This is achieved by encoding an indicator into the input vector, so that u L ψ ( x q , · ) = u ψ ( x q , · , -1) and u U ψ ( x q , · ) = u ψ ( x q , · , 1) . Furthermore, to balance the weights of the terms in the objective of Equation (5) more effectively, PINNs are separately trained with boundary conditions z L ( x 1: q K ) and z U ( x 1: q K ) . In this regard, u ψ is obtained by

and λ ∈ [0 , 1] is a weight to balance the two loss terms, of which the first aims to enforce close predictions of CIs at query points, and the second to utilize extra information in PDE solutions for better generalization. We refer to this model as EffiPICProp and provide the corresponding implementation in Algorithm 2 in Appendix H. Specifically, the EffiPICProp version with λ = 0 is called SimPICProp, as it simply focuses on close predictions of confidence intervals at query points, while disregarding additional information at other locations.

5 Computational experiments

We provide three computational experiments that are relevant to physics-informed machine learning. We will start with a 1D pedagogical example to illustrate our proposed approaches. Next, we will move to more complicated problems involving standard representative equations met in many real-world problems including heat transfer, fluid dynamics, flows through porous media, and even epidemiology [21, 38]. Details regarding implementation, query points, and computational times can be found in Appendix J.

5.1 A Pedagogical Example

We consider the following equation, which has been adopted from the pedagogical example of [11],

where f ( x ) = -π 2 sin( πx ) -π cos( πx ) sin 2 ( πx ) . For the standard BCs u ( -1) = u (1) = 0 , the exact solution is u ( x ) = sin( πx ) . The purpose of this one-dimensional PDE with only two boundary points, i.e., with a two-dimensional search space in Equation (4), is to demonstrate the applicability of our method in different data scenarios (see Appendix D). Further, the small size of the problem is ideal for comparing computationally expensive methods, such as exhaustive search (ES) and employing PICProp separately at all domain locations, with the more efficient implementation of Algorithm 2.

(a) x² Cl

(c）Hoeffding Cl

0.2

x² CI

T² CI

Hoeffding CI

sample

samples

samples

0.1

1

0.0

x)n

-0.1

-0.2

=0.2

-0.1

0.0

0.1

0.2

-0.2

-0.1

0.0

0.1

0.2

-0.2

-0.1

0.0

0.1

0.2

u(x= -1)

u(x= -1)

u(x= -1)

We consider two types of noisy BCs (Figure 1). First, we consider BCs with Gaussian noise given as

where the standard deviations are σ 1 = σ 2 = 0 . 05 , i.e., the data is assumed to be Gaussian and the standard deviations are known (or estimated if unknown). We sample one or five datapoint(s) for each side of the boundary, for each case, respectively. Given the dataset, a χ 2 CI and a T 2 CI are constructed according to the corresponding problem scenarios discussed in Appendix D; see also Figure 1(a),(b).

Second, we consider BCs with uniform noise given as

where D 1 = D 2 = 1 . 5 are assumed to be known. We sample five datapoints for each boundary side. Given the dataset, a Hoeffding CI is constructed; see also Figure 1(c).

For the query set, six points are used for implementing SimPICProp and EffiPICProp in Section 4.2, whereas 41 points are used for implementing the brute-force version of PICProp, which amounts to solving the bi-level optimization problem of Equation (4) for each point. The implementation details of the PICProp-based methods are summarized in Table J.1. Further, 5,000 BCs are sampled for solving Equation (4) with ES, which can be considered as a reference solution.

Chi-squared

T-squared

Hoeffding

1.0

sin(nπx)

sin(πx)

sin(πx)

EffiPICProp

EffiPICProp

EffiPICProp

PICProp

PICProp

PICProp

0.5

ES

ES

ES

x

sample

sample

sample

0.0

-0.5

-1.0

-1.0

-0.5

0.0

0.5

1.0

-1.0

-0.5

0.0

0.5

1.0

-1.0

-0.5

0.0

0.5

1.0

X

The CI predictions for the three cases are visualized in Figure 2, whereas the meta-learning curves can be found in Appendix K.1. Note that the sizes of the CIs are approximately equal to 0.1 - 0.2 near the boundaries, which are comparable with the propagated CI sizes of Figure 1, and become larger around x of ± 0 . 5 . In all cases, the PICProp results match those of ES, despite the fact that PICProp requires solving a challenging bi-level optimization method with approximate implicit gradient methods. Further, the efficient method EffiPICProp performs accurately as compared to its reference method PICProp, while retaining a significantly lower computational cost (see Figure 3 for χ 2 results and Figure K.4 K.5 for T 2 and Hoeffding results). On the other hand, the predictions of SimPICProp match the brute-force PICProp results only on the query points, but generalize poorly on other locations. In contrast, EffiPICProp, which is based on a meta-model that fits the PDE solutions for multiple query points, generalizes well on unseen locations.

Some additional experimental results are summarized in Appendix K.1 for the completeness of our empirical study: Firstly, a comparison among PICProp and EffiPICProp with various λ s is provided in Figure K.3 to better demonstrate how the balance between two EffiPICProp loss terms in Equation (7) affects the CI prediction. Next, we conduct an additional experiment to empirically verify the validity of EffiPICProp CIs in Appendix K.1.4. Finally, a brief comparison with Bayesian PINNs is presented in Appendix L.

5.2 Two-dimensional Poisson equation

Next, we consider a two-dimensional Poisson equation given as

Chi-squared

Chi-squared (zoomed in)

sin(πx)

1.2

1.0

SimPICProp

EffiPICProp

PICProp

1.0

X

sample

0.5

x

Points for EffiPICProp

0.8

0.0

=

0.6

-0.5

EffiPICProp

sin(πx)

SimPICProp

0.4

PICProp

-1.0

sample

x

Points for EffiPiCProp

-1.0

-0.5

0.0

0.5

1.0

0.2

0.3

0.4

0.5

0.6

The exact solution is u ( x, y ) = e x + e y , and the corresponding f ( x, y ) is obtained by substituting u ( x, y ) into Equation (11). This example serves to demonstrate the effectiveness of our method for treating cases with large numbers of noisy BC datapoints. In such cases, an exhaustive search approach for solving the bi-level optimization of Algorithm 1 generally fails to identify the global optimum, unless we increase the number of trials and the computational cost at prohibitive levels.

We propagate the following fixed CI from the boundary to the rest of the domain ( x, y ) ∈ ( -1 , 1) × ( -1 , 1) :

To solve the problem, 10 boundary points are sampled for each side of the domain, i.e., 40 boundary points in total are used for training and uncertainty is considered at all points. The search space is thus 40-dimensional. The implementation details of the PICProp-based methods are summarized in Table J.1, whereas 4,000 steps of standard PINN training are run for each BC for ES. For a fair comparison, 121 query points for PICProp are selected in the domain, see Figure J.1, and the computational time is approximately the same as of ES with 1,000 trials. Moreover, we include the ES result with 5,000 trials, which is considered to be exhaustive, as a reference.

(a.1) x= -0.5

(a.2)x= -0.5 (zoomed in)

(b.1) x =0.0

(b.2) x= 0.0 (zoomed in)

3.5

2.0

2.4

1.9

3.5

2.3

3.0

2.5

1.8

2.2

1.7

2.1

2.0

1.6 -

> 2.5

2.0

1.5

1.5

2.0

1.9

1.0

1.4

1.5

1.8

1.0

0.5

0.0

0.5

1.0

1.3

0.2

0.1

0.0

0.1

0.2

1.0

0.5

0:0

0.5

1.0

1.7

0.2

-0.1

0.0

0.1

0.2

y

y

y

y

(c.1) x= 0.5

(c.2)x=0.5 (zoomed in)

(d.1) x=1.0

(d.2) x=1.0 (zoomed in)

4.5

3.0

5.5 -

4.1

2.9

4.0

4.0

5.0

2.8

3.9

3.5

2.7

4.5

3.8

=

0

2.6

4.0-

3.7

2.5

2.5

3.5

3.6

2.4

3.5 -

2.0

3.0

2.3

3.4

-1.0

0.5

0.0

0.5

1.0

-0.2

0.1

0.0

0.1

0.2

1.0

0.5

0.0

0.5

1.0

-0.2

0.1

0.0

0.1

0.2

y

y

y

y

Exact

EffiPICProp (121 points)

ES (1000 points)

ES (5000 points)

Figure 4 plots the slices of CI predictions for x ∈ {-0 . 5 , 0 . 0 , 0 . 5 , 1 . 0 } , whereas Appendix K.2 includes slices along the y axis and the heatmaps over the entire domain. For reference, the size

of the CIs obtained with EffiPICProp is approximately 0.1 across the domain, which is equal to the size of the propagated CI of Equation (12). The predicted CI size is also larger in most cases than that of the ES-based CIs. We conclude that 1,000 trials for ES are clearly not sufficient, as the intervals of ES with 5,000 trials are significantly wider; thus, a global optimum is not reached with 1,000 trials. In contrast, EffiPICProp converges to similar or wider intervals compared to ES with 5,000 trials. Overall, the optimization-based method EffiPICProp can identify the solution to the bi-level optimization problem more efficiently and accurately, as compared with the computationally expensive, search-based method ES.

5.3 One-dimensional, time-dependent Burgers equation

In this example, we study the Burgers equation given as

where ν = 0 . 01 π is the viscosity parameter. We consider clean data for the BCs and noisy data for the initial conditions (ICs). Specifically, for the ICs, we propagate the following fixed CI from t = 0 to the rest of the time domain t ∈ (0 , 1] and for all x ∈ [ -1 , 1] :

This problem serves to demonstrate the reliability of our method even in challenging cases with steep gradients or discontinuities of the PDE solution. For solving the problem using PINNs, we consider 200 boundary points, 256 initial points, and 10,000 samples in the interior domain. The 256 initial points are evenly spaced along [ -1 , 1] . The implementation details of the PICProp-based methods are summarized in Table J.1, whereas 2,500 and 5,000 trials are considered for ES. The 246 PICProp query points are visualized in Figure J.1 and the respective computational time is approximately equal to that of ES with 2,500 trials.

Slices of CI predictions for t ∈ { 0 . 0 , 0 . 25 , 0 . 5 , 0 . 75 } are plotted in Figure 5, whereas Appendix K.3 includes the respective meta-learning curves. We conclude that EffiPICProp, although computationally cheaper, in most cases provides wider, i.e., more conservative, CIs, as compared to ES with varying numbers of trials. It also succeeds in capturing the expected increase in uncertainty as the query points move away from the source of uncertainty, which corresponds to larger t values. Specifically, the size of the CIs at t = 0 is approximately 0.4, which is equal to the size of the propagated CI in Equation (14), and becomes approximately 2 at t = 0 . 75 and near x = 0 . On the other hand, since we consider 256 random initial points, the search space of ES is 256-dimensional. It requires a prohibitively large number of trials to identify the global optima of Equation (4). Finally, note that the results of EffiPICProp in Figure 5 correspond to λ = 0 in Equation (7), because we found that the additional information provided by the second loss term in Section 4.2 adversely affects generalization. This is partly due to the large number of query points in this problem.

6 Conclusion

Physics-informed machine learning is transforming the computational science field in an unprecedented manner, by solving ill-posed problems, involving noisy and multi-fidelity data as well as missing functional terms, which could not be tackled before. However, uncertainty quantification is necessary for deploying such methods in critical applications. In this work, we considered a novel problem pertaining to propagating randomness, in the form of CIs, from data locations (at the boundary but can easily be extended to inner or collocation data points) to the entire domain, with probabilistic guarantees. We proposed PICProp, a method based on bi-level optimization, and its efficient version, with a theorem guaranteeing the validity of the obtained CIs (Algorithms 1-2 and Theorem 4.1). To the best of our knowledge, this is the first work on joint CI estimation with theoretical guarantees in the context of physics-informed learning, while our method can be applied in diverse practical scenarios without imposing any distributional assumptions.

In the computational experiments of Section 5, involving various partial differential equations, we demonstrated the effectiveness and efficiency of our proposed methods. Specifically, the predicted CIs are wider, i.e. more conservative, in most cases, as compared with the more computationally demanding exhaustive search for solving the bi-level optimization. Although not encountered in the

(a.1) t=0.0

(a.2)t=0.0 (zoomed in)

(b.1) t= 0.25

(b.2) t=0.25 (zoomed in)

1.0

0.6

1.0

1.0

0.4

0.5

0.5

0.5

0.2

！

0.0

0.0

00

F0°0

-0.2

0.5 -

0.5 -

0.5

0.4 -

1.0

0.6

1.0

1.0

1.0

0.5

0.0

0.5

1.0

0.2

0.1

0.0

0.1

0.2

1.0

0.5

0.0

0.5

1.0

0.2

0.1

0.0

0.1

0.2

X

X

X

X

(c.1) t = 0.5

(c.2) t= 0.5 (zoomed in)

(d.1) t=0.75

(d.2) t=0.75 (zoomed in)

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.5

0.5

0.5

0.5

1.0 -

-1.0

1.0

1.0

1.0

-0.5

0.0

0.5

1.0

0.2

0.1

0.0

0.1

0.2

-1.0

0.5

0.0

0.5

1.0

0.2

0.1

0.0

0.1

0.2

X

X

X

X

Exact

EffiPICProp (246points)

ES (2500 points)

ES (5000 points)

experiments, a potential limitation of our method is that the obtained CIs can be loose in some cases, even when propagating a tight CI from the boundaries, due to the max/min operations in the bi-level optimization of Equation (3). Because of the theoretical challenges involved, we plan to study this limitation and perform pertinent experiments in a future work.

References

Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion , 76:243-297, 2021.

Vidmantas Bentkus. An extension of the hoeffding inequality to unbounded random variables. Lithuanian Mathematical Journal , 48(2):137-157, 2008.

BVSS Bharadwaja, Mohammad Amin Nabian, Bharatkumar Sharma, Sanjay Choudhry, and Alankar Alankar. Physics-informed machine learning and uncertainty quantification for mechanics of heterogeneous materials. Integrating Materials and Manufacturing Innovation , pages 1-21, 2022.

Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng, editors. Handbook of Markov Chain Monte Carlo . Chapman and Hall/CRC, may 2011.

Nicolas Brosse, Alain Durmus, and Eric Moulines. The promises and pitfalls of stochastic gradient langevin dynamics. Advances in Neural Information Processing Systems , 31, 2018.

Min Cai, George Em Karniadakis, and Changpin Li. Fractional seir model and data-driven predictions of covid-19 dynamics of omicron variant. Chaos: An Interdisciplinary Journal of Nonlinear Science , 32(7):071101, 2022.

Kamil Ciosek, Vincent Fortuin, Ryota Tomioka, Katja Hofmann, and Richard Turner. Conservative uncertainty estimation by fitting prior networks. In International Conference on Learning Representations , 2020.

Morris L Eaton and Michael D Perlman. Concentration inequalities for multivariate distributions: I. multivariate normal distributions. Statistics & probability letters , 12(6):487-504, 1991.

Andrew YK Foong, David R Burt, Yingzhen Li, and Richard E Turner. Pathologies of factorised gaussian and mc dropout posteriors in bayesian neural networks. stat , 1050:2, 2019.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050-1059. PMLR, 2016.

Yihang Gao and Michael K Ng. Wasserstein generative adversarial uncertainty quantification in physicsinformed neural networks. arXiv preprint arXiv:2108.13054 , 2021.

Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo. On the iteration complexity of hypergradient computation. In International Conference on Machine Learning , pages 3748-3758. PMLR, 2020.

Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh. Bayesian deep ensembles via the neural tangent kernel. Advances in neural information processing systems , 33:1010-1022, 2020.

Zheyuan Hu, Khemraj Shukla, George Em Karniadakis, and Kenji Kawaguchi. Tackling the curse of dimensionality with physics-informed neural networks. arXiv preprint arXiv:2307.12306 , 2023.

Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International Conference on Machine Learning , pages 4882-4892. PMLR, 2021.

George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics , 3(6):422-440, 2021.

Tze Leung Lai. On confidence sequences. The Annals of Statistics , pages 265-280, 1976.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017.

Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics , pages 1540-1552. PMLR, 2020.

Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193 , 2019.

Xuhui Meng, Hessam Babaee, and George Em Karniadakis. Multi-fidelity bayesian neural networks: Algorithms and applications. Journal of Computational Physics , 438:110361, 2021.

Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Botao Hao, Morteza Ibrahimi, Dieterich Lawson, Xiuyuan Lu, Brendan O'Donoghue, and Benjamin Van Roy. The neural testbed: Evaluating joint predictions. arXiv preprint arXiv:2110.04629 , 2022.

Apostolos F Psaros, Kenji Kawaguchi, and George Em Karniadakis. Meta-learning pinn loss functions. Journal of Computational Physics , 458:111121, 2022.

Apostolos F Psaros, Xuhui Meng, Zongren Zou, Ling Guo, and George Em Karniadakis. Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons. Journal of Computational Physics , page 111902, 2023.

Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics , 378:686-707, 2019.

Jonathan Sadeghi, Marco De Angelis, and Edoardo Patelli. Robust propagation of probability boxes by interval predictor models. Structural Safety , 82:101889, 2020.

John G Saw, Mark CK Yang, and Tse Chin Mo. Chebyshev inequality with estimated mean and variance. The American Statistician , 38(2):130-132, 1984.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research , 15(1):1929-1958, 2014.

Bartolomeo Stellato, Bart PG Van Parys, and Paul J Goulart. Multivariate chebyshev inequality with estimated mean and variance. The American Statistician , 71(2):123-127, 2017.

Chaoqi Wang, Shengyang Sun, and Roger Grosse. Beyond marginal uncertainty: How accurately can bayesian regression models estimate posterior predictive correlations? In International Conference on Artificial Intelligence and Statistics , pages 2476-2484. PMLR, 2021.

Zhuo Wang, Chen Jiang, Pengwei Liu, Wenhua Yang, Ying Zhao, Mark F Horstemeyer, Long-Qing Chen, Zhen Hu, and Lei Chen. Uncertainty quantification and reduction in metal additive manufacturing. Npj Computational Materials , 6(1):175, 2020.

Ian Waudby-Smith and Aaditya Ramdas. Confidence sequences for sampling without replacement. Advances in Neural Information Processing Systems , 33:20204-20214, 2020.

Zheng Wen, Ian Osband, Chao Qin, Xiuyuan Lu, Morteza Ibrahimi, Vikranth Dwaracherla, Mohammad Asghari, and Benjamin Van Roy. From predictions to decisions: The importance of joint predictive distributions. arXiv e-prints , pages arXiv-2107, 2021.

Liu Yang, Xuhui Meng, and George Em Karniadakis. B-pinns: Bayesian physics-informed neural networks for forward and inverse pde problems with noisy data. Journal of Computational Physics , 425:109913, 2021.

Liu Yang, Dongkun Zhang, and George Em Karniadakis. Physics-informed generative adversarial networks for stochastic differential equations. SIAM Journal on Scientific Computing , 42(1):A292-A317, 2020.

Yibo Yang and Paris Perdikaris. Adversarial uncertainty quantification in physics-informed neural networks. Journal of Computational Physics , 394:136-152, 2019.

Dongkun Zhang, Lu Lu, Ling Guo, and George Em Karniadakis. Quantifying total uncertainty in physicsinformed neural networks for solving forward and inverse stochastic problems. Journal of Computational Physics , 397:108850, 2019.

Zongren Zou, Xuhui Meng, Apostolos F Psaros, and George Em Karniadakis. Neuraluq: A comprehensive library for uncertainty quantification in neural differential equations and operators. arXiv preprint arXiv:2208.11866 , 2022.

A Indicative applications

Our proposed method, delineated in Algorithms 1 and 2 of Appendix H, provides a way to quantify how the data uncertainty affects the obtained solution. In addition, if the unseen clean data lies within the constructed/given data bounds, the ground-truth solution lies within our predicted CIs. Some indicative applications include:

In additive manufacturing problems, understanding, through our computed CIs, how the model responds to different material properties arising from the manufacturing process can help in model correction and thus quality control (see [31]).

Further, in material science, cracks affect the elasticity constants which together with geometric properties affect deformations (see [3]). The range of these deformations can be identified with our method. Accurately predicting the upper bound of such deformations can help designers to adjust cross-sectional properties of structural elements to better account for potential cracks.

More generally, our model can be extended to account for PDE model uncertainty. For example, PINNs have been used for short-term predictions of infected cases and deaths due to COVID-19 (see [6]). In such problems, our approach can be used to provide bounds of uncertainty for the predicted variables of interest. Accurately predicting the upper bound of infected cases can help the states better prepare for short-term hospitalization needs.

B Additional details regarding standard UQ

Standard UQ considers that the data is generated from a distribution p ( u | x, θ = θ ∗ ) , where θ ∗ is the NN parameters that are assumed to represent exactly the PDE solution u ( x ) . Given a dataset z , standard UQ approximates the solution u θ ∗ ( x ) with the distribution

where p ( θ | z ) ∝ p ( θ ) p ( z | θ ) according to Bayes' theorem and p ( z | θ ) is the data likelihood. However, as a closed-form p ( θ | z ) is typically intractable, UQ draws approximate samples from p ( θ | z ) , denoted as { ˆ θ j } N θ j =1 . Next, it constructs ˆ p ( u | x ; z ) for each x based on the collected samples to approximate p ( u | x ; z ) . Typically, a Gaussian distribution is fitted to the PINN solutions { u ˆ θ j ( x ) } N θ j =1 and its corresponding statistical intervals are used in lieu of CIs for the PDE solution. Loosely speaking, statistical intervals of { ˆ θ j } N θ j =1 can be viewed as credible intervals from Bayesian statistics, and intervals of { u ˆ θ j ( x ) } N θ j =1 as the corresponding prediction intervals.

Alternatively, one could use a concentration inequality to construct directly a prediction interval using the samples { u ˆ θ j ( x ) } N θ j =1 . Nevertheless, such intervals still correspond to the Bayesian estimate E θ ∼ p ( θ |D ) u θ , rather than to the exact solution u ( x ) , unless further assumptions are made. Even if the assumption that the Bayes estimate corresponds to θ ∗ is made, i.e., θ ∗ = E θ |D θ , the typically considered E θ |D u θ ( x ) is not guaranteed to equal u E θ |D θ ( x ) as u θ can be arbitrary nonlinear (see also Appendix C).

Standard UQ approaches have several other limitations. For example, strong assumptions regarding the noise distribution and data likelihood are often required, and depending on the problem it is notoriously difficult to tune Bayesian NNs [5]. Furthermore, it is not possible to obtain/sample p ( u | x, z ) exactly in practical problems and thus, additional approximations are required, typically referred to as approximate posterior inference . In addition, UQ approaches for PINNs are not supported by theoretical guarantees regarding the quality of uncertainty propagation due to the physics-informed constraints [11]. Finally, for the case of multiple available datasets, see Appendices C and E for the limitations of Monte Carlo-based approaches.

C Additional details regarding Monte Carlo Simulation

̸

Assume that we have m i.i.d. datasets denoted as { z ( j ) } j m =1 and a 0 -proper data-driven PDE solver (see Definition 3.3). It is natural to consider solving the PDE with each dataset in a Monte Carlo manner and then using a concentration inequality to construct CIs. However, a CI for the exact solution u ( x ) cannot be identified without further assumptions. The reason is that we only have access to E [ u ( x ; Z )] , rather than to u ( x ) given as u ( x ) = u ( x ; ¯ Z ) = u ( x ; E [ Z ]) , considering data-driven solutions from a 0 -proper solver. Clearly, the difference between the two is in the form of E [ f ( X )] = f ( E [ X ]) , where f in our case is the solver. In Figure C.1 we provide an illustration of this point as applied to linear/convex/concave functions, as s straightforward consequence of Jensen's inequality.

0.0

0.2

0.4

0.6

0.8

1.0

x

0.0

0.2

0.4

0.6

0.8

1.0

f(x)

f

(

E

[

X

]) =

E

[

f

(

X

)]

(a)

f

(

x

) =

x

f

(

x

) =

x

0.0

0.2

0.4

0.6

0.8

1.0

x

0.0

0.2

0.4

0.6

0.8

1.0

f(x)

f

(

E

[

X

])

>

E

[

f

(

X

)]

(b)

f

(

x

) = (

e x

1)/(

e

1)

f

(

x

) = (

e

x

-

1)

/

(

e

-

1)

0.0

0.2

0.4

0.6

0.8

1.0

x

0.0

0.2

0.4

0.6

0.8

1.0

f(x)

f

(

E

[

X

])

>

E

[

f

(

X

)]

(c)

f

(

x

) = sin(

x

/2)

f

(

x

) = sin(

πx/

2)

D Confidence Interval Construction with Data

Consider a dataset { z ( j ) } j m =1 , which contains m independent samples of an n -dimensional random vector Z . Based on the data { z ( j ) } j m =1 , we construct a CI for Z , denoted as ˜ Z p , where p is the confidence level. Equivalently, Pr (¯ z ∈ ˜ Z p ) ≥ p .

In this regard, we consider the following problem scenarios:

The random vector Z follows a multivariate Gaussian distribution with known covariance matrix Σ of rank n . Given one observation z , i.e., m = 1 , the chi-squared statistic χ 2 = ( z -µ ) T Σ -1 ( z -µ ) follows a chi-squared distribution with n degrees of freedom. The corresponding p -CI is ˜ Z p = { z ′ | || z -z ′ || Σ -1 ≤ k χ 2 n,p } , where || z -z ′ || Σ -1 = √ ( z -z ′ ) T Σ -1 ( z -z ′ ) and k χ 2 n,p is the squared root of the upper (1 -p ) / 2 critical value of the chi-squared distribution with n degrees of freedom; see Figure 1 for an example corresponding to the case of n = 2 .

The random vector Z follows a multivariate Gaussian distribution with unknown covariance matrix Σ and m observations are available. The sample mean and covariance are estimated as ¯ z = 1 m ∑ m i =1 z i and ˆ Σ = 1 m -1 ∑ m i =1 ( z i -¯ z )( z i -¯ z ) T , respectively. The Hotelling's t -squared statistic t 2 = N (¯ z -µ ) T ˆ Σ -1 (¯ z -µ ) follows a Hotelling's T -squared distribution with parameters n , m -1 . The corresponding p -CI is ˜ Z p = { z ′ | || ¯ z -z ′ || ˆ Σ -1 ≤ m -1 / 2 k T 2 n,m -1 ,p } , where || ¯ z -z ′ || ˆ Σ -1 = √ (¯ z -z ′ ) T ˆ Σ -1 (¯ z -z ′ ) and k T 2 n,m -1 ,p denotes the squared root of the upper (1 -p ) / 2 critical value of the aforementioned Hotelling's T -squared distribution; see Figure 1 for an example corresponding to the case of n = 2 and m = 5 .

The components of the random vector Z follow an unknown distribution, and m independent observations are available. For this case, a concentration inequality can be used for each component, i.e., for each data location, and the obtained bounds can be combined using a union of bounds for obtaining a p -CI (see also Appendix E). Note that many popular concentration inequalities require specific assumptions for the unknown distrivarious true population statistics (e.g., the population mean and variance) for computing the bounds,

bution besides the fact that independent samples are available. For example, Hoeffding's inequality and its variants require the random variable to be bounded. [2] extends Hoeffding's inequality to unbounded random variables, but it is limited to Pareto-type tails. Other concentration inequalities without heavy assumptions for the distribution, such as Chebyshev's inequality and Chernoff's inequality, require which are not available in the cases we consider. In this regard, [27] provides a Chebyshev-type bound with estimated mean and variance. [29] further extends the bound of [27] to the multivariate case.

The CI ˜ Z p can alternatively be given by domain experts.

E Joint Confidence Intervals for PDE Solutions

In this section, we provide the necessary definitions, lemmas, and remarks that are required for introducing and addressing the CI propagation problem in Section 4.1. That is, how to propagate, with probabilistic guarantees, a CI constructed at the locations of the available data to arbitrary sets of query points of the domain.

To this aim, we first define the marginal CI for a single query point x q .

Definition E.1 (Marginal CI) . A pair of random variables ( L, U ) ∈ R 2 is a p -CI for u ( x q ) on the single query point x q ∈ Ω , if the random event E := { L < u ( x q ) < U } holds with probability at least p .

Intuitively, considering repeating experiments of drawing random datasets { z ( j ) } 1: m , the proportion of the ( L, U ) CIs, constructed according to Definition E.1, that contain u ( x q ) is at least p . Next, we define the joint CI for multiple query points.

Definition E.2 (Joint CI) . A pair of functions ( L, U ) ∈ R X q × R X q is a p -CI for u on query set X q ⊆ Ω , if the random event E := ∩ x q ∈X q { L ( x q ) < u ( x q ) < U ( x q ) } holds with probability at least p .

For simplicity, in the rest of the paper we refer to the p -CI for u on the whole domain Ω as p -CI for u , and to the p -CI for u on X q ⊂ Ω as query set p -CI .

Remark E.3 . Let ( L, U ) be a p -CI for u . For any X q ⊆ Ω , ( L, U ) is a p -CI for u on X q , i.e., the random event E ( X q , Z ) := ∩ x q ∈X q { L ( x q ) < u ( x q ) < U ( x q ) } holds with probability at least p .

Remark E.3 indicates that a p -CI for u on an arbitrary query set can be derived from a p -CI for u by slicing. The query set X q can be either discrete or continuous including multiple or even infinite queries. Note that the p -CIs of Definition E.2 and Remark E.3 hold with probability at least p ; i.e., there is no statement about their tightness. In this regard, the next remark shows that slicing using larger sets X q yields tighter p -CIs for u on X q . Nevertheless, the maximum tightness that can be achieved, for the p -CI for u on X q , is determined by the tightness of the p -CI for u on the whole domain X q . This is corroborated by the following remark.

Remark E.4 . Let ( L, U ) ∈ R Ω × R Ω be a p -CI for u . For query set X 1 q ⊆ X 2 q ⊆ Ω , the probabilities of random events E 1 := ∩ x q ∈X 1 q { L ( x q ) < u ( x q ) < U ( x q ) } , E 2 := ∩ x q ∈X 2 q { L ( x q ) < u ( x q ) < U ( x q ) } satisfy

Next, we define the union of single-query p -CIs. It is often used in practice as a proxy to the query set p -CI and can be used also within the context of the present study for addressing problem scenarios 3 and 4, as described in Appendix D.

Definition E.5 (Union of CIs) . A pair of functions ( L, U ) ∈ R X q × R X q constructed based on { z ( j ) } 1: m is a union of p -CIs for u on query set X q ⊆ Ω if for each x q ∈ X q , ( L ( x q ) , U ( x q )) is a p -CI for u ( x q ) , i.e., each random event E := { L ( x q ) < u ( x q ) < U ( x q ) } holds with at least probability p .

The next lemma provides insight into the connection between the marginal p -CI on X q and the union of p -CIs.

Lemma E.6. For u and X q , a p -CI is a union of p -CIs, whereas a union of p -CIs is not necessarily a p -CI.

That is, u ( x q ) , ∀ x q ∈ X q , belongs to the p -CI constructed by slicing of the p -CI for u with probability at least p . On the other hand, the probability of the event ∩ x q ∈X q { L ( x q ) < u ( x q ) < U ( x q ) } , where ( L, U ) are constructed based on a union of p -CIs, may be less than p . This is supported by the following two remarks as well as the computational experiment of Appendix F.

Remark E.7 . The probability of the random event ∩ x q ∈X q { L ( x q ) < u ( x q ) < U ( x q ) } , where ( L, U ) are constructed based on a union of p -CIs, is upper-bounded by the minimum probability of the events { L ( x q ) < u ( x q ) < U ( x q ) } , ∀ x q ∈ X q . That is,

Remark E.8 . If one of the p -CIs is tight, i.e. ∃ x ′ q ∈ X q such that Pr ( { L ( x ′ q ) < u ( x ′ q ) < U ( x ′ q ) } ) = p , the union of these p -CIs is a p -CI for u if and only if it holds ∀ x q ∈ X q \ { x ′ q } that

Finally, in order to obtain a guarantee that a union of CIs is a p -CI, we can equate a lower bound of the probability of the former with p . This lower bound can be obtained by disregarding correlations among the different locations, as formalized in the following lemma.

Lemma E.9. For u and finite query set X q , a union of (1 -(1 -p ) / |X q | ) -CIs is a p -CI for u as

However, as the size of the query set X q increases, the required confidence level of each interval must approach 1, which leads to loose union bounds. Finally, another approach for obtaining a p -CI for multiple queries is to use multivariate concentration inequalities [8]. Nevertheless, this approach is limited to a finite number of queries.

F Toy Linear System

In this section, we highlight the differences between joint CIs and union of CIs, as described in Appendix E, with a simple example. Consider a true function u and a model given as u θ ( x ) = (1 -x ) θ 0 + xθ 1 , x ∈ [0 , 1] , with θ = ( θ 0 , θ 1 ) ∈ R 2 . We assume that there exists θ ∗ such that u θ ∗ = u , and that we are given measurements Z = ( Z 0 , Z 1 ) ∼ N ( θ ∗ , I ) . Thus, for one observation Z , we simply estimate ˆ θ = Z . As a result, the estimator for u is given as

Next, define as k -interval the lower and upper intervals ( L k , U k ) obtained by subtracting and adding k standard deviations from the estimated mean of u , respectively, i.e.,

Remark F.1 . We define k N p -interval as the interval constructed as a union of one-query p -CIs for u θ ( x ) and for different values of x ∈ Ω , similarly to Definition E.5. The value k N p is the upper (1 -p ) / 2 critical value of the standard normal distribution.

Proof : For x ∈ Ω ,

Remark F.2 . We define k χ 2 p -interval as the p -CI for u , constructed as ( L k χ 2 p ( x ) , U k χ 2 p ( x )) for each x ∈ Ω , where

The value k χ 2 p is the squared root of the upper (1 -p ) / 2 critical value of the chi-squared distribution with 2 degrees of freedom.

Proof : According to Theorem 4.1,

The left side of Figure F.1 provides an indicative example of a k N p -interval and a k χ 2 p -interval, with p = 95% and θ ∗ = (0 , 0) . The results are obtained using only one sample Z = ( Z 0 , Z 1 ) , which can be visualized by considering that u θ (0) = Z 0 and u θ (1) = Z 1 . Further, the empirical confidence level of an interval ( L, U ) can be estimated as ¯ p := 1 J ∑ J j =1 1 ( ∩ k =0 , 1 ,..., ⌊ 1 /ϵ ⌋ { L Z j ( kϵ ) < u ( kϵ ) < U Z j ( kϵ ) } ) with ϵ > 0 . That is, we calculate the proportion of times that such a construction of the interval contains the true values on { 0 , ϵ, ..., ⌊ 1 /ϵ ⌋ ϵ } . According to the law of large numbers, the empirical confidence level converges with increasing J , in probability, to the true confidence level p . The right side of Figure F.1 shows that the empirical confidence level curve of the k χ 2 p -interval (or the k N p -interval) converges to a value above (or below) 95% . This result implies that the k χ 2 p -interval is a valid 95% -CI for u while the k N p -interval is not.

y

0.0

0.2

0.4

0.6

0.8

1.0

x

2

1

0

1

2

3

k p

-interval

k

2

p

-interval

Ground Truth

Sample

0

2000

4000

6000

8000

10000

# of samples

0.800

0.825

0.850

0.875

0.900

0.925

0.950

0.975

1.000

p

k p

-interval

k

2

p

-interval

95%

G Proofs

G.1 Proof of Theorem 4.1

If ¯ z ∈ ˜ Z p , then L ( x ) = min z ∈ ˜ Z p u ( x ; ¯ z ) ≤ u A ( x ; ¯ z ) ≤ max z ∈ ˜ Z p u ( x ; ¯ z ) = U ( x ) , for all x ∈ Ω . Further, according to the definition of the proper solver, we have u A ( x ; ¯ z ) -η ≤ u ( x ) ≤ u A ( x ; ¯ z ) + η , for all x ∈ Ω , and thus L ( x ) -η ≤ u ( x ) ≤ U ( x ) + η , for all x ∈ Ω . Finally, we can conclude that Pr { L ( x ) -η ≤ u ( x ) ≤ U ( x ) + η, ∀ x ∈ Ω } ≥ Pr { ¯ z ∈ ˜ Z p } ≥ p .

H Algorithms

The developed methods are summarized in Algorithms 1 and 2.

Algorithm 1 PICProp: Physics-Informed Confidence Propagation (Gradient-based)

Input: Z p ( p -CI for ˜ z ), query point x , inner-steps schedule { N k } o k =1 , learning rates α, β , outer steps K o .

I Bi-level Optimization

A bi-level optimization problem can be formulated as

where θ , ϕ are the parameters and the hyperparameters, whereas L and J are the lower-level objective and the upper-level objective. Many problems like meta-learning, hyperparameter optimization, data poisoning, and auxiliary learning can be naturally formulated in a bi-level optimization way.

The hardest part of bi-level optimization is that θ ∗ ( ϕ ) is not available in a closed-form in many scenarios, particularly with an empirical lower-level objective that requires multiple steps of gradient descent. In such cases, the explicit gradient of high-level objective with respect to hyperparameter is intractable.

According to the chain rule,

The most challenging part is the approximation of the implicit gradient ∇ ϕ θ ∗ ( ϕ ) , for which many advanced methods have been proposed and studied in recent years [15]. In this paper, three methods referred to as Reverse , approximate implicit differentiation with Neumann Series (AID-NS), and approximate implicit differentiation with conjugate gradient (AID-CG) are evaluated on the confidence interval identification tasks.

I.1 Reverse

Reverse is an iterative differentiation-based method to approximate the hypergradient by unrolling the inner optimization θ 1: K trajectory in a reverse way. The implicit gradient can be approximated by

where ˜ θ k +1 = θ k -α ∇ θ l ( θ k , ϕ ) and α is a pre-defined inner-learning rate.

I.2 Approximate Implicit Differentiation (AID)

Another family of methods to approximate the hypergradient is Implicit Differentiation, where the implicit function theorem (IFT) is used to evaluate ∇ ϕ θ ∗ ( ϕ ) . If L is second-order derivable and strong convex, we have

Then we can approximate the gradients of ϕ using

The most challenging part is the approximation of the inverse Hessian vector product (inv-hvp) A -1 p , where A ∈ R d × d is a given symmetric positive matrix and p ∈ R is a given vector.

AID-NS [19] is a method based on the Neumann Series to approximate the inv-hvp as

where λ is a hyperparameter that ensures convergence and K is a hyperparameter for precision control.

AID-CG [12] approximates the hypergradient by solving the system of homogeneous linear equations Ax = p with conjugate gradient.

J Details regarding implementation, hyperparameters, query points, and computational times

J.1 Implementation and hyperparameters

For each PDE, we use a multilayer perceptron (MLP) for the PINN model and another MLP of the same architecture for EffiPICProp. Query points are collected in a grid manner. For each query point, PICProp starts with standard PINN training with a randomly initialized virtual boundary condition. We refer to this as the warmup phase. Subsequently, PICProp alternates between performing a number of inner steps to update the PINN model and computing the hyper-gradient for a meta step to update the virtual boundary condition. The inner/outer learning rate is selected from { 0 . 001 , 0 . 01 , 0 . 1 } . The inner/outer optimizer is selected from SGD and Adam . The hyper-gradient method is selected from Neumann Series (NS) , Conjugate Gradient (CG) , Reverse ; see more details in Appendix I. The numbers of warmup/inner/meta steps are tuned to balance the stability of bi-level training and the computational cost. For the pedagogical example, as six points are too few for train-validation splitting, we manually set λ = 1 for EffiPICProp. In the rest of the examples, we split 10% of the training data as the validation set to select the best λ from { 0 . 0 , 0 . 25 , 0 . 5 , 0 . 75 , 1 . 0 } . The hyperparameters are summarized in Table J.1.

J.2 Query Points

The query points are summarized in Figure J.1.

(a）Poisson 2D:Query Points

(b) Burgers: Query Points

1.0

1.0

×

×

×

5

xxXXXXXXXXXXXXXXX

KXXX

xXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXX>

0.75

×

×

×

XXXXXXXXXX>

0.5

0.5

0.50

×

×

×

×

×

4

0.25

×

×

×

×

×

×

X

交

y

0.0

×

×

×

×

×

×

3

0.0

XXXXXXXXXXXXXXX

0.00

XXXXXX

XXXXXXXXXXXXX

+

×

×

×

×

×

-0.25

×

×

×

×

×

×

×

xXXXXXXXXX>

xXXXXXXXXX

-0.5

2

-0.5

-0.50

×

×

×

×

×

×

×

×

-0.75

×

×

×

×

×

1

-1.0 +

-1.0 +

-1.0

-0.5

0.0

0.5

1.0

0.00

0.25

0.50

0.75

1.00

t

J.3 Computational times

In our computational experiments, all methods were run serially and thus, the runtimes shown below are much larger than the required ones for using these methods in practice. For this reason, we have also included a 'pseudo-parallel' runtime that shows the runtimes if parallel processing had been used instead. The runtimes for the examples of Section 5 are summarized in Table J.2.

K Additional Experimental Results

K.1 Pedagogical example

K.1.1 Learning Curves

Figure K.1 provides the meta-learning curves corresponding to the CI predictions of Figure 2 (main text). Figure K.2 provides the results of all considered meta-learning methods and of ES on six query points. We conclude that the combination of Neumann Series (NS) and SGD stably converges to the reference result of ES.

K.1.2 λ s

Figure K.3 provides comparison among PICProp and EffiPICProp with various λ s. Notice that EffiPICProp with λ = 0 is referred to as SimPICProp in Figure 3. The mean squared errors (MSE) between PICProp CI and EffiPICProp CI are summarized in Appendix K.1.4. We conclude that EffiPICProp CI with λ = 1 . 0 is the best approximation of PICProp CI.

K.1.3 SimPICProp v.s. EffiPICProp

Figure K.4 and Figure K.5 provide the comparison between SimPICProp and EffiPICProp on T 2 and Hoeffding boundary CI, respectively.

K.1.4 Validity of EffiPICProp CIs

Although it has been already presented in Figure 2 that PICProp and EffiPICProp predictions well match the ES results, we conduct the following experiment to further verify the validity of EffiPICProp CI predictions.

We independently sampled 500 boundary conditions and propagate the corresponding χ 2 CIs using EffiPICProp to construct CIs for the entire domain. According to the definition of joint CIs (see Definition 3.1), an instance of CI is valid if it contains the solution u ( x ) for any x ∈ [ -1 , 1] . In this regard, we uniformly sampled 101 data points from [ -1 , 1] and regard an instance of CI as valid if the solution was enclosed within the interval for these data points. Further, we run vanilla PINNs with clean boundary data and use the max absolute error among these 101 data points as η . As a result, 486 out of these 500 ( 97 . 2% > 95% ) instances of CI are valid, which empirically indicates that EffiPICProp CI predicitons are valid and conservative.

0

10

20

30

40

50

meta steps

0.10

0.05

0.00

0.05

0.10

CI prediction

x

=

1.0

0

10

20

30

40

50

meta steps

1.05

1.00

0.95

0.90

0.85

CI prediction

x

=

0.6

0

10

20

30

40

50

meta steps

0.70

0.65

0.60

0.55

0.50

CI prediction

x

=

0.2

0

10

20

30

40

50

meta steps

0.500

0.525

0.550

0.575

0.600

0.625

0.650

0.675

CI prediction

x

=0.2

0

10

20

30

40

50

meta steps

0.85

0.90

0.95

1.00

1.05

CI prediction

x

=0.6

0

10

20

30

40

50

meta steps

0.10

0.05

0.00

0.05

0.10

CI prediction

x

=1.0

PICProp (NS + SGD)

Exaustive Search

0

10

20

30

40

50

meta steps

0.10

0.05

0.00

0.05

0.10

CI prediction

x

=

1.0

0

10

20

30

40

50

meta steps

1.05

1.00

0.95

0.90

0.85

x

=

0.6

0

10

20

30

40

50

meta steps

0.70

0.65

0.60

0.55

0.50

x

=

0.2

0

10

20

30

40

50

meta steps

0.500

0.525

0.550

0.575

0.600

0.625

0.650

0.675

CI prediction

x

=0.2

0

10

20

30

40

50

meta steps

0.85

0.90

0.95

1.00

1.05

x

=0.6

0

10

20

30

40

50

meta steps

0.10

0.05

0.00

0.05

0.10

x

=1.0

PICProp (CG + ADAM)

PICProp (CG + SGD)

PICProp (NS + ADAM)

PICProp (NS + SGD)

PICProp (REVERSE + ADAM)

PICProp (REVERSE + SGD)

Exaustive Search

K.2 2-D Poisson equation

Figure K.6 includes slices along the y axis and accompanies Figure 4 (main text), which includes slices along the x axis. Heatmaps with CIs over the entire domain are presented in Figure K.7.

1.0

0.5

0.0

0.5

1.0

x

1.0

0.5

0.0

0.5

1.0

u

Chi-squared

x

)

PICProp

EffiPICrop,

=0.0

EffiPICrop,

=0.25

EffiPICrop,

=0.5

EffiPICrop,

=0.75

EffiPICrop,

=1.0

sample

Points for EffiPICProp

0.2

0.3

0.4

0.5

0.6

x

0.4

0.6

0.8

1.0

1.2

u

Chi-squared (zoomed in)

x

)

PICProp

EffiPICrop,

=0.0

EffiPICrop,

=0.25

EffiPICrop,

=0.5

EffiPICrop,

=0.75

EffiPICrop,

=1.0

sample

Points for EffiPICProp

.

1.0

0.5

0.0

0.5

1.0

x

1.0

0.5

0.0

0.5

1.0

u

T-squared

x

)

EffiPICProp

SimPICProp

PICProp

sample

Points for EffiPICProp

0.2

0.3

0.4

0.5

0.6

x

0.4

0.6

0.8

1.0

1.2

u

T-squared (zoomed in)

x

)

EffiPICProp

SimPICProp

PICProp

sample

Points for EffiPICProp

K.3 Burgers equation

Figures K.8 & K.9 provide the meta-learning curves corresponding to the CI predictions of Figure 5 (main text).

L Comparison with Bayesian PINNs

In this section, we compare PICProp with Bayesian PINNs (BPINNs) [34] for the problem of Section 5.1. For BPINNs, the inference method for posterior estimation is Hamiltonian Monte Carlo (HMC) [4]. Although provided here for completeness, we believe that such a comparison can also be misleading because of the many differences between the two methods. The first difference pertains to the fact that PICProp does not consider parameter uncertainty. Further, the assumptions for these two methods are different: BPINN requires heavy assumptions about the data likelihood, for instance, while PICProp only requires the noise to be mean-zero. Furthermore, PICProp produces joint CIs for the whole domain, while the BPINN intervals are marginal. Finally, although supported by a sound Bayesian justification, BPINNs are not supported by a strict confidence guarantee for the produced intervals, in contrast to Theorem 4.1 that supports PICProp.

We summarize and contrast the two approaches in the following: BPINNs assume a NN parameter prior, and subsequently approximate and sample the parameter posterior to obtain samples from the solution posterior. This is done in order to fit a solution posterior distribution and subsequently compute its statistical intervals. In contrast, the problem of UQ is rethought ab initio in our paper. PICProp first considers either CIs given at the boundaries or data collected at the boundaries and estimates the corresponding CIs. Subsequently, the boundary CIs are propagated to the interior domain with mild assumptions and with the theoretical guarantee that the obtained CIs are indeed CIs according to the definition.

From the results summarized in L.1, we observe from Figure L.1 that the PICProp CIs are more conservative than the BPINN. From a design perspective, conservative uncertainty estimation is the preferred route for critical and risk-sensitive applications [7, 13], although it can be more costly. In contrast, most widely used UQ methods, such as deep ensembles [18] and Monte-Carlo dropout [28, 10], are known to be overconfident in practice [9, 7]. Finally, note that the BPINN results of Figure L.1 were obtained by using the software NeuralUQ developed in [38].

1.0

0.5

0.0

0.5

1.0

x

1.0

0.5

0.0

0.5

1.0

u

Hoeffding

x

)

EffiPICProp

SimPICProp

PICProp

sample

Points for EffiPICProp

0.2

0.3

0.4

0.5

0.6

x

0.4

0.6

0.8

1.0

1.2

u

Hoeffding (zoomed in)

x

)

EffiPICProp

SimPICProp

PICProp

sample

Points for EffiPICProp

(a.1) y = - 0.5

(a.2) y= -0.5 (zoomed in)

(b.1) y = 0.0

(b.2) y =0.0 (zoomed in)

3.5

2.0

2.4

3.0

1.9

3.5

2.3

2.5

1.8

3.0

2.2

1.7 -

2.1

2.0

1.6

> 2.5

2.0

1.5

1.5

2.0

1.9

1.0

1.4

1.5

1.8

1.0

0.5

0.0

0.5

1.0

1.3

0.2

0.1

0.0

0.1

0.2

1.0

0.5

0.0

0.5

1.0

1.7

0.2

0.1

0.0

0.1

0.2

X

(c.1) y = 0.5

(c.2) y = 0.5 (zoomed in)

(d.1) y = 1.0

(d.2) y = 1.0 (zoomed in)

4.5

3.0

5.5

4.1

2.9

4.0 -

4.0

5.0

2.8 -

3.9

3.5

2.7

4.5

3.8

3.0

2.6

4.0

3.7

2.5

ERETETE

3.6

2.5

3.5

2.4

3.5

2.0

1.0

0.5

0.0

0.5

1.0

2.3

0.2

0.1

0.0

0.1

0.2

3.0

1.0

0.5

0.0

0.5

1.0

3.4

0.2

0.1

0.0

0.1

0.2

X

X

X

Exact

EffiPICProp (121 points)

ES (1000 points)

ES (5000 points)

y

1.0

0.5

0.0

0.5

1.0

x

1.0

0.5

0.0

0.5

1.0

Upper bound: ES (500 points)

1.0

0.5

0.0

0.5

1.0

x

Upper bound: EffiPICProp (50 points)

1.0

0.5

0.0

0.5

1.0

x

Lower bound: ES (500 points)

1.0

0.5

0.0

0.5

1.0

x

Lower bound: EffiPICProp (50 points)

1

2

3

4

5

(a)

t

= 0

.

0

(b)

t

= 0

.

25

(c)

t

= 0

.

5

(d)

t

= 0

.

75

0.4

1.00

0.75

1.0

0.75

0.2

0.50

0.5

0.50

0.0

0.25

0.25

0.00

0.0

0.00

0.2

0.25

0.25

0.4

0.6

PICProp - 100 inner steps

PICProp - 40 inner steps

ES

0.50

5'0

0.5

-1.0

0.75

0.50

1.00

0

100

200

meta step

300

400

500

0

100

200

meta step

300

400

005

0

100

200

meta step

000

400

005

0

100

200

meta step

300

400

500

(a)

t

= 0

.

0

(b)

t

= 0

.

25

(c)

t

= 0

.

5

(d)

t

= 0

.

75

0.75

0.4

0.7

=0.5

8°0

0.80

0.65

0.6

0.9

0.85

0.70

口

0.7

1.0

0.90

0.75

0.8

ES

-1.1

=0.95

0.9

PICProp - 40 inner steps

PICProp - 100 inner steps

1.2

1.00

0.80

100

200

meta step

300

400

500

100

200

meta step

000

400

500

0

100

200

meta step

300

400

005

O

100

200

meta step

0OE

400

500

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

x

1.0

0.5

0.0

0.5

1.0

u

Chi-squared

x

)

EffiPICProp

BPINNs+HMC

sample

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents methods to estimate confidence sequences for the solution of a differential equation given noisy boundary conditions. Various methods are proposed of different complexity and the confidence sequences are illustrated for the solution of three different problems.\n\nSTRENGTHS:\nThe overall problem to determine the confidence sequence is laid out.  Under the given assumptions, the problem is well formed and an approximation of the exact solution via exhaustive search is clear.  The faster methods that incorporate meta-learning techniques and surrogate solutions for the lower and upper sequences are innovative and clever.\n\nWEAKNESSES:\nThe main assumption of the paper is that the accuracy of the solution provided by the machine learning model is either perfect (\\eta = 0) or \\eta is known and fixed when the boundary conditions are given perfectly.  Even with clean boundary data, the accuracy of the model will depend of the number of internal and boundary training points, and these are not incorporated in the computation of the confidence sequences.\n\nWhile it might be OK to consider the exhaustive search as the gold standard, the examples do not confirm whether or not the derived confidence sequences are indeed satisfying Definition 3.1. In light of the first comment, this should be validated. \n\nAlgorithm 1 and 2 are part of the main contribution of the paper.  However, they are tucked away in the supplementary appendix. As such, a reader if forced to go to the supplementary material to even understand the gist of the paper.\n\nQUESTIONS:\nIn eqn. (5) it seems that the lower and upper boundary data points should be derived by finding the points that  minimized u_{\\hat{\\theta}}(x) at x = x_q^k rather than for all x.  Is that correct?\n\nFour lines above eqn. (4), should z minimize (rather than maximize) the corresponding value to determine L(x)?\n\nThird to last last line of section 4, SimPICProp is not defined in the main paper.  It may be better to state that SIMPICProp is the version of EffPICProp where \\lambda = 0. \n\nThe paper use the term confidence interval for Def. 3.1.  Based on the work of Ramdas, I believe this is a confidence sequence. The authors should verify if this is indeed the case.\n\nLIMITATIONS:\nNo negative societal impact as this is foundational work that is not considering a specific application or using an datasets.  The actual differential equations being solved are toy problems.\n\nLimitations of the method were addressed by the review in terms of capturing the accuracy of the neural solution.  This limitation was not discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on a novel algorithm for confidence interval estimation for deterministic partial differential equations. It provides leverages bi-level optimization to compute a valid CI, that is supported by both theoretical and numerical investigations, focusing on physics-informed learning.\n\nSTRENGTHS:\nThe paper is very well written, with clearly structured sections and appendices. It provides a lot of technical detail and manages to effectively balance what to be included in the main text and what to be pushed to the appendices in order to not distract. I particularly enjoyed reading this work as in blends theoretical aspects with computational ones, and has direct and immediately applicability to the computational science world, particularly error estimations.\n\nThe authors have performed a number of benchmarks (of increasing complexity) for the methodology, and have clearly showed its advantages as well as observed (\"We conclude that EffiPICProp, although computationally cheaper, in most cases provides wider [...] CIs, as compared to ES with varying numbers of trials\") or potential drawbacks (\"Although not encountered in the experiments, a potential limitation of our method is that the obtained CIs can be loose in some cases, even when propagating a tight CI from the boundaries, due to the max/min operations in the bi-level optimization of Equation 3\").\n\nOverall, I believed the presentation of the work is of high quality and has impact implications to its directed target.\n\nWEAKNESSES:\nA weakness of the paper is the fact that some benchmark results are incomplete. Notably SimPICProp in the pedagogical example is only shown for Chi-squared, not for T-squared or Hoeffding. Also SimPICProp and PICProp are not shown for the 2D Poisson equation or the 1D time-dependent Burgers equation.\n\nLastly, while as the authors say \"The aim of this paper is to propose a principled and theoretically guaranteed solution approach and thus, we refrain from applying such ad hoc solutions in the computational experiments and plan to consider parameter uncertainty rigorously in a future work. Nevertheless, we provide in Appendix L a comparison with Bayesian PINNs for completeness.\", the figure in Appendix L suggest a potential better performance of the Bayesian PINNs compared to EffiPICProp. For objectivity reasons, I would suggest moving some of that discussion into the main text.\n\nQUESTIONS:\n1. Please complete table J.2 and the show the other comparisons in the figures 3,4,5 of SimPICProp and PICProp to ES and EffiPICProp.\n2. In appendix C, you should mentioned the well-known result of Jensen's inequality as the figures are an illustration of this.\n3. Please consider discussing the Bayesian PINNs in the main text.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors present a method to propagate confidence intervals from some locations to the entire domain. In specific, they consider the problem of propagating uncertainty / confidence internals at the boundary, in partial differential equations based systems, to the rest of the domain. The proposed method PICProp is a bi-level optimization method. Authors present theoretical guarantees for PICProp and then develop a computationally efficient version of the PICProp under some simplifications. The authors show the efficacy of the method via some computational experiments.\n\nSTRENGTHS:\nThe paper tackles an important problem of estimating and propagating uncertainty in physics-informed ML systems. The proposed methods are the first methods (to the best of my knowledge) to develop confidence interval propagation method for PDE systems.\n\nThe authors present a method, Physics-Informed Confidence Propagation (PICProp), to propagate confidence intervals from any boundary points in the domain to any other points in the domain. They show theoretical guarantees which demonstrate that PICProp propagates confidence intervals with high probability.  The theoretical analysis offers insights into the inner workings of PICProp.\n\nThen authors identify that PICProp is computationally expensive and might not be applicable beyond small domains. To address this they develop an efficient version of PICProp known as EffiPICProp which makes confidence interval propagation much more computationally viable. It is great that authors through about the practicality of the algorithm and developed this. \n\nAuthors test their method on computational experiments. In spite of being small scale experiments, they offer intuition which is important for transferring these methods to other domains / problems. It is great that the authors have actually documented the computational time for running different methods on these problems.\n\nWEAKNESSES:\nAll the computational experiments seems to be conducted on 2D domains. It is not clear how the computational costs scale once we go to higher dimensions and if EffiPICProp can still be computationally feasible. It might be useful to discuss about this.\n\nThis work deals with deterministic systems and often most real-life applications involve stochastic systems (for example, due to inaccurate modeling of the systems). It might be useful to discuss about how the proposed methods can or cannot transfer to stochastic systems.\n\nAs of now, the proposed method only propagates uncertainty from boundary points.\n\nQUESTIONS:\nIt would be great if authors can comment about points mentioned in \"Weakness\" section\n\nLIMITATIONS:\nPlease refer to \"Weakness\" section. \n\nEven though the points mentioned in the weakness section can be thought of as limitations, I don't find them to be major.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the limitations of current approaches for uncertainty quantification in deep learning and physics-informed learning, such as the need for strong assumptions regarding data likelihood and the computational cost associated with approximating the posterior. The paper introduces a novel problem of confidence interval estimation for deterministic partial differential equations and proposes a method called Physics-Informed Confidence Propagation (PICProp) based on bi-level optimization to compute a valid confidence interval without making heavy assumptions. The paper provides a theorem regarding the validity of the method and computational experiments focusing on physics-informed learning.\n\nSTRENGTHS:\n1. The paper is simple and clear to follow\n2. The paper is well structured \n3. The paper aims at solving an important problem involving UQ in PINN\n4. The theoretical analysis is helpful to better understand the benefits of the method\n\nWEAKNESSES:\n1. The paper claims efficiency but did not demonstrate the cost saving with enough supports\n2. The scope is narrowed to PDE but did not see practical applications, e.g., large-scale cases \n3. The experiments only focus on toy examples that did not show the strength of this work.\n\nQUESTIONS:\nSimilar to the weakness, I have several concerns about the claims and experiments demonstration \n\n1. The experiments can be solved by some baseline methods as the author mentioned because the cost is acceptable. If the authors would like to show the real power of this method, they may consider some real-world problems, either from PDE or simulations. \n\n2. The computational cost is the major barrier to the UQ problem. The method leverages bi-level optimization, how about the cost compared with the traditional methods, e.g., MC-based approaches, specifically in complex cases? For example, high-fidelity simulations, CDF or FEM, can you handle the optimization efficiently without any approximations?  How about the computational cost? \n\n3. I only saw the analytical experiments and have been thinking about how to apply this method to solve real-world applications, is that useful or helpful to handle complex cases in AI for science problems? \n\n----\nAfter rebuttal\n\nI posed several follow-up questions but unfortunately, I did not receive the authors' positive responses although they made efforts to argue. As a result, I incline to keep my original score.\n\nLIMITATIONS:\nDid not see a clear discussion about the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a PICProp, a method to propagate uncertainty through spatiotemporal nonlinear PDE systems. PICProp (1) is more valid and makes less assumptions compared to NN based UQ methods such as dropout and deep ensembles, and (2) can be modified to be more computationally efficient by learning a meta-model. The paper provides extensive computational to show the effectiveness of PICProp.\n\nSTRENGTHS:\n- The methodological section is very clear and easy to follow. PICProp is a principled, concise, and elegant method for the proposed problem setup. \n- I really enjoy the writing and organization in the experiment section as well. The pedagogical example is illustrative of how the algorithm works for different types of CIs, and the following sections showcase the effectiveness of the method in 2d and in PDEs with steep gradients.\n\nWEAKNESSES:\n1. The aper mentions that PICProp produces conservative CIs that are \"clearly preferred in critical and risk sensitive applications\". I don't think this statement is necessarily true. In many cases, one wants the CIs to be valid and **informative**. You can have very large CIs that are trivially valid but too wide to have any meaning in decision-making. (In general, wide CI = safety is neither a correct or scientifically sound statement, don't use it in your papers. Line 285 is an example.) There exist metrics such as Winkler Score and interval scores [1] that principally evaluate the quality of a CI; I recommend the authors to use them. The authors did mention this shortcoming in the conclusion section and say that they will study it in a future work - My opinion is that these studies need to be a part of *this* work for the method to be convincing and usable for the community.\n\n[1] Gneiting, Tilmann, and Adrian E. Raftery. \"Strictly proper scoring rules, prediction, and estimation.\" Journal of the American statistical Association 102.477 (2007): 359-378.\n\n2. The marginal probability framing of CIs (Theorem 4.1) is very close to the guarantees provided by frequentist UQ methods (jackknife and conformal prediction, for example). These are also \"standard\", distribution-free, and widely used UQ methods; it's important to mention and compare to them. Comparing to them will likely help you prove your case of why it's important to have CIs be physically informed. \n\n3. This might be more from of a machine learning community perspective, but I'm curious to see PICProp's performance on a real dataset instead of just on synthetic experiments. The applications authors mentioned in appendix A are all good candidates.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}