Conformalized Physics-Informed Neural Networks

Authors: lena  podina
mahdi  torabi
mohammad  kohandel
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-02-11 02:03:34

Abstract: Physics-informed neural networks (PINNs) are an influential method of solving differential equations and estimating their parameters given data. However, since they make use of neural networks, they provide only a point estimate of differential equation parameters, as well as the solution at any given point, without any measure of uncertainty. Ensemble and Bayesian methods have been previously applied to quantify the uncertainty of PINNs, but these methods may require making strong assumptions on the data-generating process, and can be computationally expensive. Here, we introduce Conformalized PINNs (C-PINNs) that, without making any additional assumptions, utilize the framework of conformal prediction to quantify the uncertainty of PINNs by providing intervals that have finite-sample, distribution-free statistical validity.

Main paper: 
 introduction: 
 Physics-informed neural networks are an influential method developed by Raissi et al. (2019) , which employs neural networks to solve a differential equation (forward problem) well as estimate its parameters (inverse problem). The differential equation (DE) may be ordinary (ODE) or partial (PDE). PINNs have been extended in many ways (e.g. UPINNs by Podina et al. (2023) for learning functions, XPINNs by Jagtap & Karniadakis (2021) for domain decomposition, and DeepONets by Lu et al. (2021) for operator learning), in order to solve increasingly complex problems. However, many of these methods produce only point estimates of the DE solution. This makes it difficult to apply these methods in any practical situation where the error of the solution needs to be quantified or bounded.Uncertainty quantification methods for PINNs have been developed in order to fill this gap, such as Bayesian PINNs in Yang et al. (2021) . This method involves modelling the uncertainty of model and DE parameters using Bayesian statistics. However, this approach has several significant drawbacks. An appropriate prior distribution on the neural network and DE parameters must be determined in advance, and in the case of model misspecification at this stage, the resulting uncertainties may not be calibrated, as shown by Fong & Holmes (2021) . Secondly, as described in Yang et al. (2021) , the noise distribution within the data needs to be assumed in advance. In Yang et al. (2021) , a Gaussian noise model is assumed, but may not be an appropriate model in all cases.Conformal prediction (CP), as introduced by Vovk et al. (2005) , is a user-friendly framework for creating uncertainty intervals on predictions made by any machine learning model, including ones that are black-box or not inherently interpretable. Its distinctive feature lies in the ability to provide intervals that have finite-sample, distribution-free statistical validity. This implies that the method offers explicit and non-asymptotic guarantees without relying on specific distributional or modelrelated assumptions. This makes conformal prediction a versatile tool that can be applied to any model, including neural networks. The most common flavor of CP methods is the Split Conformal method as outlined by Papadopoulos et al. (2002) , which splits the otherwise training set into training and calibration sets; it then uses the former to train a point predictor and the latter to construct uncertainty intervals.Here, we introduce Conformalized PINNs (C-PINNs) and show how they can quantify the uncertainty of: (1) the inferred DE solution (the forward problem) and (2) estimated DE parameters (the inverse problem). We also show how a procedure that utilizes the strong law of large numbers and an underlying property of Split Conformal methods can be used to rigorously check the exact coverage validity of C-PINN intervals. 
methodology and results: 
 For the forward and inverse problems, we consider the logistic growth Ordinary Differential Equation (ODE). This equation is widely used to model logistic growth as described by West & Newton (2019) ; Kohandel et al. (2006) (e.g. in a context where cells (N ) are growing over time (t) but are limited by the nutritional value of the medium):EQUATIONIn Section 2.1, we fix the value of β and the initial condition N 0 , and show the coverage validity of the solution intervals inferred by a Conformalized Physics-Informed Neural Network (C-PINN), for both noise-free and noisy datasets. In Section 2.2, we focus on the inverse problem and show a similar validity for parameter intervals. Model training and architectures are described in Appendix A.2.For the forward problem only, we also consider the 1D Buckley-Leverett Equation Wang et al. (2021) : Li & Mei (2021) showed that a continuous PINN was not able to successfully learn the solution to the equation, likely due to the discontinuous nature of the solution.   u t -f (u) x = 0, x ∈ [-1, 1], t ∈ [0, 1]. u(0, x) = -3 if x < 0 3 if x > 0 (2) taking f (u) = 4u 2 4u 2 + (1 -u) 2 .To see if conformal prediction can still be used to construct intervals with valid coverage when PINNs do not approximate the solution well, we conformalize the forward problem for eq. 2 in Section 2.1.2.In the following two subsections, we conformalize the forward problem for the logistic growth ODE (Section 2.1.1) and a PDE with a discontinuous solution, the 1D Buckley-Leverett equation (Section 2.1.2).First, a numerical solver is used to solve Equation 1 for the initial condition N 0 = 0.1 and β = 0.05 for time 0 to 150. A dataset of 150 tuples {t i , N i } are generated in total. Gaussian noise (noise level 0.08 as per Podina et al. (2023) ) is added to the solution. These are randomly split into holdout set of points (100), training points (25), and test points (25). Then, a single continuous PINN is fit to the training data as per Raissi et al. (2019) , with the parameter value of β given to the PINN in advance. After solving the forward problem, the PINN returns a surrogate solution to the differential equation. Since the solution is modelled using a neural network, it can be evaluated at any time, not just at the training time.We implement split conformal prediction as per Papadopoulos et al. (2002) We then empirically show that the 90% interval has valid coverage (Fig 1b ). To do this, we perform split conformal prediction over many different splits of the validation and calibration points. This procedure is outlined in Angelopoulos & Bates (2021). After constructing the interval using the calibration set, we check how many of the validation points in fact fall in the constructed interval.Angelopoulos & Bates (2021) also shows that for n c calibration points and miscoverage level α, the coverage of the split conformal interval is exactly ⌈(1 -α)(n c + 1)⌉/(n c + 1). Fig 1b shows that the mean coverage converges to exactly this coverage over 10000 random splits (trials), which implies that the constructed intervals are valid.Hence, conformal prediction can be used to quantify the uncertainty and fit of a PINN solution even when there is only a single dataset available (with significant noise). Furthermore, the method yields valid coverage, which we have verified rigorously. We repeat this procedure for noiseless data, showing the fit and coverage convergence in Fig 5 . The predicted 90% and 50% confidence intervals are much smaller since the PINN is better able to predict N test on an unseen test point t test . However, we still have valid coverage as can be seen in Fig 5b . Hence, this method can be used to quantify the performance of the PINN on unseen test data, which is a good measure of the generalization of the solution.We conformalize the forward problem as before for the Buckley-Leverett equation. A PINN is used to solve eq. 2 with 25 noiseless training points for t = 1. The resulting solution matches the training data, but does not adequately represent the solution near discontinuities at x = -2.5, 3.0. In this section, we conformalize the inverse problem. Suppose that we are tasked with providing a valid interval over βtest for a new dataset D test , given a distribution over possible datasets D i . In the simplest form, which we explore here, the given distribution is a distribution over possible β.Then, the C-PINN can be used to construct a valid interval over the estimate βtest for D test . We outline the procedure to do this, and show that the method has valid coverage.To conformalize a new βtest estimate, we generate holdout pairs {β j , D j , βj } (sampled true β j , generated datasets D j using β j , inferred βj of D j ), shown in Fig 3a . We sample β 1000 times from the given distribution, which we assume is Uniform[0, 0.5). We then solve Eq 1 via an ODE solver for these values of β, generating 10 points {t i , N i } per dataset. We obtain a set of 1000 tuples {β j , D j = {t i , N i } j }. Then, for each D j , we employ a standard continuous PINN to estimate the parameter β j in Eq 1. This yields one β estimate per dataset. Since each PINN can only infer β after being trained on a specific dataset, we must use a separately initialized PINN to find β for each dataset. We explore alternative model options in the Conclusion section.We perform split conformal prediction using the previously generated holdout set. For a new dataset D test , we first use a PINN to estimate βtest for the dataset D test . Then, we sample 800 points from the holdout set {β j , βj } to form the calibration set, and construct an 80% confidence interval as per Section 2.1. The interval given by the C-PINN will have valid coverage as long as β test in fact comes from the given distribution (Uniform[0, 0.5) in this case).The mean interval coverage converges to the expected coverage (  
conclusion and discussion: 
 In this work, we conformalize physics-informed neural networks (PINNs) in two ways: first, we conformalize the surrogate solution of the logistic growth ODE and the Buckley-Leverett PDE (Eqs. 1 and 2, forward problem) using a holdout calibration set and split conformal prediction as per Papadopoulos et al. (2002) . In order to validate the method, we employ an averaging approach to compute the coverage of the intervals over many calibration/validation splits. Even when the data is significantly noisy, we show that the empirical coverage converges to the the theoretical coverage.Secondly, we conformalize one inferred parameter (Eq 1, inverse problem). We use the parameter estimates from many PINNs to conformalize the parameter estimate for a new dataset. The resulting interval provides valid coverage for a new, unseen value of β. In order to reduce the computational cost of this method, we note that it is possible to use UPINNs as introduced by  
 
 Appendix:  


Decision: This paper uses conformal prediction within PINNs for uncertainty quantification, which is important in scientific applications. Both reviewers make it clear that this paper is a clear acceptance and so I also vote for acceptance.

Reviews: review: title: title: decent work
main_review: main_review: This paper addresses uncertainty in PINNs through two main approaches:

1. conformalize the surrogate solution of the logistic growth ODE, ensuring that the empirical coverage aligns with theoretical expectations.
2. conformalize the inferred parameters, utilizing parameter estimates from multiple PINNs to establish valid coverage intervals for new datasets, and for unseen values of β.

Suggestions for Improvement:
1. Experimentation with more complex ODE/PDE models could enhance the method's validation.
2. The current study primarily relies on empirical evidence, lacking strong theoretical underpinnings.
title: title: Review on Conformalized Physics-Informed Neural Neworks
main_review: main_review: The paper proposes using conformal prediction in the context of physics-informed neural networks (PINNs) for uncertainty quantification.

It is clearly written and well structured, and the results seem novel. 

My biggest concern / question is regarding the computational cost of the method: The first experiment is made on the very simple logistic ODE, but PINNs are typically used in the context of PDEs. Is there something holding back the method from being applied to more complicated problems? This is not explicitly discussed in the paper, but it would be helpful so that readers can properly assess the utility of the proposed method. Additionally, it is good to see that the method can also be used for uncertainty quantification in parameter inference, but to the best of my understanding the proposed algorihtm does not seem very practival, as it requires many simulations, and training many different PINNs for the individual tasks; but this is acknowledged in the conclusion.

Overall, I think the paper is a good contribution to the topic of uncertainty quantification for PINNs and I therefore recommend its acceptance.