Heteroscedastic uncertainty quantification in Physics-Informed Neural Networks

Authors: olivier  claessen
yuliya  shapovalova
tom  heskes
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-02-09 18:05:23

Abstract: Physics-informed neural networks (PINNs) provide a machine learning framework to solve differential equations. However, PINNs do not inherently consider measurement noise or model uncertainty. In this paper, we propose the UQ-PINN which is an extension of the PINN with additional outputs to approximate the additive noise. The multi-output architecture enables approximation the mean and standard deviation over data using negative Gaussian log-likelihood loss. The performance of the UQ-PINN is demonstrated on the Poisson equation with additive noise.

Main paper: 
 introduction: 
 Alternative approaches to approximate partial differential equation (PDE) solutions have emerged with advances in scientific machine learning (SciML) . One of the methods that recently emerged in SciML is Physics-Informed Neural Networks (PINNs) (Lu et al., 2021) . Using this approach, physical constraints are encoded as additional loss functions, where information about the physical system is used as a regularizer when fitting the model (Raissi et al., 2017; Karniadakis et al., 2021) . PINNs, also called vanilla PINNs, typically utilize a mean squared error loss function and cannot quantify uncertainty.Several approaches have been proposed to enable uncertainty quantification in PINNs. These include negative Gaussian log-likelihood as a loss function and a single scalar, which is used to approximate additive noise in the data set Xiang et al. (2021) . In addition, Bayesian neural networks in combination with sampling methods such as Monte Carlo or Variational Inference Sun & Wang (2020) ; Dandekar et al. (2020) ; Linka et al. (2022) ; Zou et al. (2022) . Another approach is to create an ensemble of models such as PINNs Zou et al. (2022) ; Psaros et al. (2023) . However, many of these types of methods are generally computationally expensive and do not always account for heteroscedastic noise in the data.We propose a multi-output PINN architecture called the UQ-PINN that can quantify aleatoric uncertainty and account for measurement noise. This approach is computationally inexpensive and can closely approximate the standard deviation for different types of homoscedastic and heteroscedastic noise. We evaluate the method on the Poisson equation corrupted with additive noise and compare its performance with a regular PINN. 
physics-informed neural networks: 
 Given a system of parameterized nonlinear partial differential equations (PDEs) of the formEQUATIONwhere x is the D dimensional space or spacetime coordinate vector defined in the physical domain Ω ⊂ R D with boundary Γ. B(•) is a boundary condition operator and b(x) are Dirichlet boundary conditions. F (•) is a differential operator and f (x) is typically referred to as the source term or driving force. The goal of a PINN is to find a surrogate model û(x, θ) that approximates the unknown PDE solution u(x) of this PDE system. The PINN û(x, θ) is parameterized by θ, the parameters of the neural network. In the remainder of the paper, the dependence of û on θ is omitted for simplicity.We focus on forward-deterministic PDE problems, in which the operators F and B are known. For the source terms f (x) and the boundary conditions b(x), we have access to noisy measurements at a finite number of (collocation) points in the domain. Specifically, the observations fi of the actual source term and measurements at the boundary conditions are corrupted with noise:EQUATIONwhereϵ f ∼ N (0, σ 2 (f ) (x (f ) i )) and ϵ b ∼ N (0, σ 2 (b) (x (b) i ))are Gaussian noise terms with heteroscedastic variance. Noise on the source term can be interpreted as a lack of knowledge of the exact parameters of the PDE. Noise at the boundary conditions can be a direct result of noisy sensor measurements. Measurements at the N f collocation points have been sampled S f times, giving a data setD (f ) of N f × S f data points. The boundary condition data set D (b) contains N b × S b data points.To quantify aleatoric uncertainty, we propose to extend the PINN with additional outputs to predict the standard deviations σf and σb as a function of input coordinates x, we call this extension UQ-PINN. In our problem set-up, we allow for additive noise both in the source terms and on the boundary condition measurements. Our goal is not only to estimate the PDE solution but also the standard deviations to help quantify the uncertainty. We use feedforward neural networks with the hyperbolic tangent activation function in all hidden layers. The output layer uses linear activation for the estimation of the mean and exponential activation for estimation of the standard deviation to ensure positive values. û σf σb NN ∂ ∂x ∂ 2 ∂x 2 ûxx - f PDE L f BC û - b Loss BC & IC L b Figure 1: UQ-PINN.An overview of the PINN architecture applied to the Poisson equation. The inputs are x for location and t for time. The feedforward neural network produces an estimate û(x, t) of the PDE solution u(x, t), as well as the estimated standard deviations σf (x, t) and σb (x, t). In the PDE block, the left-hand side of ( 1) is obtained by automatic differentiation, which is compared with the residual of the PDE and combined with the standard deviation σf (x, t) to contribute to the loss L f . In the boundary condition block, the mean estimate is compared with measurements at the boundary conditions and combined with standard deviation σb (x, t) to contribute to the loss L b .PINNs typically use the mean squared error loss function to estimate the solution u of the PDE and are therefore not able to estimate the standard deviations of the noise, σ f and σ b . Instead, we utilized the mean negative Gaussian log-likelihood (NLL) as a loss function to enable estimating the standard deviation. Combining the NLL loss with additional output in the architecture enables UQ-PINN to estimate the mean and standard deviation of the data.The loss function L = L f + L b combines individual losses L f for noisy observations of the source term and L b for the boundary conditions. In effect, L f acts as a regularizer that penalizes solutions that do not adhere to the PDE dynamics, and L b enforces the consistency with the boundary conditions. These two individual losses are defined asL f = 1 N f S f N f S f i=1 NLL( fi ; F (û(x (f ) i ), σ2 (f ) (x (f ) i )) , L b = 1 N b S b N b S b i=1 NLL( bi ; û(x (b) i ), σ2 (b) (x (b) i )),with û the mean prediction and σ2 (•) the estimated standard deviation. 
experiments: 
 Data. To validate our UQ-PINN method, we consider the Poisson equation -∂ 2 u ∂x 2 = π 2 sin(πx), with boundary conditions u(x) = 0, x ∈ {-1, 1}, and analytical solution u(x) = sin(πx) in the domain Ω = [-1, 1]. D f has 16 uniformly sampled sensor locations with S f = 5 samples per domain point. D b contains of two boundary points, x ∈ {-1, 1}, which are sampled S b = 5 times. For performance evaluation, a test data set with N t = 1000 uniformly sampled points was generated in the Poisson equation domain and the exact solution was calculated. Table 1 summarizes the variances of the noise added to the data sets. Table 1 : Different types of generated noise added in the data sets.Experimental setup and training details. The PINN architectures consist of 3 hidden layers with 50 neurons per layer initialized using the uniform Glorot distribution. The network was trained for 100,000 epochs using the ADAM optimizer with a learning rate of 1 × 10 -3 . The σ networks of the UQ-PINN consist of 3 layers with 10 neurons per layer. The weights of the subnetworks are frozen for half of the number of epochs. After half the number of epochs, the weights are unfrozen and the σ networks are updated together with the mean estimation network.Evaluation metrics. We use various metrics to compare the performance of the different approaches.All considered architectures, except vanilla PINN, also output an estimate of the standard deviations σ(b) and σ(f) , which correspond to the standard deviations of noise at the boundary conditions and of the noise for the source term. We measure the quality of our estimates using the Jensen-Shannon divergence, which is a symmetric distance measure between two distributions P and Q Fuglede & Topsoe (2004) :EQUATIONTo evaluate the ability of the different methods to estimate the PDE solution, we use the relative error L2 calculated at all test points:L2 = 1 N test ∥u -û∥ 2 2 ∥u∥ 2 2, where u is the solution the PDE and û is the predicted solution with PINNs.Figure 2 shows examples of the estimated solution for the Poisson equation and standard deviations of the noise using the split-hidden-unit network. We observe that our approach is capable of estimating accurate solutions while also estimating the correct pattern of the standard deviation of the noise. Table 2 presents the Jensen-Shannon divergence (JS PDE for the source term and JS BC for the boundary conditions), which measures the difference between the original and the estimated distribution.We observe that the UQ-PINN methods outperform the constant method in terms of JS-divergence for both estimation of the noise added to the boundary conditions and the source term. The only exception is homoscedastic noise, where the PINN with constant variable architecture slightly outperforms the UQ-PINN in terms of JS-divergence for the source term. This is not surprising, since the constant-variable architecture has a homoscedastic noise assumption due to having a single constant to estimate noise. (Raissi et al., 2017) . 
conclusion: 
 In this paper, we provide a computationally inexpensive approach for quantification of aleatoric uncertainty in PINNs called UQ-PINN. In the UQ-PINN architectures, we extend the PINN by incorporating additional output. These outputs enabled the network to learn a function that approximates the standard deviation of noise in the datasets with respect to the input. We used the negative Gaussian log-likelihood loss function for the proposed UQ-PINN. We compared the UQ-PINN architecture with vanilla PINN and PINN with an external constant to estimate the standard deviation of the noise in data. The comparison was made in terms of the quality of the approximation of the PDE solution using L2 score and JS-divergence in terms of the noise variance estimation accuracy.The UQ-PINN performed similarly to other PINN architectures in terms of mean error L2. However, the vanilla method is unable to approximate noise, and the constant architecture can only estimate homoscedastic noise. We found that the constant architecture demonstrates performance comparable to that of UQ-PINN in scenarios involving homoscedastic noise or a limited number of data samples.In the case of heteroscedastic noise, the UQ-PINN showed a performance superior to that of constantvariable PINN. Both in terms of the estimation of the mean solution and standard deviation of the noise. 
 
 Appendix:  


Decision: Both of the reviewers acknowledge the contribution of adding UQ to Physics-Informed Neural Networks. UQ is very relevant to the workshop and needed in SciML. I recommend the authors expand the literature review and references, e.g., Zou et al. "NeuralUQ: A comprehensive library for uncertainty quantification in neural differential equations and operators" (https://arxiv.org/abs/2208.11866) and benchmarking against other UQ approaches, such as conformal prediction. I vote for acceptance for the workshop.

Reviews: review: title: title: This work presents a noise guided analysis for Physics-informed neural networks (PINNs) to solve differential equation problems.
main_review: main_review: This work presents a noise guided analysis for Physics-informed neural networks (PINNs) to solve differential equation problems. 
The paper is well written and some conclusions on the performance of the UQ-PINN are demonstrated on the Poisson equation with additive noise.
title: title: Review for UQ-PINN's
main_review: main_review: The authors have introduced UQ-PINN's for uncertainty quantification of the inherent noise present in the system. They have introduced a simple framework by adding multiple outputs to the PINN and modifying the regular MSE loss function with negative loss likelihood.

The method is not exactly novel and is widely used in traditional probabilistic deep learning community.
The example shown in the paper is quite simple and straightforward. It might be better if more complicated examples with PDE's are shown. It is natural since PINN's were first introduced to solve PDE's.