Uncertainty Quantification for Physics-Informed Neural Networks with Extended Fiducial Inference
Authors: Frank Shih
Zhenghao Jiang
Faming Liang
Venue: NeurIPS 2025 poster ()
Keywords: Adaptive Stochastic Gradient MCMC
Extended Fiducial Inference
Physics-Informed Neural Network
Stochastic Neural Network
Uncertainty Quantification
Primary area: deep_learning

Abstract: Uncertainty quantification (UQ) in scientific machine learning is increasingly 
critical as neural networks are widely adopted to tackle complex 
problems across diverse scientific disciplines. 
For physics-informed neural networks (PINNs), a prominent model in scientific machine learning, uncertainty is typically quantified using 
Bayesian or dropout methods. However, both approaches suffer from a fundamental limitation: the prior distribution or dropout rate required to construct 
honest confidence sets cannot be determined without additional information.
In this paper, we propose a novel method within the framework of extended fiducial inference (EFI) to provide rigorous uncertainty quantification for PINNs. The proposed method leverages a narrow-neck hyper-network to learn the parameters of the PINN and quantify their uncertainty based on imputed random errors in the observations. This approach overcomes the limitations of Bayesian and dropout methods, enabling the construction of honest confidence sets based solely on observed data.
This advancement represents a significant breakthrough for PINNs, greatly enhancing their reliability, interpretability, and applicability to real-world scientific and engineering challenges. Moreover, it establishes 
a new theoretical framework for EFI, extending its application to large-scale models, eliminating the need for sparse hyper-networks, and significantly improving the automaticity and robustness of statistical inference.

Uncertainty Quantification for Physics-Informed Neural Networks with Extended Fiducial Inference
Frank Shih
Zhenghao Jiang Department of Statistics Purdue University
Department of Epidemiology and Biostatisties Memorial Sloan Kettering Cancer Center New York, NY 10017 shihf@mshe Caihf@mskec.org
West Lafayette, IN 47907 j jiang96@purdue. edu
Faming Liang
Department of Statistics Purdue University
West Lafayette, IN 47907 fml ImliangSpurdue
edu edu
Abstract
Uncertainty quantification (UQ) scientific machine learning increasingly critical as neural networks are widely adopted to tackle complex problems across diverse scientific disciplines. For physics-informed neural networks (PINNs), a prominent model in scientific machine learning, uncertainty is typically quantified using Bayesian or dropout methods. However, both approaches suffer from a fundamental limitation: the prior distribution or dropout rate required to construct honest confidence sets cannot be determined without additional information. this paper, we propose novel method within the framework of extended fiducial inference (EFI) to provide rigorous uncertainty quantification for PINNs. The proposed method leverages narrow-neck hyper-network learn the parameters the PINN and quantify their uncertainty based on imputed random errors in the observations. This approach overcomes the limitations of Bayesian and dropout methods, enabling the construction of honest confidence sets based solely on observed data. This advancement represents significant breakthrough for PINNs, greatly enhancing their reliability, interpretability. and applicability to real-world scientific and engineering challenges. Moreover, it establishes a new thcoretical framework for EFI, extending its application to large-scale models, eliminating the need for sparse hyper-networks, and significantly improving the automaticity and robustness of statistical inference.
Keywords: Adaptive Stochastic Gradient MCMC, Deep Learning, Black-Scholes Model, Partial Differential Equation, Porous-FKPP Model
1 Introduction
Physics-informec neural networks (PINNs) (Raissi et al., 2019) are class of scientific machine learning models that integrate physical principles directly into the training process of the DNN models. They achieve this by incorporating terms derived from ordinary differential equations (ODEs) or partial differential equations (PDEs) into the DNN's loss function. PINNs take spatial-temporal coordinates as input and produce functions that approximate the solutions to the differential equations. Because they embed physics-based constraints, PINNs can typically address problems that are described by few data while ensuring adherence to the given physical laws (Zou et al., 2025; Cuomo
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

et al., 2022). paving the ways for the use of neural networks in out cet-of-distributtion (OOD) prediction (see e.g. Yao et al. (2024)).
From a Bayesian viewpoint, PINNS can be seen as using "informative priors" drawn from mathe- matical models, thus requiring less data for system identification (Zhang et al., 2019). However, this naturally raises a challenge in uncertainty quantification: How should one balance the prior information and the observed data to ensure faithful inference about the underlying physical system? As emphasized in Zou et al. (2025), uncertainty quantification is becoming increasingly critical as neural networks are widely employed to tackle complex scientific problems, particularly in high-stake application scenarios. However, addressing this issue within the Bayesian framework is difficult, as the prior information and data information are essentially exchangeable under Bayesian formulations. The use of "informative priors" can conflict with the spirit of "posterior consistency" (see e.g.. Ghosal et al. (2000)), a foundational principle in Bayesian inference. This conflict creates a dilemma: using an informative prior risks overshadowing the data, while using weak prior may lead to violations of the underlying physical law. In practice, this makes it difficult, if not impossible, to properly calibrate the resulting credible intervals without additional information.
In addition Bayesian methods (Yang et al., 2021), dropout (Srivastava et al., 2014) has also been employed to quantify the uncertainty of PINNs, as demonstrated by Zhang et al. (2019). Dropout primarily used as regularization technique to reduce overfitting during DNN training. Gal and Ghahramani (Gal and Ghahramani, 2016) showed that dropout training in DNNs can be interpreted as approximate Bayesian inference in deep Gaussian processes, allowing model uncertainty to be estimated from dropout- trained DNN models. However, this approach shares a limitation similar to Bayesian methods: The dropout rate, which directly influences the magnitude of the estimated model uncertainty, cannot be determined without additional information, making it challenging to ensure consistent and reliable uncertainty quantification.
This paper introduces an EFI (Liang et al., 2025) approach to quantify the uncertainty in PINNs. EFI provides rigorous theoretical framework that addresses the limitations of Bayesian and dropout methods by formulating the problem as a structural equation-solving each observation expressed as data- mata-generating equation, with the random errors (contained in observations) and DNN parameters treated as unknowns (see Section 2). EFI jointly imputes the random errors and estimates the inverse function that maps the observations and imputed random errors to DNN parameters. Consequently, the imputed random errors are propagated to the DNN parameters through the estimated inverse function, allowing the model uncertainty to be accurately quantified without the need for additional information. Our contribution in this paper is two-fold:
task. In this framework,
A new theoretical framework for EFI: We develop new theoretical framework for EFI that significantly enhances the automaticity of statistical inference. Originally, EFI was developed in Liang et al. (2025) under a Bayesian framework, where a sparse prior is itpossStochasticd on the hyper-network (referred to as the w-network in Section 2) to ensure consistent estimation of the inverse function. However, due to the limitations of existing sparse deep learning theory (Sun et al., 2022), this Bayesian approach could only be applied to models with dimensions fixed or increasing at very low rate with the sample size. In this paper, we propose learning the inverse function using a narrow-ned neck wo- w-network, which ensures consistent estimation of the inverse function without relying on the use of sparse priors. Moreover, it enables EFI to work for large-scale models, such as PINNs, where the number of model parameters can far exceed the sample size. By avoiding the need for Bayesian sparse priors, our framework allows EFI to fulfil the original goal of fiducial inference: Inferring the uncertainty of model parameters based solely on observations.
Open-source software for uncertainty quantification in PINNs: We provide an open source software package for uncertainty quantification in PINNs, which can be easily extended to conventional DNNs and other high- nigh-dimensional statistical models.
Related Work The proposed method belongs to the class of imprecise probabilistic techniques (Augustin et al., 2014). However, compared to other methods in the class, such as credal Bayesian deep learning (Caprio et al., 2023a), imprecise Bayesian neural networks (Caprio et al., 2023b), and other Bayesian neural network-based methods, the key advantage of EFI is that it avoids the need for prior specification while ensuring accurate calibration of predictions.
Another related line of work concerns uncertainty quantification for machine learning models. Beyond the Bayesian and dropout methods noted above, this line includes conformal prediction

---

(Vovk et al., 2005), deep ensembles (Lakshminarayanan et al., 2016), and stochastic deep learning (Sun and Liang. 2022; Liang al., 2022), among others. These methods primarily target predictive uncertainty and are often ineffective inapplicable for quantifying uncertainty in model parameters. In contrast, EFI addresses both predictive uncertainty and parameter uncertainty, and further provides theoretical guarantees for the validity of the resulting prediction and confidence intervals. The ability to accurately quantify uncertainty in deep neural network parameters is a distinctive advantage of EFI
2 A Brief Review of EFI
While fiducial inference was widely considered big blunder by R.A. Fisher, the goal he initially set inferring the uncertainty of model parameters based solely observations - has been continually pursued by many statisticians, see e.g. structural inference (Fraser, 1966, 1968), generalized fiducial inference (Hannig, 2009; Hannig et al., 2016; Murph et al., 2022), and inferential models (Martin and Liu, 2013, 2015; Martin, 2023). To this end, Liang et al. (2025) developed the EFI method based on the fundamental concept of structural inference.
Consider a regression model: Y f(X, Z, 0). where Y R and X Rd represent the response and explanatory variables, respectively; 0 E I"' represents the vector of parameters; and Z E R represents a scaled random error following known distribution =n(+). Suppose that random sample of size n, denoted by ((11221) (U2, 222 )..)..... (Un, Fnn), has been collected from the model. In structural inference, the observations can be expressed data-generating equations as follows:
y/ = fxx, 2x, 0),
(1)
This system of equations consists n + unknowns, namely, {0, 21, 2....., 2n), while there are only n equations. Therefore, the values of 9 cannot be uniquely determined by the data- gentragnng equations, and this lack of uniqueness of unknowns introduces uncertainty in 0.
Let Zn {=1, #2,..... 2n denote the unobservable random errors contained in the data, which are also called latent variables EFI. Let G(-) denote an inverse function/mapping for 0, i.e.,
= G(Y n. Xn, Z,)
(2)
It is worth noting that the inverse function is generally non-unique. For example, it can be constructed by solving any p equations in (1) for 9. As noted by Liang al. (2025), this non-uniqueness of inverse function mirrors the flexibility of frequentist methods, where different estimators of 0 can be constructed achieve desired properties such as efficiency, unbiasedness, and robustness.
Since the inverse function G(-) is generally unknown, Liang et al. (2025) proposed to approximate using sparse DNN, Figure Al in the Appendix for illustration. They also introduced adaptive stochastic gradient Langevin dynamics (SGLD) algorithm, which facilitates the simultaneous training of the sparse DNN and simulation of the latent variables Z,. See Algorithm 1 for the pseudo-code. Refer to Section A1 of the Appendix for the mathematical formulation of the method. Briefly, they let w,, denote the weights of w--etwork and define energy function (Y,,,, n, X Zn, u.,). subsequently, they define a posterior distribution म. (wW,,, x,, Y, Z,,) for w,, and a predictive distribution n. n. X n Y as w.. for Z.. where can be read as a temperature. They treat Zn as missing data and learn w,, through solving the following equation:
Vv, log (w X Y.)
[[V VVw, log (w.,
Xn, Y... (w.|X (Znl Y,, wn) dZn = 0,
(3)
using Algorithm
Under mild conditions for the adaptive SGLD algorithm, it can be shown that
[w " P 0, as lt + 0o,
(5)
where w denotes a solution to equation (3) and & denotes convergence in probability, and that
z (k) d n. Zn X n. Y ni w")), k oc,
(6)
2- Wasserstein distance, where d denotes weak convergence. study the limit of (6) as e decays to 0, i.c., pi, (z Y X w = lim (Z X,, Y, n. w where pi, (z Y n. Xn, w;) is referred
3

---

Algorithm 1: Adaptive SGLD for EFI computation
(i) (Initialization) Initialize w ( M (the number of fiducial samples to collect), and K. (burn-in iterations).
for 1=1.2..., + M do
(ii) (Latent variable imputation) Given wn simulate =h (k=-1 using the SGLD algorithm::
[k+1) Z Uk+1 z., log ne ( (k) [nn, n. Y nn (x)) + v 22n ie (k+11)
where Uk+1 is the learning rate, and e(*+1) ~ ld.)
.... (ymm; Ao,; 2m and (iti) (Parameter updating) Draw minibatch {(y,, 1 21 (10)), )..))....
(8))
(k)
update the network weights by the SGD algorithm:
m
log Tee (V w (+)) + Ve.., log (w)())) ) (4)
w (k-1) w + 7k+1
n
V Vwn
m
where 7 is the step size, and log Te (e(Vi according to (A6).
wn ( can be appropriately defined
(iv) (Fiducial sample collection) If l: + K., calculate 0 ( i 3, 23 for each i E 21, n) and average them to get fiducial 0 sample as calculated in (A5). end
(k+1) w
(v) (Statistical Inference) Conducting statistical inference for the model based the collected fiducial samples.
as the extended fiducial density (EFD) of Zn, Liang al. (2025) impose specific conditions on the structure of the w-network, including that the w-network is sparse and that the output layer width (i.e.. the dimension of 0) is either fixed or grows very slowly with the sample size n. Under these assumptions, they prove the consistency of w,, based on the sparse deep learning theory developed in Sun al. (2022). This consistency further implies that
GO X Zn)
Σ
(1y Bi Zi, wi)),
serves as a consistent estimator for the inverse function/mapping 0 = G(Y n; X,, Zz), where 0(-) denotes the learned neural network function. Refer Appendix A2 for the expression of (2 Y n: X,, n: )
n. Xn, Zn, w;, 0} denote the zero-energy set. Under some regularity Let Z,, { E R" UU,YY
conditions on the energy function, Liang et al. (2025) proved that Z, invariant to the choice of G(-). Let θ (0 RR = GO n. X,, 2). Zn} denote the parameter space of the target model, which represents the set of all possible values of that G.) takes when z runs over Z,,- Then, for any function b(0)
n: X,)) associated with G*)+) is given by of interest, its EFD (.I.,
{ d2 n: X w;;): pr (BYY M. X.) 2.,
(8)
for any measurable set B C θ, where (B) (z E 2, ((G (onnnnn,3)) x,,))) E B}. and P. X Yn. ) denote the cumulative distribution function (CDF) corresponding to uncertainty measure for 6 Practically, it can be Om} denotes the fiducial O-samples collected at step (iv) of Algorithm 1. As practical application, Kim and Liang (2025) applied EFI to quantify the uncertainty of individual treatment effects in causal inference.
(z X n, Y n: ) The EFD provides constructed based on the samples (b(O)), 6(02). b(0m)) }. where (0,,
Finally, we note that for neural network model, its parameters are only unique up to certain loss-invariant transformations, such as reordering hidden neurons within the same hidden layer or simultaneously altering the sign scale of certain connection weights (Sun et al., 2022). Therefore, for the netwworrk, the consistency of un; refers to its consistency with respect to one of the equivalent solutions to (3), while mathematically W., can still be treated as unique.

---

EFI for Uncertainty Quantification in PINNS
3.1 EFI Formulation for PDES
Consider multidimennsional dynamic process, u(z), defined on domain Q C. R2 through PDE:
F(u(z);; - f()), a Ω B(u(x)) = b(x), E an,
where a: (I11 52. -......-------.... (d 1)² Rd indicates the space-time coordinate vector, u(.) represents the unknown solution, B are the parameters related to the physics, and f and are called the physics term and initial/boundary term, respectively. The observations are given in the forms {æ us u i==1 {xo f f and (a",b, by b Let u denote the DNN approximation to the solution u(x), where denotes the DNN parameters. In data-generating equations, the observations be expressed
us = uo (x(")) + z,² i = 1,2,...... flu
f: Flus (3 ): + = i 1, , f1ft
b, = uu (x²))
+ + = 1.2,...,, 1bb.
where 2!", 2/. and 2 independent Gaussian random errors with mean. Our objective is to infer u and/or B, as well as to quantify their uncertainty, given the data and governing physical law. In physics, inferring u with known is termed the forward problem, while inferring when B is unknown is referred to as the inverse problem.
When applying EFI to address this problem, the EFI network comprises two DNNs. The first, referred to as the data modeling network, is used to approximate u(æ). The second, called the w-network. used approximate the parameters of the data modeling network as well as other parameters in equation (9). For an illustration, see Figure Specifically, for the inverse problem, the output the w-network corresponds to 0 {00 3); for the forward problem, its output corresponds
(0).
8-nehwork
822
Y
8
Ga
Z
B
Figure 1: An EFI network with a double neural network (double-NN) structure.
Given the data- calta-generatng equations, the energy function for EFI can be defined as follows:
Un Un(uns Sa f.x 9n. Rn, w.. 7 Σ 110 0 na
lu u=(xx))) - 11
1=1
(10)
||5 F(uo(a))::)) - 2:12 1/b
bb B(ua(x?)) -2/122
M
1=1
1=1
bn (2I,..., = / ...., s. Fe... 2 2T #(2 + 0, o + 0, 2 + 0} denotes the total num-
where Un (u) (u),..,, Un f
(b....., bb)²². Zn
ber of noisy observations in the dataset; and ne. Tue ns and mp are belief weights for balancing
5

---

different terms. Fortunately, as shown in Liang et al. (2025), the choices for these terms will not affect much the performance of the algorithm as long as e + O.
Using EFT to solve PINNs, if we can correctly impute nn and consistently estimate the inverse function G(-), the uncertainty of 9 can be accurately quantified according (8). However, consistent estimation of the inverse function is unattainable under the current EFI theoretical framework due to the high dimensionality of O. which often far exceeds the sample size n. This limitation arises from the existing sparse deep learning theory (Sun et al., 2022), which constrains the dimension of 0 to remain fixed or grow very slowly with n. To address this challenge, we propose a new theoretical framework for EFI, as detailed in Section 3.2, which extends EFI to accommodate large-scale models and addresses the constraints of the current framework.
3.2 A New Theoretical Framework of EFI for Large-Scale Models
EFI treats the random errors in observations as latent variables. Consequently, training the w-network reduced problem of parameter estimation with missing data. Under Bayesian setting, Liang et al. (2025) addressed the problem by solving equation (3) and employ Algorithm 1 for the solution. By imposing regularity conditions such as smoothness and dissipativity (Raginsky et al., 2017), Liang et al. (2025) established the following convergence results for Algorithm 1:
Lemma 3.1. (Theorem 4.1 and Theorem 4.2, (Liang et al., 2025)) Suppose the regularity conditions
in Liang et al. (2025) hold, and the learning rate sequence {UR = 1.2....) and the step size sequence {7 le 1,2,...) are set as: Uk cu+ka and Ykk
Co > 0, C. > 0 and e, > O, and 3 E (0, satisfying 3 S S c,+k# for some constants C. >
Cy
min
(i) (Roet Consistency) There exists a root wh E {w : Vw log π (w) rn) = 0} such that
E B - wi'll S STK,
k: z ko,
for some constant Ç > and iteration number ko > O.
(ii) (Weak Convergence of Latent Variables) Let = = T TZZn|X Y wi) and Tk
Ee Us+1, and let T2=
(Th) denote the probability law of thh Then,
W2(m (T)) m))) S (co0f/4 +cr
1/4 TT. + e2e Te/ens
for any K N. where Wa(..) :) is the 2- Wasserstein distance, Co. C₁, and C2 are some positive constants, CLS is the logarithmic Sobolev constant T², and ög is a coefficient reflecting the variation of the stochastic gradient used in latent variable imputation step.
In our implementation, the latent variable imputation step is done for each observation separately, ensuring 6g = 0. We choose a E (0, and set 71 < T/ for any Tg. which ensures W2 (Tx) xx))
0 as k + co. It is worth noting that Lemma 3.1 holds regardless of the size of the weneework. Thus, it remains valid for large-scale models. However, in this work, we do not impose any priors on wn. which corresponds to the non- -on-informative prior setting T Tww.) x 1.
Given the root consistency result, it is still necessary to establish that the resulting inverse function estimator (7) is consistent with respect to the true parameter 0* to ensure valid downstream inference. Liang et al. (2025) established this consistency using sparse deep learning theory (Sun et al., 2022), but their approach was limited to settings where the dimension of 0 is fixed or increases very slowly with the sample size In this work, we achieve the consistency by employing a narrow-neck w-network, which overcomes the limitation of the current framework of EFI.
To motivate the development of the narrow-neck w-network, we first note an important mathematical fact: As implied by (A6), each 0 RR in the EFI network tends converge a constant vector as 0. Consequently, different components of 0 become highly correlated across n observations and Ô can be effectively represented in much lower-dimensional space, even though the dimension of 0 may be very high. A straightforward solution to address this issue is to incorporate a restricted Boltzmann machine (RBM) (Hinton and Salakhutdinov, 2006) into the w-network, as illustrated in Figure A2, where the neck layer is binary and the last two layers form a coussian-biiaat RBM (Gu et al., 2022; Chu et al., 2017). As discussed in Liang et al. (2025), O can be treated Gaussian

---

random variables in the EFI network. The binary layer of the RBM serves as a dimensionality reducer for O. Such RBM-embedded we-newwork can be trained using the imputation-regullrizee optimization (IRO) algorithm (Liang et al., 2018a) in a similar way to that used Wu et al. (2019).
Since the primary role of the w-network is forward learning of O, it can also be formulated as a stochastic neural network (StoNett) (Liang et al., 2022; Sun and Liang, 2022), where the binary layer can be extended to be continuous. This StoNet-formulation leads to the following hierarchical model:
0 = m 5 teij
(11)
m. = R(u,; Vs),
Ci ~ NOO, o(). v, E Rd denotes a vector of random errors following a known distribution, d, denotes the width of the stochastic neck layer, h denotes the number of hidden layers of the w-network, ,4; gX,, Y,, Z, w!") the mean of the feeding vector to the stochastic neck layer, and R(.) represents transformation. With a slight abuse of
where = 11,.....,..7. ,.j_,...,..j_...
1,2....
notation, we assume that m; has been augmented with constant component to account for the intercept term in the regression model for 0."). For R(.), we recommend the setting:
The neck layer stochastic with: R(p; 0;) = (H: + Uz), where v; NOO, o.) with pre- pre-specified value of o and Y is an relement-wise activation function.
Under this setting, the nnttwrkk forms a nonlinear Gaussian regression with response 14; + v; for i = 1, .., n. Conceptually, the StoNet can be trained using the stochastic EM algorithm (Celeux et al., 1996) by iterating between the steps: (i) Latent variable imputation: Impute the latent variables (v, : 1,....71 n) and Zn = {=1.-... , 2n} conditioned on the current estimates of wn = (w!) wx ). where wh - (c, j 1.2...,.. (ii) Optimization: Conditioned on the imputed latent variables, update the estimates of w/ and wxx separately. Specifically, (1) can be estimated by training the wii' (1) -ntwork using SGD, and wi!)
(2) can be estimated by performing p
linear regressions as specified in (11).
Following the standard theory of the stochastic EM algorithm (Nielsen, 2000; Liang et al., 2018a), (w (1) un(")) will converge to a solution to the equation: Ve,, log n.. (u,,,,,,,,,,,,, Yn) V w. log n. (w,,, n,, Y,.. Z,, Vn)=.(Z, Vn|X,, Yn, wn) dV,, size and the number of iterations of the algorithm become large, where the prior ((w,) 1 and Vn = =(v1, V2,.. v,,)T. Denote the converged solution by w" = (w:((),
O. as the sample
w. " w. The con- sistency of the resulting inverse function estimator can be established by leveraging the sufficient dimension reduction property of the StoNet (Liang et al., 2022) and the prediction property of sufficient dimension reduction
1,-..,7t n) serves for ((X,, Y,, Zz) : = 1,...,,, .n). The consistency of the inverse mapping can thus be ensured by the linear relationship 0 ~ m,, as described in (11), along with the prediction property of linear regression. Note that the linear relationship 0 m. can be generally ensured by the universal approximation ability of the DNN.
linear regressions. Specifically, (m,
Toward rigorous mathematical development, we impose the following conditions:
(i) Amin (B(m,m?)) 2 Pmin == co 8 0, (tt) Amax (B(mm S Pmmx 00, (12)
where B(.) denotes expectation; Amin (-) and Amax denote the minimum and maximum eigenvalue of a matrix, respectively; and ç' > 0, 5 > O. and Pmax > 0 are some constants. Condition (i) generally satisfied by choosing a sufficiently narrow neck layer, in particular, we set d, < n. Condition (ii) is justified in Appendix A4.2.
By the asymptotic equivalence between the StoNet and conventional DNN (Liang et al., 2022) (see also Appendix A4.1), the StoNet can be trained by directly training the DNN. Based on this asymptotic equivalence, Theorem 3.2 establishes the consistency of the inverse mapping learned by Algorithm for large-scale models, see Appendix A4.3 for the proof.
Theorem 3.2. Suppose that the narrow neck layer is set as in (*) with 0² nhdap' the activation function v (.) is c-Lipschitz continuous, and dn, - nn is sufficiently small such that the conditions in
7

---

(12) are satisfied while admitring con-eyppyy-ooo zero energy set Zu. Additionally, assume that the other regularity conditions (Assumptions A4.1-A4.2 in Appendix A4) hold. If < minf ,²..." pda b then the inverse mapping 0 G(Y x, n.). learned by Algorithm I with narrow neck w-network, is consistent.
This narrow neck setting for the w-network eliminates the need to specify a prior for wn" Notably, under this setting, the resulting estimate 0 is not necessarily sparse, introducing a new research paradigm for high-dimensional problems: moreover, the wx -newwork does not need to be excessively large, keeping the overall size of the w-network manageable. This facilitates the application of EFI to high-dimensional and complex models.
4 Simulation Studies
We compared EFI with Bayesian and dropout methods across multiple simulation studies, including the Poisson equation under various settings and the Black-Scholes model. Below, we present the results for the I-D Poisson equation, with results from other studies provided in Appendix A5. The experimental settings for all simulation studies are detailed in Appendix A6.
4.1 1-D Poisson Equation
Consider I-D Poisson equation as in Yang et al. (2021):
82 u
J,
x EEΩ.
(13)
2x2
where Ω = [-0.7 B 0.01, u sin (6x). and f can be derived from (13). Here, we assume
the analytical expression of f is unavailable; instead, 200 sensor mcasurements of f are available with the sensors equidistantly distributed across Ω2 Additionally, there are two sensors at -0.7 and x = 0.7 to provide the left/right Dirichlet boundary conditions for u. We model the boundary noise 2/ N(,, 0.05²) for i 1, 2, ................................ 20, with 10 observations drawn from each boundary sensor. For the interior domain, we assume noise- free measurements of f. This simulation is repeated over 100 independent datasets evaluate the robustness and reliability of the proposed approach.
The Bayesian method was first applied to this example as in Yang et al. (2021) using a two-hidden- layer DNN, with 50 hidden units in each layer, to approximate u(z). The same network architecture was also used for this example by other methods as shown in Table 1. The negative oog-posterior is given follows:
By
IIʃ: - (x 20²
ni
||l Blus (uu))))))1 20²
((0)
Σ 1=1
+
20
(14)
log( iT (0) CG
where C' denotes the log- nogmoooma constant, ni, = 20, f2 = 200, n. 0, and o. = 0.05. The prior T T0) is specified as in Yang et al. (2021), where each unknown parameter assigned an independent standard Gaussian distribution. The same prior is used across all Bayesian simulations in this paper. We implement the method using the Python package hamiltorch (Cobb and Jalaian, 2021). Since / is observed exactly, the corresponding variance o should theoretically be zero. However, the formulation in (14) does not allow Ø = O, necessitating experimentation with different non-zero values of o. These variations resulted in different widths of confidence intervals, as illustrated in Figure A4. Notably, there is no clear guideline for selecting o to achieve the appropriate interval width necessary for the desired coverage rate. This ambiguity underscores the dilemma (mentioned in Introduction) inherent in the Bayesian method. Specifically, the f-term in (14) functions as part of the prior for the parameter 9.
similar issue arises when selecting the dropout rate for PINNs. As illustrated in Figure A3, the uncertainty estimation in PINNs is highly sensitive to the choice of the dropout rate. When the dropout rate approaches zero, the confidence interval collapses into a point estimate, failing to capture uncertainty. Conversely, excessively large dropout rates introduce significant bias, leading to unreliable interval estimates. This highlights the challenge of determining an appropriate dropout rate to balance bias and variability in uncertainty quantification. As possible way to address this issue,
8

---

Concrete Dropout (Gal et al., 2017) was implemented for the example with the code provided at https://gi thub.
/oncaettrrrrr.o..u.......o Unlike the conventional dropout method using fixed dropout rate, Concrete Dropout treats the dropout rate as a hyperparameter and learns it simultaneously when training the model.
2m
0.0 MocuE
(a) Imputed errors vs true errors
(h) QQ-plot of imputed errors
(c) Confidence interval
Figure 2: EFI-PINN diagnostic for ID-Poisson
We then applied EFI to this example, using the same DNN structure for the data-modeling network. Figure summarizes the imputed random errors by EFI for 100 datasets, showing that EFI correctly imputes the realized random errors from the observations. Consequently, EFI achieves the correct recovery of the underlying physical law.
Table 1: Metrics for ID- averaged over 100 runs.
Method
MSE
Coverage Rate
CI-Width
PINN (no dropout)
0.000121 (0.000012)
0.0757 (0.011795)
0.002139 (0.000024)
Dropout (0.5%)
0.000228 (0.000024)
1.0000 (0.000000)
0.191712 (0.004367)
Dropout (1%)
0.000851 (0.000111)
0.9999 (0.000104)
0.274651 (0.006734)
Dropout (5%)
0.006276 (0.000982)
0.9893 (0.004024)
0.660927 (0.030789)
Concrete Dropout
0.000184 (0.000020)
0.1185 (0.018002)
0.003706 (0.000073)
Bayesian (a, 0.05)
0,000826 (0.000102)
0.9997 (0.000302)
0.372381 (0.002635)
Bayesian f = 0.005)
0.000170 (0.000020)
0.9914 (0.008600)
0.081557 (0.000692)
Bayesian (ay 0.0005)
0,001460 (0.000219)
0.5884 (0.032390)
0.055191 (0.001123)
EFI
0.000148 (0.000016)
0.9517 (0.014186)
0.050437 (0.000462)
We evaluated the accuracy and robustness of each method by recording three metrics, as summarized average across 100 repeated experiments. The 'MSE' measures the mean
in Table 1, based squared distance between the network prediction and the true solution. The coverage rate represents the percentage of the true solution contained within the 95% confidence interval, while the CI-Width quantifies the corresponding interval width. As discussed above, the dropout method produces inflated coverage rates and excessively wide confidence intervals, with CI widths ranging from 0.1917 to 0.6609. Similarly, the Bayesian method is sensitive to the choice of oy, with larger o values yielding inflated coverage rates, while smaller a values result in underestimated coverage rates. In contrast, EFI is free of hyperparameter tuning, making inference on the model parameters based on the observations and the embedded physical law. It produces the most balanced results, with mean squared eтror (MSE) of 1.48 x 10 and coverage rate of 95.1 15.1,, closely aligning with the nominal 95% target. Moreover, it provides the shortest Cl-width 0.0504. Concrete Dropout achieves comparable MSE as EFI, indicating similar model estimation accuracy. However, it performs significantly worse in terms of uncertainty quantification, with a coverage rate of only 1 11.85% far below the nominal 95%. This substantial under-coverage is due to an underestimation of predictive uncertainty, as evidenced by the markedly narrower confidence interval width.
5
Real Data Examples
This section demonstrates the abiliy of the EFI-PINN framework to quantify uncertainty for models learned from real data. We considered two models: the Montroll growth model,
du
1 1--())
4 ))
u
(15)
dt

---

where k, C. and & are unknown parameters; and the reaction-diffuse model governed by the generalized Saeiss-iihhr-oomoogeee Kolmogoriv-Petroosky www.ersky--ssskiiiiee (P-FKPP) equation:
du dt
m Ou ru
(m K
(16)
dx
dr
where D, m, and unknown parameters. We modeled the Chinese hamster V79 fibroblast tumor cell growth data (Rodrigues, 2024) using equation (15), and the scratch assay data (Jin and Cai, 2006) using equation (16). Additional details about the datasets are provided in the Appendix. Figure illustrates that the proposed EFI-PINN framework is not only capable of learning the PDE models from data but also effectively quantifying the uncertainty of the models. Further results and discussions can be found in the Appendix. The prediction uncertainty can also be quantified with the proposed method.
125
Bons
196
125
1000
-12
Bin
Dribter
Figure 3: Confidence bands (shaded arcas) for the learned models: (left): Montroll growth model; (right): generalized P-FKPP model.
6 Conclusion
This paper presents a novel theoretical framework for EFI, enabling effective uncertainty quantifica- tion for PINNs. EFI addresses the challenge of uncertainty quantification through unique approach of "solving data-generating equations," transforming it into an objective process that facilitates the construction of honest confidence sets. In contrast, existing methods such as dropout and Bayesian approaches rely on subjective hyperparameters (e.g., dropout rates or priors), undermining the honesty of the resulting confidence sets. This establishes new research paradigm for statistical inference of complex models, with the potential to significantly impact the advancement of modern data science.
Although Algorithm 1 performs well in examples, its efficiency be further improved through several enhancements. For instance, in the latent variable imputation step, the SGLD algorithm (Welling and Teh, 2011) can be replaced with the Stochastic Gradient Hamiltonian Monte Carlo (Chen et al., 2014). which offers better sampling efficiency. Similarly, in the parameter updating step, the SGD algorithm convergence to a good local minimum during the early stages of training. As training progresses, the momentum be gradually reduced zero, ensuring alignment with the convergence theory of EFI. This approach balances computational efficiency with theoretical rigor, enhancing the overall performance of EFI.
be accelerated with momentum. Momentum-based algorithms facilitate faster
While the examples presented focus on relatively small-scale models, this is not a limitation of the approach. In principle, EFI be extended to large-scale models, such ResNets and CNNs, through transfer learning a direction we plan to explore in future work.
Acknowledgments
Liang's research is supported in part by the NSF grant DMS-2210819 and the NIH grant R01- GM152717.
Shih's 's research is partially supported by MSK Cancer Center Support Grant/Core Grant (P30 CA008748).
10

---

References
Augustin, T., Coolen, F. P. A., de Cooman, G., and Troffaes, M. C. M. (2014), Introduction to imprecise probabilities, John Wiley & Sons.
Black, F. and Scholes, M. S. (1973), "The Pricing of Options and Corporate Liabilities," Journal of Political Economy, 81, 637 - 655.
Caprio, M., Dutta, S., Jang, K. J., Lin, V., Ivanov, R., Sokolsky, O., and Lee, I. (2023a), "Credal Bayesian Deep Learning,"
(2023b), "Imprecise Bayesian Neural Networks," ArXiv, abs/2302.09656.
Celeux, G., Chauveau, D., and Diebolt, J. (1996), "Stochastic versions of the EM algorithm: an experimental study in the mixture case," Journal of Statistical Computation and Simulation, 55, 287-314.
Chen, T., Fox, E., and Guestrin, C. (2014), "Stochastic gradient hamiltonian monte carlo," in International conference on machine learning, pp. 1683-1691.
Chu. J., Wang, H., Meng, H., Jin, P., and Li, T. (2017), "Restricted Boltzmann Machines With Gaussian Visible Units Guided by Pairwise Constraints," IEEE Transactions Cybernetics, 49, 4321-4334
Cobb, A. D. and Jalaian, B. (2021), "Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splitting." Uncertainty in Artificial Intelligence.
Cuomo, S., Cola, V. S. D., Giampaolo, F., Rozza, G., Raissi, M., and Piccialli, (2022), "Scientific Machine Learning Through Physics-Informed Neural Networks: Where we are and What's Next," Journal of Scientific Computing, 92.
Deng, W., Zhang, X., Liang, F., and Lin, G. (2019), "An adaptive empirical Bayesian method for sparse deep learning," Advances in neural information processing systems, 32.
Dittmer, S., King, E. J., and Maass, (2018), "Singular Values for ReLU Layers," IEEE Transactions on Neural Networks and Learning Systems, 31, 2599-36055
Fraser, D. A. S. (1966), "Structural probability and generalization." Biometrika, 53, 1-9.
(1968), The Structure of Inference, New York-London-Sydite John Wiley & Sons.
Gal, Y. and Ghahramani, Z. (2016), "Dropout Bayesian Approximation: Representing Model Uncertainty in Deep Learning," in International Conference on Machine Learning, vol. PMLR 48, pp. 1050-1059.
Gal, Y., Hron, J., and Kendall, (2017), "Concrete dropout," Advances in neural information processing systems, 30.
Ghosal, S., Ghosh, J. K., and der Vaart, A. (2000), "Convergence rates of posterior distributions," Annals of Statistics, 28, 500-531.
Gu, L., Yang, L., and Zhou, F. (2022), "Approximation properties of Gaussian couusian-biiary restricted Boltzmann machines and Gaussian- binary deep belief networks," Neural networks the official journal of the International Neural Network Society. 153, 49-63.
Hannig, J. (2009), "On generalized fiducial inference." Statistica Sinica, 19, 491-544.
Hannig, J., Iyer, H., Lai, R. C. S., and Lee, T. C. M. (2016), "Generalized Fiducial Inference: A Review and New Results," Journal of the American Statistical Association, 111, 1361-136..
Higham, N. J. and Cheng, S. H. (1998), "Modifying the inertia of matrices arising in optimization," Linear Algebra and its Applications, 261-279.
Hinton, G. E. and Salakhutdinov, R. (2006), "Reducing the Dimensionality of Data with Neural Networks." Science, 313, 504 104-5507 507.

---

Jin, J. and Cai, T. T. (2006), "Estimating the Null and the Proportion of Nonnull Effects in Large-Scale Multiple Comparisons," Journal of the American Statistical Association, 102, 495 506.
Kim, S. and Liang, F. (2025), "Extended fiducial inference for individual treatment effects via deep neural networks," Statistics and Compuling, 35.
Lagergren, J. H., Nardini, J. T., Baker, R. E., Simpson, M. J., and Flores, K. B. (2020), "Biologically- informed neural networks guide mechanistic modeling from sparse experimental data," PLoS Computational Biology, 16.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2016), "Simple and Scalable Predictive Uncer- tainty Estimation using Deep Ensembles," in Neural Information Processing Systems.
Liang, F., Jia, B., Xue, J., Li, Q., and Luo, Y. (2018a), "An imputation-rogullrize optimization algorithm for high- high-dimensional missing data problems and beyond," Journal of the Royal Statistical Society, Series B, 80, 899-926.
Liang, F., Kim, S., and Sun, Y. (2025), "Exended Fiducial Inference: Toward an Automated Process of Statistical Inference," Journal of the Royal Statistical Society Series 87, 98-131.
Liang, F., Li, Q., and Zhou, L. (2018b), "Bayesian Neural Networks for Selection of Drug Sensitive Genes," Journal of the American Statistical Association, 113, 955-972.
Liang, S., Sun, Y., and Liang, F. (2022), "Nonlinear Sufficient Dimension Reduction with a Stochastic Neural Network," NeurlPS 2022.
Martin, R. (2023), "Fiducial inference viewed through a tossibility-theoretic inferential model lens," Journal of Machine Learning Research, 215, 299-310.
Martin, R. and Liu, C. (2013), "Inferential Models: Framework for Prior-Free Posterior Probabilis-
tic Inference," Journal of the American Statistical Association, 108, 301 - 313.
- (2015), Inferential Models: Reasoning with Uncertainty, CRC Press.
Marusic, M., Bajzer, Z., Freyer, J. P., and Vuk-Pavlović, S. (1994), "Analysis of growth of multicellular tumour spheroids by mathematical models," Cell Proliferation. 27.
Milnor, J. and Stasheff, J. D. (1974), Characteristic Classes, Princeton University Press.
Murph, A. C., Hannig, J., and Williams, J. (2022), "Generalized Fiducial Inference on Differentiable Manifolds," arXiv:2209.1554
Niclsen, S. (2000), "The stochastic EM algorithm: Estimation and asymptotic results," Bernoulli, 457-489.
Raginsky, M., Rakhlin, A., and Telgarsky, M. (2017), "Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis," in Conference on Learning Theory, PMLR, pp- 1703-1703.
Raissi, M., Perdikaris, P., and Karniadakis, G. E. (2019). "Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations," J. Comput. Phys., 378, 686-707.
Rodrigues, J. (2024), "Using Physics-Informed Neural Networks (PINNs) for Tumor Cell Growth Modeling," Mathematics, 12.
Song, Q. Sun, Y., Ye, M., and Liang, F. (2020), "Extended Stochastic Gradient MCMC for Large- Scale Bayesian Variable Selection," Biometrika, 107, 997-1004.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, (2014), "Dropout: simple way to prevent neural networks from overfitting." The journal of machine learning research, 15, 1 1958.
Sun, and Liang, (2022), "A kernel-expanded stochastic neural network," Journal of the Royal Statistical Society Series B, 84, 547-578.
12

---

Sun, Y., Song. Q. and Liang, F. (2022). "Consistent Sparse Deep Learning: Theory and Computation." Journal of the American Statistical Association, 117, 1981-1995.
Vovk, V., Gammerman, A., and Shafer, (2005), Algorithmic Learning in a Random World, Springer.
Welling, M. and Tch, Y. W. (2011), "Bayesian learning via stochastic gradient Langevin dynamics," " in Proceedings of the 28th international conference machine learning (ICML-II), pp. 681-688.
Wu, M., Luo, Y., and Liang, F. (2019), "Accelerate Training of Restricted Boltzmann Machines via Iterative Conditional Maximum Likelihood Estimation." Statistics and its interface, 12, 377-385.
Yang. L., Meng, X., and Karniadakis, G. E. (2021), "B-PINNs: Bayesian physics neural networks for forward and inverse PDE problems with noisy data," Journal of Computational Physics, 425, 109913.
Yao, Y., Yan, S., Goehring, D., Burgard, W., and Reichardt, J. (2024), "Improving Out-of-Distribuuic Generalization Trajectory Prediction for Autonomous Driving via Polynomial Representations, ArXiv, 13./24
Zhang, D., Lu, L., Guo, L.,, and Karniadakis, G. E. (2019), "Quantifying total uncertainty in physics- informed neural networks for solving forward and inverse stochastic problems," J. Comput. Phys., 397, 108850.
Zou, Z., Meng. X., and Karniadakis, G. E. (2025), "Uncertainty quantification for noisy ouputs- in physics-informed neural networks and neural operators," Computer Methods in Applied Me- chanics and Engineering.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract states that we provide theoretical proof for the proposed framework and numerical experiment for overcoming limitations for Bayesian PINN and Dropout PINN.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the conclusion, mentioned that the examples presented are relatively small scale models, however, the proposed algorithm can be extended easily to large scale models through transfer learning.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: In this paper, we provide convergence theory for proposed algorithm.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide our source code and hyperparameters in the supplementary mate- rial.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide our source code and hyperparameters in the supplementary mate- rial.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide source code and hyperparameters in the supplementary mate- rial.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All our experiments replicated 100 times with randomly chosen random seeds, each metic is averaged over 100 runs to ensure statistical significance.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
14

---

Answer: [Yes]
Justification: provide the wall-clock time for the Poisson-1D experiment with different algorithms in the supplementary material.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: www.rriss.cpppuuiiiie "ptpbiii//tiiccssiielleeeee"
Answer: [Yes]
Justification:
The research relies solely simulated and publicly available open-source data. No human, animal, or private data were used. The methods developed and evaluated pose foreseeable ethical concerns and comply fully with the NeurlPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This work develops foundational algorithm for uncertainty quantification in PINNs. While it does not involve sensitive data or direct deployment, may support future applications scientific modeling and engineering.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper does not release any pretrained models or datasets with potential for misuse. All experiments are conducted using simulation or publicly available open-source data, and the developed methods pose no foreseeable dual-use concerns.
Justification:
Licenses existing assets
Question: Are the creators original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external assets used in this work- such as open-source code and datasets are properly credited in the paper. We ensured that their licenses (e.g., MIT, Apache 2.0) were respected, and we include relevant citations and links where applicable.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets yet.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing research with human subjects.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
15

---

Answer: [NA]
Justification: This paper does not involve research with human subjects or any activity requiring IRB or equivalent approval.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: This paper does not involve the use of large language models (LLMs) as part the core methodology. Any use LLMs was limited to minor writing or editing support and did not influence the scientific content originality of the research.
16

---

Appendix
This appendix is organized as follows. Section A1 provides brief review of the EFI method and its self-diagnostic property. Section A2 presents the analytical expression of the extended fiducial density (EFD) function of n. Section A3 includes an illustrative plot of the proposed narrow-neat w-network. Section A4 gives the proof of Theorem 3.2. Section A5 presents additional numerical results, Finally, Section A6 details the parameter settings used our numerical experiments.
A1
A Brief Review of the EFI Method and Its Self-Diagnosis
A1.1
The EFI Method
To ensure smooth presentation of the EFI method, this section partially overlaps with Section 2 of the main text. Consider a regression model:
fXX, Z.0),
where Y R and X Id represent the response and explanatory variables, respectively: Rº rep- resents the vector of parameters; and Z E R represents scaled random emror following a known distri- bution (-).). Suppose that random sample of size n, denoted by { 1)), (V2, .2))..., (y' Tx)), has been collected from the model. In structural inference, the observations can be expressed in generating equations as follows:
3/ f(z,, 0), = 1,2, .2.
(A1)
This system of equations consists n unknowns, namely, {0, 21, ..,..., )n) while there are only equations. Therefore, the values of I cannot be uniquely determined by the data-generating equations, and this lack of uniqueness unknowns introduces uncertainty in
Let Zn {=1, #2, 2n denote the unobservable random errors contained in the data, which are also called latent variables EFI. Let G(-) denote an inverse function/mapping for i.e.,
0 G(Y n. X Zn).
(A2)
is worth noting that the inverse function is generally non-unique. For example, can constructed by solving any p equations in (1) for 0. As noted by Liang al. (2025), this non-uniqueness of inverse function mirrors the flexibility of frequentist methods, where different estimators of can be constructed achieve desired properties such efficiency, unbiasedness, and robustness.
Ncural Nets (w,,)
6
As
X
012
Figure A1: Diagram of EFI given in Liang al. (2025): The orange nodes and orange links form deep neural network (DNN), referred to as the w-network, which is parameterized by w,, (with the subscript n indicating its dependence on the training sample size n)); the green node represents the latent variable to impute; and the black lines represent deterministic functions.
Since the inverse function G(-) is generally unknown, Liang et (2025) proposed to approximate using sparse DNN, see Figure Al for illustration. The EFI network has two output nodes defined,
17

---

respectively, by
6111 == |10, -
(A3)
and c, = d(vi. ú:) = d(ys- Ti 0),
where y; - f(,,, 2,, 0). f() is as specified (1), and d(.) is function measuring the difference between yi and Di- That is, the node 6i, quantifies the variation of 0,, while the node 6122 represents the fitting error. For normal linear/nonlinear regression, d(-) can be defined
d(ye: 2, 0) l f(z,, 24, 0)1
(A4)
For logistic regression, it is defined as ReLU function, Liang et al. (2025) for details.
Let 0, := (V,, 2; 2;, w,) denote the DNN prediction function parameterized by the weights wx in the EFI network, and let
1
Nyy. 2i, 24, Wn),
(A5)
T1
T1
which serves as an estimator of the inverse function G(-). EFI defines energy function
Un (Y n: X Z w.) 110 Σ 10: 01
Σ
d(y,, Fy 24, 0),
(A6)
where Y/o 0 a regularization parameter, 0,'s and can be expressed as functions of (Yn, Xn, Zn. wn), and dd(-) is function measuring the difference between y/ and jr. The likelihood
function is given by
Te (Y (Yn|X X,, Wn) x exe-Us NY..
(A7)
for some constant close to 0. As discussed in Liang al. (2025), the choice of nh does not affect much on the performance of EFI as long as e sufficiently small. Subsequently, the posterior of wn, is given by
( X n. Y,, Za) x x(W,)e x..,,..........0
where N (wn) denotes the prior of wn: and the predictive distribution of Dn is given by
Te Z Xn, Y,, wn) x TD @n (Z.)e U..YY.Y..... n.. o..........f
where 8n (2,) I[= To(=+) under the assumption that z,'s are independently identically dis- tributed (i.i..). In EFI, Wn, is estimated through maximizing the posterior т. (wn| Xn, Y.) given the observations {X,,, n. Y J. By the Bayesian version of Fisher's identity (Song et al., 2020), the gradient equation V Vu,, log n. (w,, [X n1 nn) = can be re-expressed as
V Vw,, log T. (w,, X Yn) = / V w. log (w, Xn, Y Zn)
(A8)
x n. Zn Xn, Y, nn w..) dZn = 0,
which can be solved using an adaptive stochastic gradient MCMC algorithm (Liang et al., 2022: Deng al., 2019). The algorithm works by iterating between the latent variable imputation and parameter updating steps, see Algorithm for the pseudo-code. This algorithm is termed "adaptive" because the transition kernel in the latent variable imputation step changes with the working parameter estimate of W.. The parameter updating step can be implemented using mini-batch SGD, and the latent variable imputation step can be executed in parallel for each observation (yi, æ.). Hence, the algorithm is scalable with respect to large datasets.
Under mild conditions for the adaptive SGLD algorithm, can be shown that
w (8)
w 3 0, as k co,
(A9)
where " denotes a solution to equation (3) and 2 denotes convergence in probability, and that
Z (k) ne ZZ X Y w,), k - oc,
(A10)
18

---

in 2- Wasserstein distance, where d denotes weak convergence. study the limit of (6) as e decays to 0, i.e.,
(n n. X,, w lim ne (Z n, Y n: w,,); <10
where P. (= n. x,, ( is referred to as the extended fiducial density (EFD) of n. Liang et al. (2025) impose specific conditions the structure of the w-network, including that the w-network is sparse and that the output layer width (i.e., the dimension of 0) is either fixed or grows very slowly with the sample size n. Under these assumptions, they prove the consistency of w,, based on the sparse deep learning theory developed in Sun et al. (2022). This consistency further implies that
GY X, n2 nn)
M(v Fy 2,, w,));
(A11)
serves as a consistent estimator for the inverse function//apping 0 G(Y,, m. X Za). Refer to Appendix A2 for the analytic expression of (= X,,,,,,..
n. X, n. Zn, w 0} denote the vero-energy set. Under some regularity conditions on the energy function, Liang et al. (2025) proved that Z, is invariant to the choice of G(.). Let θ (0 TRP 0 G G" n. X 2). E ] denote the parameter space of the target model, which represents the set of all possible values of that G*(.) takes when z runs over Z,,. Then, for any function b(0)
Let 2,, { F" (Y
(-) is given by of interest, its EFD (. n: x,,) associated with G*)-)
H (B n. X..) f (z X.,Y,.... wx))
(A12)
(B)
for any measurable set B O, where (B) = (= Z, (G n. X z)) B). and (= Xn, Y,, w denote the cumulative distribution function (CDF) corresponding to (= n. Y, n: ) The EFD provides an uncertainty measure for b(0). Practically, can be constructed based on the samples {b(01). b(02)..... b(0m), where {01, O....., Om) denotes the
fiducial samples collected at step (iv) of Algorithm 1.
Finally, we note that for a neural network model, its parameters are only unique up to certain loss-invariant transformations, such reordering hidden neurons within the same hidden layer simultaneously altering the sign or scale of certain connection weights (Sun et al., 2022). Therefore, for the wenttwork, the consistency of ",,,, refers its consistency with respect one of the equívalent solutions to (3), while mathematically w. can still be treated as unique.
A1.2 Self-Diagnosis in EFI
Given the flexibility of DNN models, reliable diagnostics are crucial in deep learning to ensure model robustness and accuracy while identifying potential issues during training. Unlike dropout and Bayesian methods, which lack self- self-diagnostic capabilities, EFI includes a built-in mechanism for self-diagnosis. Specifically, this can be achieved through (i) analyzing the QQ-plot of the imputed random errors, and (ii) verifying that the energy function Un, converges to zero
According to Lemma 3.1, the imputed random errors 2n should follow the same distribution as the true random errors Z. Since the theoretical distribution of Z, is known, the convergence of 2,, can be assessed using QQ-plot as shown in Figure 2(b). When Z,, has been correctly imputed, the energy function must converge to zero to ensure the consistency of the inverse function estimator. In practice, we can check whether U, (Y,,. n: X,, Zn, un,) = o(e) as e - The validity of inference for the model uncertainty can thus be ensured if both diagnostic tests are satisfied. This diagnostic method is entirely data-driven, offering a simple way for validating the EFI results.
If the diagnostic tests are not satisfied, the hyperparameters of EFI can be adjusted to ensure both tests are met for valid inference. These adjustments may include modifying the width of the neck layer, the size of the wi -network, the size of the data-modeling network, as well as tuning the learning rates and iteration numbers used in Algorithm 1.
A2 Extended Fiducial Density Function of Zn
Let Σ, = (2 E R. UY n, X,, Zn, un = 0} denote the zero-energy set, and let (x Y w denote the cumulative distribution function (CDF) corresponding to p (z) X,,, Y n, W.). Under
19

---

some regularity conditions on the energy function, Liang et al. (2025) proved that 2n is invariant to the choice of G()). Furthermore, they studied the convergence of lim, (=|x,, Y., n. w in two
cases: IT ( (..) > O and II (2n) = O, where IT (.) denotes the probability measure corresponding to the density function To (z) on RR.. Specifically,
(a) (2,) > 0): In this case, (2 Y,, w given by
dP X Y m. wx
(z), EE 2n
(A13)
dz
(2,)
which invariant to the choices the inverse function G(-) and energy function U,(()). For example, the logistic regression belongs to this case as shown in Liang et al. (2025).
(b) (II, (Z,) = 0): In this case, Z, forms manifold in Iº with the highest dimension and ph (= Y X,, w concentrates on the highest dimensional manifold and is given by
"0 @n (z) (det (iU,())))) 1/2
(z Y W (z) du
2 E Zn,
(A14)
Jz. To son (z) (det(V)U,,, 1/2 dv
where v is sum of intrinsic measures on the p-dimensional manifold in Zn. and E R"-P denotes the coefficients of the normalized smooth normal vectors in the tubular neighborhood decomposition (Milnor and Stasheff, 1974) of z.
By Theorem 3.2 and Lemma 4.2 in Liang et al. (2025), if the target model is addiiiv and d(-) specified as in (A4), then PT Xn, Y, n, w in (A14) can be reduced to
dP" (z Xax n Y,,. n: w
25 70
(A15)
du
2. nO (z)dv
That is, under the consistency of w;, (2| n: Y... w is reduced to truncated density function of Thn (z) on the manifold Z,, while Z, itself is also invariant to the choice of the inverse function. In other words, for noise-additive models, the EFD of Z, is asymptotically invariant the inverse function we learned given its consistency.
A3 Narrow-Neck w-Networks
Output O
Neck
X
Y
2
(2)
W n
w
Figure A2: A conceptual structure of narrow neck w-networks.
A4 Proof of Theorem 3.2
A4.1 Asymptotic Equivalence between StoNets and DNNs
We first provide brief review of the theory regarding asymptotic equivalence between the StoNet and DNN models, which was originally established in Liang et al. (2022).
20

---

Consider StoNet model:
Y1 bj + wT X + +e1,
Y. b, + w; ( Y (-1)
et == 2,3, .....
(A16)
Y bp+1 + wh+1 (Y eh-1:
where X Iº and Y Rdh++ represent the input and response variables, respectively; Y, Rd are latent variables; e, E Rd: are introduced noise variables; b, R and w; E IR xd are model parameters, do p. and V (Y (-1) (v(Y,
i-.2) (Y (1,,,,)) is an element-wise activation function for = L.... , h 1. The StoNet defines a latent variable model that reformulates the DNN composition of many simple regressions. In this context, assume that e; ~ (0, Ia.) for = 1, .......... h + 1, though other distributions can also be considered for e, 's (Sun and Liang, 2022).
The DNN model corresponding to (A16) 16) is given as follows:
Y = 61 + w X, Y. = b, + wi V (Y: 1), i= 2,
(A17)
Y b wh+1 1 (Y a)) + Ch+1-
Let 0 (b1, W....., bb+1. Wh+11 } denote the parameter set of the DNN model. DDNNNNNN 9) denote the likelihood function of the DNN model (A17), and let TY Y [X, denote the likelihood function of the StoNet (A16). (0) E(log TDNN (Y)X, O)), where the expectation taken with respect to the joint distribution (X, Y), Liang et al. (2022) made the following assumption regarding the network structure, activation function, and the variance of the latent variables of StoNet:
Assumption A4.1. (i) Parameter space θ (of Q) is compact; (ii) For any & E +.
E(log Y mis] X, 0))² < xc; ; (iii) The activation function v=)-) is e-Lipschitz continuous; (iv) The network's widths di's and depth h are allowed to increase with (v) The noise introduced in
StoNet satisfies the following condition: < 02 S.... .... Oh-1, 1. and da ([ di d
c for any E (1,2,...,11
Assumption A4.2. (i) (*(0) is continuous in 2 and uniquely maximized 0": (ii) for any
0, supace B(c) B(c (*)) exists, where Bec) = {0 : ||0 0"ll < c), and å = (0))
supece, (0) 0.
Under Assumptions A4.1 and A4.2, Liang et al. (2022) proved the following lemma. Lemma A4.3, (Liang et al., 2022) Suppose that Assumptions A4.1-A4.2 hold, and x (Y, Y miss XX, 0) continuous O. Then
(i) sup
Σ
log T (+),
rú misl (x()
0)
Σ
log TDNN (
x
00 B (100))
DEE
7
n
(A18)
(ii)
in. 11
1 0,
as
max B(log TDNN (Y 0)) denotes the true parameters of the DNN model as specified in (A17), and i, = arg max ( En= log (Y yli) X ()))) {}} denotes the mis maximun likelihood estimator of the StoNet model (A16) with the psendo-complete data.
where 0" = arg max a
Lemma A4.3 implies that the StoNet and DNN are asymptotically equivalent the training sample size n becomes large, and it forms the basis for the bridging the StoNet and the DNN. The asymptotic equivalence can be elaborated from two perspectives. First, suppose the DNN model (A17) is true. Lemma A4.3 implies that when becomes large, the weights of the DNN can be learned by training a StoNet of the same structure with of's satisfying Assumption 14.1-(v). On the other hand, suppose that the StoNet (A16) is true, and then Lemma A4.3 implies that for any StoNet satisfying Assumptions A4.1 & A4.2, the weights can be learned by training DNN of the same structure when the training sample size is large.
21

---

A4.2 Justification for Condition (12)-(ii)
To justify this condition, we first introduce the following lemma:
Lemma A4,4. Consider a random matrix M Raad with 2 d. Suppose that the eigenvalues
of MM' M are upper bounded, i.e,, Xuax (MI (M MV) S Fimax for some constant Kmax > O. Let Vi (MI) denote un elementwise transformation of M.. Then Amax (MV())) (MMM) S Kimax for the tanh, sigmoid and ReLU transformations.
Proof. For ReLU, the result follows from Lemma of Dittmer et al. (2018). For tanh and sigmoid, since they are Lipschitz continuous with Lipschitz constant of 1, Lemma 5 of Dittmer et al. (2018) also applies.
Since the connection weights take values in a compact space θ there exists a constant < Tmax < 00 such that
Amax ( (w,w() I Tmax::
for any / = 1,2, 1, where wi E Rdxx x is the weight matrix of the DNN at layer I. Let M, nxd denote the output of hidden layer I E (1,2.....hh hh of the StoNet; that is,
Mr V (M(M))))) 1 1= 1,,,,,...,
(A19)
M, V (M,, V).
Consider the case that the activation functions bounded, such as sigmoid and tanh. Then the matrix E(M, M,)) has the eigenvalues upper bounded by R6maax for some constant < &'max < co.
For the case that the activation functions are unbounded, such as ReLU or leaky ReLu, we can employ the layer-normalization method in training. In this case, by Lemma A4.4, the matrix V/ (M g2r 4/ (M,,)) 1) has the eigenvalues upper bounded by 7Kmax. provided that the eigenvalues of the matrix - M1-1 is bounded by n6maa after layernnormalization.
Let M, ((M,--))us Then, by an extension of Ostrowski's theorem, see Theorem 3.2 of Higham and Cheng (1998), the eigenvalues of the matrix MA M, uy (VMM)))) (M,,)) is bounded by
Amax (M, Mn) S 718mmaxxmma
Also, we have Amax (E(V*)) of under the assumption (+). Then, with the use of Lemma A4.4 and the Cauchy-Sehwar inequality, we have
Xmax (EMM) M.)) FimaxTmax + a, + 20, /KmmaxTTaa
as the sample size becomes large. This concludes the proof for condition (12)-(i)).
A4.3 Proof of Theorem 3.2
Proof. First, we note that o². the variance of each component of 0;(E RP), is of the order Oee/n) under the setting of the energy function (A6). By setting o < "'d,,p it is easy to verify that the w-network satisfies Assumption A4.1. Therefore, under the additional regularity conditions given in Assumption A4.2, the proposed stochastic w-network asymptotically equivalent to the original w-network. Furthermore, the stochastic w--neworrk can be trained by training the original wn-ntwork using Algorithm 1.
Next, we prove that the inverse mapping learned through training the stochastic w-network is consistent. For Lemma 3.1, Liang al. (2025) first established the convergence of the weights to w", and subsequently established the weak convergence of the latent variables. Therefore, due to the Markov chain nature of Algorithm 1, suffices to prove that the inverse mapping produced by the StoNet is consistent, assuming the latent variables have been correctly imputed (i.e., the true values of Z,, are known). Once the inverse mapping's 's consistency established, the latent variables will be correctly imputed in the next iteration, owing to the algorithm's 's equation-solving nature, which is ensured by setting e - 0 upon convergence to the zero-energy region. This ensures that algorithm remains in its equilibrium, enabling accurate inference of model uncertainty.
22

---

Under the StoNet setting, estimation of wn particular, given M (m1, m2..... m. E gaxd, solving each of the regressions contributes a parameter estimation error:
(2) involves solving p low-dimensional regressions. In
a2 Th E _ Tr( (T((mmmmm)))) EIEE
d 0
due nr)
)
0
(A20)
nPmin
where ? denotes the ith column of wn (2)) i.e. Wn (2) =
Ep) E Rexds; 6) m
MMTM))-MTT (1) is the OLS estimator for the regression coefficients; and 0 0()) (0 O
Let in (2) (11 ..... 2T RRxxd,, By fundamental property of linear regression, the mean prediction 0. w m(2) m, is consistent with respect to 0 B(O,), provided that d, < n. the m;'s extract all O-relevant information from the input variables, and the w-network has sufficient capacity to establish the linear relationship 0, m; for i 1, ........ n. The equality 0- E(O,) is guaranteed by the setting of é O and by the construction of the energy function, which approaches to zero if and only if the empirical mean 1.2 > O; converges to 0 and the variance of 0 approaches to zero. Furthermore, as shown Liang et al. (2022), the stochastic layer effectively provides sufficient dimension reduction for the imput variables.
........
Finally, we prove that the inverse mapping obtained by Algorithm I during the training of the non- stochastic w-network is also consistent. Let 0; w (x(3))(( (p,)). Let 0,(3) and 0:50 denote the jth element of 0; and 0-_ respectively. From equation (*), have 10)O - 0,101 = (E) m )d m, E,) 3))m,; B (E))² (m:; ((x)))]
(B)|E) {3 m.
(BIEE) (B)))) m. ((2))113)122
(1 11
due
2 nn
V dn + edgdho.:
(A21)
where dg denotes the radius of the parameter space Ô (centered at O), the second inequality fol- lows from Cauchy-Schward incquality, the third inequality follows from the Taylor expansion for L (H,, + v.) (at the point n.),, and the last inequality follows from (A20), condition (12)-(i), and the boundedness of E as stated in Assumption 14..
Substituting o, x V nhdap and ignoring some constant factors in (A21), we obtaín
[ / eepda , S pdn V n
Ell0; 0, 111 S pEE
h
where || |/2 denotes the l norn of vector.
By setting x min p2dg pdj, }. we have Ellē, 01 = o(1), which implies 0 is also consistent
h
with respect to 0* E OO)) Consequently, the inverse mapping 1 Σn= 0 produced by Algorithm I in training the con-stochastic w-network also consistent with respect to 0", This concludes the proof of the theorem.
A5 Additional Numerical Results
Regarding uncertainty quantification, we note that there are two types of uncertainties:
1. Aleatoric uncertainty: This refers to the irreducible noise inherent in the data-generating process. can be modeled as
y f(x: 0) + Ess
23

---

where 6, ~ VO,00²) Estimating the unknown variance o² corresponds to quantifying the aleatoric uncertainty (system random error). This is precisely what we addressed in the last experiment included in our previous rebuttal, where x² = 0.1 was treated as unknown.
2. Epistemic uncertainty:
This refers to the reducible estimation error due to limited data incomplete knowledge of the true model (see, for example, model comparison in Sec- tion A5.8). In classical statistics, confidence intervals quantify epistemic uncertainty: as the dataset size increases, epistemic uncertainty- and thus the width of the confidence denrnal-eecreaees..
In the following section, we demonstrate through different examples that EFI is able to accurately quantity both types of uncertainties. We use the coverage rate as the key metric to quantify epistemic uncertainty. The coverage rate can reach the nominal level only when the parameter estimates are unbiased and the uncertainty estimation is accurate.
A5.1
1-D Poisson Equation
Figure A3 and Figure A4 provide typical trajectories learned for the 1-D Poisson model (13) using the methods: PINN, Dropout, and Bayesian PINN. For the ablation study, we vary the noise standard deviation from 0.01 to 0.1 to further assess the validity and accuracy of the proposed method in uncertainty quantification. The results, presented in Tables AI, A2, and A3, show that under different noise levels, the EFI algorithm consistently achieves a 95% coverage rate for the 95% confidence intervals.
To further investigate the relationship between confidence intervals and epistemic uncertainty, we conducted an additional experiment using two different sample sizes: 20 and 80. The results, presented in Table A4, clearly show that as the sample size increases, the width of the confidence interval decreases while the coverage rate remains consistent. This demonstrates that the confidence interval effectively captures the reducible nature of epistemic uncertainty.
-12
(a) PINN
(b) PINN dropout 0.5%
14
-12
-.4
.14
(c) PINN dropout 1%
(d) PINN dropout 5%
Figure A3: 1-D Poisson model (13): (a) Trajectory learned by PINN (without uncertainties); (b)-(d)
Trajectorics learned by Dropout with different dropout rates.
A5.2 1-D Poisson Equation with f-measurement error
We revisit the same 1-D Poisson model as defined in (13), but now incorporate measurement errors in both and f. Specifically, we consider 4 sensors for and 40 sensors for f. with each sensor
24

---

(a) 0 = 0.05
(b) o. = 0.005
(c) 0 = 0.0005
Figure A4: 1-D Poisson model (13): Typical trajectories learned by Bayesian PINN with different of values.
Table Al: Comparison of different methods for ID-Poisson with o 0.01
Method
MSE
Coverage Rate
CI- Cidtt
PINN (no dropout)
0.000008 (0.000001)
0.2808 (0.02872)
0.002079 (0.000026)
Dropout (0.5%)
0.000160 (0.000037)
1.0000 (0.000000)
0.199798 (0.004135)
Dropout (1%) ())
0.000676 (0.000082)
1.0000 (0.000000)
0.276534 (0.006642)
Dropout (5%)
0.009080 (0.001516)
0.9639 (0.011218)
0.593541
(0.025812)
Bayesian (ay = 0.05)
0.000181 (0.000025)
0.9984 (0.000400)
0,254861 (0.001554)
Bayesian (ay = 0.005)
0.000007 (0.000001)
0.9915 (0.003220)
0.028386 (0.000201)
Bayesian (oy 0.0005)
0,000007 (0.000001)
0.9544 (0.013751)
0.010437 (0.000046)
EFI
0.000007 (0.000001)
0.9423 (0.015229)
0.010759 (0.000117)
recording 10 replicate measurements. Measurement errors are modeled as =/" ~ (O, 0.05²) and z! (0, 0.05*), , and we simulate 100 independent datasets under this setting. The experimental results are summarized in Table A.5.
For the Bayesian PINN (B-PINN) method (Yang et al., 2021), we set o. = as 0.05, ensuring the likelihood function is correctly specified. However, as shown in Table A5, B-PINN produces excessively wide confidence intervals, resulting in inflated and inaccurate coverage rate. This highlights another issue inherent to Bayesian DNNs, as noted in Liang et al. (2018b); Sun et al. be significantly affected by the choice of prior in small-n-large-p
(2022): their performance settings. Here, p refers to the number of parameters the DNN used to approximate the solution u(x). Similarly, the dropout method continues produce overly wide confidence intervals and inflated coverage rates.
In contrast, EFI achieves coverage rate of 94.88% with the smallest confidence interval width. Notably, this experiment involves noise both u and f observations. To evaluate the imputed errors, the QQ-plot of 2nd and &/ across 100 experiments is shown in Figure A5. The Q-Q plot confirms that the distribution of imputed errors closely follows its theoretical distribution, supporting the validity of the EFI approach in handling measurement noise from different sources.
Table A2: Comparison of different methods for ID-Poisson with o = 0.025
Method
MSE
Coverage Rate
CI-Widih
PINN (no dropout)
0.000046 (0.000006)
0.1529 (0.019175)
0.002110 (0.000025)
Dropout (0.5%)
0.000157 (0.000028)
1.0000 (0.000000)
0.195658 (0.003892)
Dropout (1%)
0.000666 (0.000084)
1.0000 (0.000000)
0.267936 (0.005751)
Dropout (5%)
0.004573 (0.000654)
0.9979 (0.001157)
0.643227 (0.031514)
Bayesian (a, = 0.05)
0.000161 (0.000019)
0.9992 (0.000307)
0.257184 (0.001612)
Bayesian (a, = 0.005)
0,000060 (0.000013)
0.9630 (0.012690)
0.037341
(0.000418)
Bayesian (o, 0.0005)
0.000090 (0.000014)
0.8504 (0.023270)
0.024676
(0.000366)
EFI
0.000048 (0.000006)
0.9577 (0.014487)
0.027845 (0.000115)
25

---

Table A3: Comparison of different methods for ID-Poisson with 0.1
Method
MSE
Coverage Rate
CI-Width
PINN (no dropout)
0.000581 (0.000066)
0.0320 (0.006518)
0.002088 (0.000024)
Dropout (0.5%)
0.000935 (0.000106)
0.9905 (0.002706)
0.196226 (0.003748)
Dropout (1%)
0.001413 (0.000175)
0.9976 (0.001929)
0.274188 (0.007653)
Dropout (5%)
0.006840 (0.001027)
0.9955 (0.002439)
0.658607 (0.031770)
Bayesian (0, 0.05)
0.000966 (0.000106)
0.9936 (0.002068)
0.285660 (0.001405)
Bayesian (a, 0.005)
0.000674 (0.000080)
0.9582 (0.0150144)
0.104743 (0.000325)
Bayesian (a, = 0.0005)
0.004944 (0.000788)
0.3827 (0.030245)
0.066296 (0.002515)
EFI
0.000624 (0.000058)
0.9493 (0.013526)
0.099543 (0.000775)
Table A4: Numerical results for ID-Poisson with = 0.1, where the confidence interval width shrinks as sample size increases. The results are computed based on 100 independent simulations.
Method
Th (sample size)
MSE
Coverage Rate
CI-Width
EFI
20
0.000624 (0.000058)
0.9493 (0.013526)
0.099543 (0.000775)
EFI
80
0.000173 (0.000017)
0.9501 (0.013459)
0.054256 (0.000723)
Table A5: Comparison of different methods for the 1-D Poisson model (13) (with f f-measurement error), averaged over 100 runs.
Method hidden layers
MSE
Coverage Rate
CI- CI-Width
PINN
[50, 50]
0.000271 (0.000019)
0.0921 (0.008076)
0.003625 (0.000085)
Dropout (0.5%)
[50, 50]
0.000310 (0.000022)
1.0000 (0.000000)
0.240585 (0.002282)
Dropout (1.0%)
[50, 50]
0.000530 (0.000046)
1,0000 (0.000000)
0.357782 (0.005141)
Dropout (5.0%)
[50, 50]
0.003527 (0.000193)
1.0000 (0.000000)
0.669799 (0.006158)
Bayesian
[50, 50]
0.000233 (0.000017)
0.9960 (0.002035)
0.086291 (0.000068)
EFI
[50, 501
0.000238 (0.000017)
0.9488 (0.009419)
0.060269 (0.000365)
AS.3
Computating Time
Table A6 reports the wall-clock time for the Poisson-1D experiment with different algorithms. As shown by the table, EFI is slower than PINN (with dropout) but much faster than B-PINN (with the HMC sampler). Importantly, we have demonstrated that EFI is the only existing method capable of correctly constructing confidence intervals in statistically rigorous manner. For larger networks, we can adopt transfer learning techniques by constructing EFI hyper-networks only for the last few layers of the f-network, which would significantly reduce the computational cost.
A5.4 Non-linear Poisson Equation
We extend our study to a non-linees Poisson equation given by:
8u u dx
8
+ k (tah(u)) = S, EEΩ
(A22)
k 0.00, where S2 - (-07, 0.01, scenario, we sensors located x (-0.7, -0.47, 0.47, 0.73 to provide noisy observations of the solution u. Additionally, we employ 40 sensors, equally spaced within 0.7], to measure f. Both and measurements are assumed to contain noise. In the simulation. measurement errors are modeled as 2/ ~ N (0, 0.05²) for i = 1, ...,.... 40, with each solution sensor providing 10 replicate measurements, and 2/ (0,, 0.05?) for i 1,2, ... 400.
0.7, u sin and f can be derived from (A22). For this
The experimental results are summarized in Table A7. The findings exhibit a similar pattern to those in Table A5: The Bayesian and dropout methods yield inflated coverage rates and overly wide confidence intervals, whereas the EFI method achieves an accurate coverage rate and the narrowest confidence interval. For a fair comparison, we exclude cases where B-PINN converged to incorrect
26

---

COPPIN'
14
44
0.3
Tranbal
(a) QQ-plot imputed errors in u (b) QQ-plot of imputed errors f
(c) Confidence interval
Figure A5: EFI-PINN diagnostic for the 1-D Poisson model (13) (with f f-measurement error)
Table A6: Wall-clock time for PINN, B-PINN and EFI-PINN
Algorithm
Hypernetwork
Epoch (x 1x
Wall-clock time (s)
Time per epoch (ms)
PINN (dropout)
200
133
0.665
B-PINN
100
1330
13.300
EFI
[16.16.16]
200
391
1 1.955
EFI
[16,16,4]
200
385
1.925
solutions, these represent instances of failure in the optimization process, Figure A6 for an instance.
Table A7: Comparison of different methods for the nonlinear 1-D Poisson model (A22) (with f-measurement error), averaged over 100 runs.
Method
hidden layers
MSE
Coverage Rate
CI-Width
PINN
[50,
0.000507 (0.000044)
0.0947 (0.007564)
0.005154 (0.000565)
Dropout (0.5%)
[50,
0.001050 (00030))3)
0.9962 (0.002068)
0.246367 (0.002182)
Dropout (1%)
[50,
0.002807 (0.000309)
0.9861 (0.003829)
0.358088 (0.005178)
Dropout (5%)
[50,
0.007010 (0.000394)))
0.9956 (0.001566)
0.543565 (0.003096)
Bayesian (unstable removed)
[50, 50]
0.000376 (0.000033)
0.9938 (0.002618)
0.104673 (0.000394)
EFI
[50, 50]
0.000385 (0.000039)
0.9483 (0.009191)
0.099880 (0.002853)
AS.5 Non-linear Poisson Inverse Problem
In this section, we consider the same non-linear Poisson equation as in (A22), but with &: = 0.7 treated unknown parameter to be estimated. For this setup, we utilize 8 sensors, evenly distributed across Ω = [-0.7, 0.7] to measure u, with measurement noise modeled as 2," No 0.05²). Additionally, 200 sensors employed measure f, and these measurements assumed to be noise-free.
To apply EFI framework to inverse problem, we extend the output of the w-network by adding an additional dimension dedicated estimating l;, as depicted Figure A7. This modification enables the EFI framework to simultaneously estimate the solution and the parameter k; along with their respective uncertainties. The results are presented in Table A8, demonstrating the capability of EFI to provide accurate uncertainty quantification for both and k. In contrast, B-PINN consistently produces excessively large confidence intervals for both the solution u and the parameter k. Notably, the confidence interval for & estimated by B-PINN is approximately twice as wide that produced by EFI, indicating a significant overestimation of uncertainty. This highlights the superior precision and robustness of the EFI framework in inverse problems.
For this problem, we also tested EFI with larger data modeling network, consisting of two hidden layers and each hidden layer consisting of 100 hidden units. As expected, EFI produced results similar to those obtained with much smaller data modeling network. As noted earlier, EFI can accurately quantify model uncertainty long as the random errors correctly imputed and the
27

---

15
15
Meat 47% EI
63
-15
-14
-14
-38
-0.1
(a) Success
(b) Failure
Figure A6: Successful and failed optimization results of Bayesian PINN for the nonlinear I-D Poisson
model (A22) (with /-measuremeet error)
e notwwakk
PDE Loss
0g
Figure A7: Diagram of EFI for inverse problems, where the orange links indicates the contribution of i to both 0 9-network and k estimation.
inverse function is consistently estimated. This capability is independent of the specific configurations of the w-network and the data modeling network, highlighting EFI's flexibility and robustness.
Table A8: Comparison of different methods for the nonlinear I-D Poisson model (A22) with parameter estimation, averaged over 100 runs.
layers
MSB(xx
Coverage
CI-Width
k-Coverage
k-Cl-Width
PINN
[S0, 50)
1.79 (0.13)
1464 (0.0097)
0.0045 (0.0001)
0.6998 (0.0006)
0.00 (0.0000)
7.9e-5 (5c-6)
Dropout (0.5%
[50,
(0.0000)
0.2393
(0.0002)
Dropout (1 (1.0%)
[50, 50)
2.72 (0.24)
1.0000 (0.0000)
0.3032 (0.0071)
0.6874 (0.0088)
0.11 (0.0314)
0.0091 (0.0007)
Dropout (5.0%)
[50. 50)
27.08 (2.12)
1,0000 (0.0000)
0.5363 (0.0131)
0.6493 (0.0023)
0.06 (0.0239)
0.0252 (0.0014)
Bayesian
[50.
(0.08)
(0.0055)
(0.0000)
0.0411
EFI
(50,50)
1.03 (0.08)
0.9473 (0.0099)
0.0396 (0,0002)
0.6985 (0.0004)
0.94 (0.0239)
0.0179 (0.0002)
EFI
[100, 100)
0.98 (0.07)
0.9560 (0.0099)
0.0395 (0.0002)
0.6995 (0.0004)
0.96 (0.0197)
0.0168 (0,0002)
A5.6
Poisson equation with unknown noise standard deviation
We now consider the where the noise standard deviation is treated as unknown parameter. In this setting, the variability in the observations due to the inherent randomness of the data- gentratnnn process-often referred to systematic error-can be interpreted aleatoric uncertainty.
As shown in Table A9, the EFI algorithm successfully recovers accurate estimates for both the solution and the noise standard deviation accompanied by well-calibrate confidence intervals.
28

---

Table A9: Numerical results for ID-Poisson with unknown 0.1 and sample size nh 60, where the number in the parentheses represents the standard error of the estimator.
Method
MSE
Coverage Rate
CI-Width
c-CR
o-Cl-Widil
EFI
0.000220 (0.000255)
0.9559 (0.0143449) 0.061347 (8.000848)
0.097269 (0.001158)
0.9400 (0.023868)
0.056768 (0.001284))
A5.7 Black-Scholes Model
As practical example, we consider the classical option pricing model in finance the Black-Scholes model (Black and Scholes, 1973):
0v
00
0
2 5²
rV 0, C(O,t) for all 1,
+ +rS
052
05
(A23)
((S,)) 5 - K as S + co, C(S max K,O),
which describes the price V(S,t) of an option. Here, S is the price of the underlying asset (e.g., stock), is time, a represents the volatility of the asset, r is the risk-free interest rate, is the strike price, and T is the expiration time of the option. The boundary conditions reflect specific financial constraints.
This model has been widely used to calculate the price of European call and put options. Specifically, the analytic solution for the call option price C(S,, t) is given by
C(S,,t) t) (dd+)S - +bdd) Ke-((T-)) 1 In
St K
0² 0
d4 oVT
+
r
(T (T- 2
(A24)
d= d+ -v,
where (--) denotes the standard normal cumulative distribution function. However, the uncertainty of the model has not yet been well studied in the literature. Accurately quantifying model uncertainty can significantly benefit opeision-making providing investors with scientific foundation for making safer and more informed choices.
In this simulation experiment, we set T = 1, 0.5, 0.05 and K = The domain is
defined on Ω = [0, T x [0, Smaxk] where Smax = 2. We assume the availability of 5 sensors for the price levels S {0.2, 0.4, 0.6, 0.8, 1.0). with each sensor providing 10 replicate measurements. Measurement errors are modeled as 2; ~ N (0, 0.05²) for = 1.... 50, representing noisy observations. For the boundaries {S = 0} and (1 = T}, we 50 sensors with noise-free measurements. For physical domain, we randomly pick 800 points from Ω to satisfy the Black- Scholes equation. The results of the simulation are presented in Table A10, where the metries are evaluated at f = 0 and t = 0.5. At t = O, the evaluation reflects the model's performance using noisy observed data, while at & 0.5, the solutions are extended from the boundaries using the Black-Scholes equation. This setup highlights the model's 's ability to handle noisy observations and accurately propagate solutions over time through the governing equation. EFI demonstrates superior performance by providing not only the most accurate solutions, as evidenced by the lowest MSE, but also the most reliable confidence intervals.
Table 110: Comparison of different methods for the SackleSholes Model, averaged over 100 runs: "CR' refers to the coverage rate with nominal value of 95%.
Mothod
hadden layers
MSE((=IXX 10-
CX((=l))
C1-Widh(t-0)
MSE((+0.5)(x110 x110
CRI(=0.5)
CL (1-W))
PINN
[50, 30)
3.08 (0.44)
0.1410 (0.0102)
0.0016 vo 0002)
15.33 (2.26)
0.2427 (0.0192)
0.0080 (0 (0.00)
Dropout (0155%)
015897 (0.0216)
00150 (QLOIE))
2.39 (0.97)
(0.0252)
0.157 (100014))
Dmpout (1.05)
0.244 (0005))
1.53
(0.0255)
00234 (0
Dropout (5.0%)
150, 50]
(0.70)
0.9137 (0.0122)
0.0338 (0.0014)
1.70 (0.21)
0.9387 (0.0110)
(0L0012)
Boyesian (ry = 0.05)
[50, 50)
1.59 (0.41)
0.9637 (0.0175)
0.516 00006)
12.61 (7.14)
0,9413 (0.0187)
00855 (0.055)
Hovesian ley 0.005)
(50
10.75 (1.46)
0.5437 (0,0255)
0.0383 (000I))
39.39 (8.82)
0.4507 (0.0225)
0.0266 (100110)
FFI
[50,50]
0.38 (0.05)
0.9440 (0.0133)
0.0158 (0,0001)
0.17 (0,02)
0.9600 (0.0082)
0.0123 (00001))
To further illustrate these findings, we visualize the prediction surface Figure A8. The figure reveals
that B-PINN extends the solution poorly toward the edge at S = 2, where no data points are available,
29

---

relying solely on physical laws for extrapolation. In contrast, EFI provides a smoother and more accurate extension. The dropout method performs reasonably well for this example with a dropout rate of 5%; however, its confidence interval remains significantly wider than that of EFI. As previously noted, determining an appropriate dropout rate is not feasible without additional information. Figure A9 highlights EFI's ability to correctly quantify uncertainties. Near the boundary at S 0, where boundary information is available, the confidence interval appropriately narrow, As the stock price S increases, and boundary information becomes scarce, the confidence interval widens, reflecting the growing uncertainty. In comparison, dropout and Bayesian methods fail to capture this behavior accurately. They produce overly broad or inconsistent intervals, particularly near the boundaries and regions with limited data, underscoring their limitations in handling uncertainty quantification for this problem.
Tew Tewto
of on
Ter on ant dot
andt
(a) True price
(b) PINN dropout (5.0%)
sapoeeaa rove
sapoyeaah bg-aod rove
(c) Bayesian PINN (ay 0.05)
EFI
Figure A8: European Call Option Price.
1/5
417
(a) PINN dropout (5.0%)
(b) Bayesian PINN (a, = 0.05)
(c) EFI
Figure A9: European Call Option Price at 0.
A5.8 Real Data: Montroll Growth Model
Consider the Montroll growth model:
dp (t) = p(t) dt
(1 (1-
p(t)) 9
(A25)
C
30

---

where k, C, and & are unknown parameters. We applied this model to published data on the growth of Chinese hamster V79 fibroblast tumor cells (Marusic et al., 1994), which also appears in Rodrigues (2024). The dataset comprises 45 measurements of tumor volumes (10º vm²) collected over 60-day period. Table A shows the parameter estimates obtained using PINN, and Figure A Al)(a) shows the learned growth curve.
Table All: Parameter estimates obtained with PINN for the model (A25),
Parameter
Estimated Value
k
0.8311
C
7.3327
0
0.1694
234
Tatle voure
312
200
(a) Montroll growth curve learned with PINN
(b) Relationship between & and 0
Figure A10: Montroll growth Model for Chinese hamster V79 fibroblast tumor cells.
However, the standard PINN method does not provide confidence intervals for p(t) or for the parameters k C, and 8. To apply EFI-PINN to this dataset, assume heteroscedastic noise structure given by vt- a,, where is treated as an additional unknown parameter. Thus, we estimate four parameters in total under the EFI framework. Using EFI-PINN, we detected that k and 0 are nearly non-identifiable, see Figure A10(b), which shows their joint distribution by plotting their samples throughout training.
To address this non-identifiability issue, we fix 0 = 0.1 and reduce the model to:
dp
(1 1 p(t)
0.1
dt (t) k p(t)
The corresponding confidence intervals for ptt) are shown in the left plot of Figure 3. The confidence intervals of k, C, and are given in Table A12.
Table A12:
Parameter estimates (with 95% confidence intervals) obtained by EFI-PINN for the Montroll growth model.
Parameter 95% Confidence Interval
Mean
0.1
(1.25, 1.40)
1.25
C
(7.30, 7.69)
7.44
0
(0.27, 0.47)
0.36
The Montroll experiment highlights the strength of EFI in quantifying uncertainty for all parameters of interest. Moreover, it demonstrates EFT's ability to detect model identifiability issues, underscoring its utility in the statistical inference of scientific models. Additionally, EFI produces an estimate of a, which enables the quantification of predictive uncertainty.
31

---

A5,9 Real Data: FKPP Model and Porous-FKPP Model
Consider the Yosber-Kolmooonyy-ervvv------- Piskuuoo (FKPP) model and the porous FKPP (P- FKPP) model, which are govemed by the following reaction-diffusion equations:
du
D 8uu dx
1
+
(A26)
Öt
K
du
u
du
ru (1 u K
D dr
(A27)
0t
K
dx
where D, r, and m are unknown parameters, and K denotes the carrying capacity. This equation has been used to model wide range of growth and transport of biological processes. We applied it to scratch assay data (Jin and Cai, 2006). The biological experiments were conducted under varying initial cell densities specifically. 10,000, 12,000, 14,000, 16,000, 18,000, and 20,000 cells per well. Cell densities were recorded 37 equally-spaced spatial positions across five equally-spaced time points specifically, 0.0 days, 0.5 days, 1.0 days, 1.5 days, and 2.0 days. See also Lagergren et al. (2020) for additional descriptions the dataset. addition to the dataset, we partitioned the space-time domain [0, 2] [0, where the first interval corresponds to the spatial domain and the second to the temporal domain, into a x 10 grid for computing the PDE loss (i.e., the f-term in equation (10)). The fitting curves of the EFI algorithm for different models and initial density values are shown in Figures All and 122. Notably, beyond point estimation, EFI also constructs confidence intervals. Additionally, parameter setting different from that listed in Table A20. In this setting, fewer sample points were allocated for computing the PDE loss, which resulted in smaller fitting errors but larger deviations from the assumed PDE model. Essentially, the two settings correspond to different datasets, as the number sample points used to evaluate the energy function differs between them.
note that the right plot of Figure 3 generated using
In Table A13, the root mean squared errors (RMSEs) are computed between the predicted solution and the observed data, reflecting a combination of epistemic and aleatoric uncertainty. Based on the model formulations in (A26) and (A27), the P-FKPP model is more flexible and therefore expected to exhibit reduced epistemic uncertainty, leading to smaller RMSE values. Consistent with this expectation, Table A13 shows that the P-FKPP model achieves lower RMSEs compared to the standard FKPP model.
Regarding parameter uncertainty, we note that EFI able to quantify the uncertainty associated each parameter. However, due to the transformation applied to the first term of (A27), the values of D are no longer on the same scale across the two models, whereas the values of r remain comparable in scale. The results are reported in Table A13.
Summary Through both simulation and real data experiments, we have demonstrated that the proposed EFI algorithm effectively quantifies uncertainties associated with the model and the data- generating process, resulting in accurate estimation of both epistemic and aleatoric uncertainties.
Table A13: RMSE and estimated parameters with 95% confidence intervals for FKPP and Porous- FKPP models.
Model
Initial Cell Density
RMSE
D
Interval
Interval
Interval
FKPP
10000
58.04
0.00936
(0.00754, 0.01 195)
0.829
(0.797, 0.877)
FKPP
12000
82.09
0.00378
(0.00281, 0.00461)
0.632
(0.603, 0.658)
FKPP
14000
82.93
0.02929
(0.02739, 0.03268)
0.534
(0.505, 0,585)
НКРP
16000
99.14
0.02636
(0.02503, 0.02789)
0.608
(0.585, 0.633)
FKPP
18000
115.27
0.037844
(0.03541, 0.04032)
0.549
(0.524, 0.575)
FKPP
20000
136.67
0.05471
(0.05007, 0.05817)
0.492
(0.458, 0.520)
P-FKPP
10000
46.90
1167.65 67
(72.48, 2719.97 97)
0.846
(0.832, 0.856)
1.335
(1.037,
1.490)
P-FKPP
12000
67.88
(32.84, 4416.53)
0.674
(0.649, 0.696)
1.433
(1.033, 1.603)
P.FKPP
14000
73.59
289.37
(79.08, 532.13)
0.625
(0.600, 0.649)
1.096
(0.951, 1.199)
P-FKPP
16000
70.83
$7.36
(19.21, 97.22)
0.628
(0.608, 0.650)
0.920
(0.804, 0.999)
P-FKPP
18000
96.50
21.58
(9.07, 35.78)
0.563
(0.536, 0,587)
0.780
(0.683, 0.863)
P-FKPP
20000
123.34
1.472
(1.058,20020)
0.496
(0.464, 0.530)
0.408
(0.370, 0.452)
32

---

1000
118
-22
2.5
(a) Initial density 10,000
(b) Initial density 12,000
1240
3040
509
000
2..
3.5
475
252
430
3.5
1.50
2.5
4.5
(c) Initial density 14,000
(d) Initial density 16,000
11dmm
215
pnd
(e) Initial density 18,000
(f) Initial density 20,000
Figure A11: FKPP model
A6 Experimental Settings
In all the simulation experiments, we begin by generating data from specific physics model. Using the simulated data, we iteratively run the algorithm estimate model parameters. To enhance convergence of Algorithm 1, some algorithmic parameters (such as learning rate, SGD momentum, 1/e, and etc.) are adjusted during the initial iterations, referred to as annealing period. To tune different parameters, polynomial. Their specific forms are Table A14. For the nonlinear Poisson inverse problem, both the DNN parameters & and the unknown parameter & are estimated; while for all other problems, only the DNN parameters & are estimated. At the end the simulation, the samples collected in the ourn-in period discarded, and the samples collected in the remaining iterations used for inference. The burn-in period set to be at least as long as the annealing period across all experiments.
three different annealing schemes, including linear, exponential and
our simulations, to ensure the weights the w--etwork remain within a compact space as required in Assumption 14.1-(i), we impose Gaussian prior, N(0, 100,, on each connection weight. However, due to the large variance, this prior has minimal impact on the algorithm's performance, serving primarily to ensure its stability.
33

---

240
120
6.5
1.25
3.50
375
320
0.55
8.50
1.25
1.5
(a) Initial density 10,000
(b) Initial density 12,000
00 Sere
675
0.55
1.25
3.50
520
0.50
3733
3.55
1.50
(c) Initial density 14,000
(d) Initial density 16,000
C35 Scatee
33001
312
Paution min
Post hamo
(c) Initial density 18,000
(I) Initial density 20,000
Figure A12: Porous-FKPP model
Table A14: Notations
Notation
Meaning
epochs
total number of sampling/aptiiiizaiion iterations the proportion total iterations allocated to the burn-in process
burn-in period
annealing period
the proportion of total iterations allocated for parameter adjustment to enhance convergence
linear_x__ (with progress p E [0, 1))
x
x)p
exp_x_y (with progress E [0. 1])
poly_x_c_k (with progress [0, 1)
Tntepy
34

---

Table A15: Parameter settings for ID-Poisson
Parameter Name
PINN
Dropout
Bayesian PINN
FFI
tunn
-0.7
-0.7
-0.7
-0.7
fed
0.7
0.7
0.7
0.7
noise sd in noise sd in of solution of solution replicates
0.05
0.05
0.05
0.05
0.0
0.0
0.0
10
10
10
# of differential sensors
200
200
200
200
# of diffcrential replicates
epochs
500000
500000
100000
200000
bum-in period
0.5
0.5
0,4
0.1
hidden layers
50, 50]
[50, 50]
[50, 50
[50, 50)
activation function
tanh
tanh
tanh
lcaming rate
3c-4
3c-4
lo-4/le-//incar 1c-4 30-6
poly_5c-6 5 S0-6__00.00_0000000000 100.0_0.55
f
1.0
1.0
1.0
dropout rate
/
0.5%
L
Ø/
0.05/exp_0ar 0.05 0.005/exp_0ar 0.05_0.0000
0.05
anneuling period
0.3
0.1
sgd momentum
linear_ (inear_0.9_11 0.9_0.0
sgld learning rate
poly
55-6 poly_5e-66
10.0_0.55
lincar 50.0_5000 55000
encoder hidden layers
[16. 16, 16]
encoder activation function
leaky relu
Table A16: Parameter settings for ID-Poisson (with f error)
Parameter Name
PINN
Dropout
Bayesian PINN
EFI
taut
-0.7
-0.7
-0.7
-0.7
land
0.7
0.7
0.7
0.7
noise sd in
0.05
0.05
0.05
0.05
noise sd in f
0.05
0.05
0.05
0.05
# of solution sensors
4
4
4
# solution replicates
10
10
10
# of differential sensors
40
40
40
40
# of differential replicates
10
10
10
10
epochs
100000
100000
50000
200000
burn-in period
0.5
0.5
0.4
0.1
hidden layers
[50, 50]
[50, 50]
[50, 50]
[50, 50]
activation function learning rate
tanh
tanh
tanh
tanh
3e-4
3e-4
le-4
poly 10ay__25e-6_500 5050_0.55
1.0
1.0
1.0 /
dropout rate
/
0.5%
oy
0.05
0.05
annealing period
0.1
sgd momentum
linear 0.9 0.0
0.0
sgid learning rate
poly 5e-6 100.0 0.55
X
linear 50.0 1000.0 1.0 664, 64, 16)
encoder hidden layers
encoder activation function
Icaky relu
35

---

Table A17: Parameter settings for nonlinear ID-Poisson (with f error)
Parameter Name
PINN
Dropout
Bayesian PINN
EFI
Luan
-0.7
-0.7
-0.7
-0.7
fend
0.7
0.7
0.7
0.7
noise sd in of solution sensors
0.05
0.05
0.05
0.05
noise in
0.05
0.05
0.05
0.05
4
4
# of solution replicates
10
10
10
10
# of differential sensors
40
40
40
40
of differential replicates
10
10
10
10
0.7
0.7
epochs
100000
100000
100000
200000
burn-in period
0.4
0.1
hidden layers
[50, 50)
[50, 50]
50, 50]
[50, 50)
activation function
lanh
tanh
tanh
tanh
learning rate
3e-4
3e-4
le-4
poly 5e-6
100.0_0000000 0.55
f dropout rate L
1.0
0.5% %
6
0,
exp_0.2 0.2.0.055
du
0.05
annealing period
0.3
0.1
sgd momentum
linear_O2. 00._0000000 0.0
sgld learning rate
poly_ 10lly__5-___111 100.0 105.0_0.55
0.55
exp ex._ 50.0 1000000000
N/e
1.0 664, 64, 16)
encoder hidden layers
encoder activation function
Icaky relu
Table A18: Parameter settings for nonlinear ID- ID-Poisson with parameter estimation
Parameter Name
PINN
Dropout
Bayesian PINN
EFT
Lann
-0.7
-0.7
-0.7
-0.7
Lend
0.7
0.7
0.7
0.7
noise sd in of solution sensors
0.05
0.05
0.05
0.05
noise in
0.0
0.0
0.0
0.0
8
8
# of solution replicates
10
10
10
10
# of differential sensors
200
200
200
200
of differential replicates
1
1
0.7
0.7
epochs
50000
500000
50000
350000
burn-in period
0.4
0.1
hidden layers
[50, 50]
[50, 50]
500 50]
[50, 50]
activation function
tanh
tanh
tanh
tanh
learning rate
3e-4
3e-4
le-4
poly 5e-6 1...100..000000 0.55
!!/ dropout rate L
1.0
0.5% %/55%
10
0,
0.05
d..
0.05
annealing period
0.1
sgd momentum
linear 0.9 0.0
sgld learning rate
poly_5e 1os__5e66_100000
1...0000..55
1inear_50.0. 50.0_1000.0
N]e
1.0
encoder hidden layers
[128, 128, 12]
encoder activation function
leaky relu
36

---

Table A19; Parameter settings for Black-Scholes Model
Parameter Name
PINN
Dropout
Bayesian PINN
S range
0..0, 2.0
0.0,2.0 2.0
0.0,2.0 2.0
(0.0, 2.0
range
[0.0, 1.U]
0.0, 1.0
0.00 1.0
[0.0. 1.0)
0.5
0.5
0.5
0.5
r
0.05
0.05
0.05
0.05
1.0
1.0
1.0
1.0
noise sd
0.05
0.05
0.05
0.05
# of price sensors
5
5
f of price replicates
10
10
houndary samples
50
differential samples
800
800
epochs
200000
200000
100000
300000
burn-in period
0.5
0.5
0.4
0.1
hidden layers
[50, 50)
[50, 50]
[50, 50]
[50, 50
activation function
softplus( = 5)
softplus(13 = 5)
softplus(13 = 5)
softplus(3 = 10)
learning rate
3e-4
3e-4
le-4/linear le-4_le.5
poly_ moly_5e-6. Saly_5e-6_100mee 100.0_0.55
1.0
1.0
1.0
dropout rate
0.5%
L
0.05/exp_00 10e_00050.00000
du
0.05
pretrain epochs
5000
annealing period
sgd momentum
linear_0.9 _0.9_000000
sgld learning rate
sgld alpha
1.0
A
linear
y)
1.0
encoder hidden layers
[64, 64, 16]
encoder activation function
leaky relu
Table A20: Parameter settings for real data
Parameter Name
Montroll growth
FKPP
P-FKPP
epochs
200000
200000
200000
burn-in period
0.1
0.1
0.1
hidden layers
[50,50]
[50, 50] tanh
[50, 50) tanh
activation function
softplus(B 10)
learning rate
1oly_le--6110000..5 10.0_0.55
poly 10.0_..55
10.0..55..
?/
1.0
1.0
1.0
sgd momentum
0.9
0.9
0.9
sgld learning rate
poly [ee4__10.00.000 0.95
poly le___e_6____ 111___l--610.0000. 10.0_0.95
Soly_le-6_10.0_10
sgld alpha
1.0
1.0
1.0
log_500 50.00500.0
log 50.0_500.0
rrg_50.0550m0
7/
1.0
1.0
1.0
encoder hidden layers
[32, 32, 16]
(64, 64, 16]
[64, 64, 16)
encoder activation function
leaky relu
leaky relu
leaky relu
37


Reviews:
[{"review_id": "CF1zzSd2Ez", "reviewer": "Reviewer_qVGg", "summary": "This paper proposes using extended fiducial inference (EFI) for uncertainty quantification for physics-informed neural networks (PINNs). This method does not rely on assuming a parameter prior, which can be very challenging for neural networks. This is unlike commonly applied methods such as Monte Carlo Dropout (MCDropout) and Hamiltonian Monte Carlo (HMC) methods. These methods are compared on simulated Poisson distribution data, and the results indicate that the confidence intervals produced by EFI outperform MCDropout and HMC. EFI is also evaluated on for two PINNs.", "questions": "MCDropout [1] uses both a Gaussian parameter prior and dropout probability hyperparameters. While poor priors limiting the effectiveness of Bayesian neural network methods was stated as a reason to use EFI, the prior variance hyperparmeter was not explored in the simulation experiment for McDropout. Also, the dropout probability hyperparameter can be learned for MCDropout using Concrete Dropout [2], although setting the prior variance hyperparameter could still be an issue for Concrete Dropout. Showing that EFI performs better than Concrete Dropout with varying prior variances would better support the claim that, by not depending on poor priors, EFI could be preferred over Bayesian neural network methods. Additionally, comparing the results of the BNN methods to those of EFI for the evaluated PINNs would further support the authors claims. Other methods, although not as popular as MCDropout or HMC, that could be compared to are Bayesian methods that, like EFI, use a double neural network approach [3,4,5]. Comparing to one of those would strengthen the paper.\n\n[1] Gal, Yarin, and Zoubin Ghahramani. \"Dropout as a bayesian approximation: Representing model uncertainty in deep learning.\" International Conference on Machine Learning. PMLR, 2016.\n\n[2] Gal, Yarin, Jiri Hron, and Alex Kendall. \"Concrete dropout.\" Advances in Neural Information Processing Systems 30 (2017).\n\n[3] Louizos, Christos, and Max Welling. \"Multiplicative normalizing flows for variational bayesian neural networks.\" International Conference on Machine Learning. PMLR, 2017.\n\n[4] Krueger, David  et al. \"Bayesian Hypernetworks.\" https://arxiv.org/pdf/1711.01297.\n\n[5] Pawlowski, Nick et al. \"Implicit Weight Uncertainty in Neural Networks.\" https://arxiv.org/pdf/1711.01297.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality: The proposed EFI method appears to be technically sound, although I am not an expert on fiducial inference methods. However, the evaluations should be significantly improved to support the claims of the authors.\n\nClarity: The paper seems to clearly presents the proposed methods and how they differ from previously proposed approaches.\n\nSignificance: High-quality uncertainty quantification methods for PINNs are not only of interest to the machine learning community but also the other scientific community, such as physics and biology. \n\nOriginality: The proposed method is an alternative to Bayesian neural networks, the current primary approach for achieving uncertainty quantification. In cases where parameter priors are not known, the proposed method could significantly improve upon those approaches in a novel way.", "was_revised": true, "final_justification": "The paper appears to cover a novel approach to uncertainty estimation in deep neural networks. For the task of estimating confidence intervals of the predicted mean of the output variable in PINNs, the approach appears to perform better than common Bayesian neural networks approaches. However, the method uses a double neural network approach and double neural network Bayesian methods are not evaluated. From the current results and discussions with the authors, I do not expect those Bayesian methods to outperform the proposed approach. I think that this would be the case due to Bayesian methods learning parameter variances related to epistemic uncertainty, while EFI learns parameter variances by focusing on prediction error. With the inclusion of a discussion on comparing the parameter uncertainty of EFI with that of Bayesian approaches, the paper would better position the method within the literature. With the addition of such a section, I would be willing to raise my score to a 5. Although, my confidence remains a 2, due to my unfamiliarity with fiducial inference.", "rebuttal": "## Q1: Concrete Dropout.  \nConsider Concrete Dropout by Gal et al. (2017). Unlike the traditional dropout method using fixed dropout rate, Concrete Dropout treats the dropout rate as a hyperparameter and learns it simultaneously when training the model. \n\n**Table 1** Metrics for 1D-Poisson with Concrete Dropout  \n\n| Method              | MSE                  | Coverage Rate        | CI-Width              |\n|---------------------|----------------------|----------------------|-----------------------|\n| PINN (no dropout)   | 0.000121 (0.000012)  | 0.0757 (0.011795)    | 0.002139 (0.000024)   |\n| Concrete Dropout    | 0.000184 (0.000020)  | 0.1185 (0.018002)    | 0.003706 (0.000073)   |\n| EFI                 | 0.000148 (0.000016)  | 0.9517 (0.014186)    | 0.050437 (0.000462)   |\n\n\nWe applied Concrete Dropout with the code provided on https://github.com/yaringal/ConcreteDropout. **Table 1** shows the result of 1-D Poisson model with Concrete Dropout. All necessary algorithm parameters in Concrete Dropout we applied were set as the code provided. \n\nCompared to our proposed EFI algorithm, Concrete Dropout achieves a comparable MSE, indicating similar model estimation accuracy. However, it performs significantly worse in terms of uncertainty quantification, with a coverage rate of only **11.85%**, far below the nominal 95%. This substantial under-coverage is due to an underestimation of predictive uncertainty, as evidenced by the markedly narrower confidence interval width.\n\nFrom another perspective, given that the performance metrics of Concrete Dropout are relatively similar to those of the PINN algorithm without dropout, we suspect that it effectively learns a dropout rate that is quite close to zero.\n\nMore importantly, regardless of whether the learned dropout rate is small or large, the method lacks theoretical justification or principled guidance to ensure that the resulting predictive uncertainty aligns with the true underlying uncertainty. Furthermore, it is not applicable for quantifying uncertainty in model parameters. In contrast, EFI provides a unified framework for quantifying both predictive uncertainty and parameter uncertainty.\n \n## Q2: Other Bayesian methods \nWe also attempted to implement the method proposed in **Implicit Weight Uncertainty in Neural Networks** in the context of PINNs. However, to the best of our knowledge, there is no existing literature that reports such an implementation. Our attempt failed to converge to the desired PDE solution. Due to time constraints, we were unable to thoroughly tune the hyperparameters, although we have tried very hard. In contrast, the existing Bayesian approach, B-PINN, as demonstrated in our paper, has already shown that Bayesian methods fail to construct faithful confidence intervals for PINNs."}, {"review_id": "zhKfHlRtOk", "reviewer": "Reviewer_ZTUj", "summary": "In this paper, the authors propose a more accurate uncertainty quantification method for Physics-Informed Neural Networks (PINNs). Traditional Bayesian approaches require specifying a prior distribution to estimate the posterior, which can be challenging and subjective. Similar limitations apply to dropout-based methods. To address this, the authors leverage Extended Fiducial Inference (EFI)—a method that avoids the need for prior specification. However, the original EFI framework has known limitations in handling high-dimensional model parameters and large sample sizes. To overcome these challenges, the authors develop a new EFI-based framework tailored to PINNs, with accompanying theoretical guarantees. The proposed method is evaluated on both synthetic toy problems and real-world datasets.", "questions": "I am not very familiar with EFI. In line 91, a distribution for the random error is required—does that function similarly to a prior? I don’t think specifying a prior is a major drawback of Bayesian methods. In fact, incorporating informative priors, such as knowledge-based priors, can improve performance (see [1]). Even if the true prior is unknown, using a Gaussian prior with small variance generally does not hurt performance (see [2]).\n\nMore discussion is needed around general uncertainty quantification methods. There are various approaches such as MCMC methods, variational inference, and ensemble-based methods.\n\nI couldn’t find a direct evaluation of uncertainty quality. Instead of relying only on MSE to assess regression accuracy, how can we compare the predicted uncertainty against ground-truth uncertainty?\n\nThe real-world evaluation is not strong. UQ is widely applied in computer vision and has been evaluated on various image classification tasks. The proposed method could also be tested on more challenging real-world tasks to better demonstrate its value. Can the method apply on classification tasks?\n\n[1] Bayesian Neural Networks with Domain Knowledge Priors\n[2] What Are Bayesian Neural Network Posteriors Really Like?", "limitations": "yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well-written and clearly organized, making it easy to follow the methodology and key contributions.\n\nEnabling uncertainty quantification (UQ) for PINNs is a valuable and timely direction, with growing importance in scientific machine learning and real-world applications.\n\nWeaknesses:\n\nThe paper lacks a thorough discussion of related work in general uncertainty quantification, particularly outside the context of PINNs.\n\nThe evaluation of the proposed UQ method is limited—there is no direct assessment of uncertainty quality.\n\nThere is a lack of real-world experiments to demonstrate the practical utility and robustness of the proposed approach.", "was_revised": true, "final_justification": "The rebuttal solved some of my concerns. I will raise the score but lower my confidence.", "rebuttal": "**[W1.] The paper lacks a thorough discussion of related work in general uncertainty quantification, particularly outside the context of PINNs.**\n\nIn recent years, a variety of methods have been developed for uncertainty quantification in machine learning models, such as conformal prediction (Vovk et al., 2005), deep ensembles (Lakshminarayanan et al., 2016), stochastic deep learning (Sun & Liang, 2022; Liang et al., 2022). However, these methods primarily focus on prediction uncertainty and are either ineffective or inapplicable for quantifying uncertainty in model parameters. In contrast, EFI is applicable to both prediction uncertainty and model parameter uncertainty, and it further provides theoretical guarantees for the validity of the resulting prediction and confidence intervals.  \n\n[1] Vovk, V., Gammerman, A., and Shafer, G. Algorithmic Learning in a Random World. Springer, 2005.    \n[2] Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Neural Information Processing Systems, 2016. URL https://api. semanticscholar.org/CorpusID:6294674.    \n[3] Sun, Y. and Liang, F. A kernel-expanded stochastic neural network. Journal of the Royal Statistical Society Series B, 84(2):547–578, 2022.    \n[4] Liang, S., Sun, Y., and Liang, F. Nonlinear sufficient dimension reduction with a stochastic neural network. NeurIPS 2022, 2022.\n\n\n\n**[W2.] The evaluation of the proposed UQ method is limited—there is no direct assessment of uncertainty quality.**\n\nBelow we provide an example, where the length of intervals provides a direct assessment of uncertainty quality. See Q2.\n\n**[W3.] There is a lack of real-world experiments to demonstrate the practical utility and robustness of the proposed approach.**\n\nWe are not sure what you are asking for. In the paper, we have provided some real data examples in the context of PINNs. Regarding real-world  applications of the original EFI method, we refer to Kim & Liang (2025), where EFI was applied to infer the uncertainty of individual treatment effects (with real data examples). Notably, as shown in Kim & Liang (2025), EFI eliminates the need for the central limit theorem in uncertainty inference for complex models, as well as the requirement for specifying the exact form of the target model.  Additionally, for potential applications of EFI on classification problems, see Q3.\n\n[5] Kim, S. and Liang, F. Extended fiducial inference for individual treatment effects via deep neural networks. Statistics and Computing, 35, 2025. URL https://api.semanticscholar.org/CorpusID:278326914.\n\n\n**[Q1.] In line 91, a distribution for the random error is required—does that function similarly to a prior? I don’t think specifying a prior is a major drawback of Bayesian methods. In fact, incorporating informative priors, such as knowledge-based priors, can improve performance. Even if the true prior is unknown, using a Gaussian prior with small variance generally does not hurt performance.**\n\nA distributional assumption for the random error is essential for any likelihood-based approach; without it, the likelihood function cannot be constructed. In Bayesian methods, a prior is specified only for the model parameters. While a well-specified prior—particularly one based on domain knowledge—can enhance statistical analysis, it can also introduce challenges. In small-n-large-p settings, such as Bayesian deep neural networks, an improperly specified prior may overwhelm the information from the data, leading to biased inference. To mitigate this issue, a theoretical justification for posterior consistency is generally required.\n\nIn the context of Bayesian PINNS, the PDEs themselves act as a form of prior, constraining the neural network parameters to lie on a sub-manifold. For instance, consider the 1-D Poisson example in Section 4.1. In Bayesian PINNs, a Gaussian prior has to be introduced to constraint the network parameters 2 to satisfy the required PDE. However, the resulting uncertainty heavily depends on the variance $\\sigma_f^2$ (see Table 1 in manuscript). This indicates the dependence of statistical inference (for the model) on the prior hyperparameters, which is undesirable for a scientific method. In contrast, EFI offers a mathematically rigorous framework for incorporating PDE constraints into neural network training, enabling accurate uncertainty quantification for both predictions and model parameters—without relying on prior distributions.\n\n**[Q2.] More discussion is needed around general uncertainty quantification methods. There are various approaches such as MCMC methods, variational inference, and ensemble-based methods. I couldn’t find a direct evaluation of uncertainty quality. Instead of relying only on MSE to assess regression accuracy, how can we compare the predicted uncertainty against ground-truth uncertainty?**\n\nSee W1 for discussions on general uncertainty quantification methods. We appreciate the feedback regarding the evaluation of our uncertainty quantification (UQ) method. In the context of Physics-Informed Neural Networks (PINNs), the ground-truth uncertainty is inherently unknown, making direct comparison between predicted and true uncertainty challenging. To address this, we adopt a statistical approach by assessing the reliability of predicted uncertainties through confidence intervals and their coverage rates. This is a well-established practice in statistics when ground-truth uncertainty is unavailable.\n\nTo further validate our method, we conducted experiments on simpler problems (e.g., linear regression) where analytical uncertainty estimates (e.g., from Ordinary Least Squares, OLS) are available and serve as the ground truth. While applying our proposed EFI algorithm to such problems may seem excessive, it serves as a sanity check. By comparing EFI-derived confidence intervals with those from OLS, we demonstrate that EFI produces well-calibrtaed uncertainty estimates (nearly same as the ground truth). Below is an example of this verification:\n\nConsider a simple linear regression model:\n$$y_i = \\beta_0  + \\beta_1 x_i + \\epsilon_i,\\quad i=1,\\dots, n$$\nwhere we set $\\beta_0=-1$ and $\\beta_1=2$. \nLinear regression can be reinterpreted within the framework of PINNs as follows:\n$$ y_i^u = u_{\\theta}(x_i^u) + z_i^u, \\quad i=1,2,\\dots,n_u, $$\n$$ 0 = \\frac{d^2}{dx^2} u_{\\theta}(x_i^f), \\quad i=1,2,\\dots,n_f.$$\n\nHere, $ u_\\theta $ is a DNN with no pre-imposed structure, while the assumption of **linearity** is enforced through the differential equation constraint. This formulation seamlessly integrates linear regression into the PINN framework. For EFI, we can derive $\\beta_1$ estimates through automatic differentiation, where $\\beta_1 = \\frac{d}{dx}u_\\theta(x)|_{x=0}$. In table below, we show that EFI obtained almost the same confidence interval as OLS.\n\n**Table:** Confidence intervals for OLS and EFI  \n\n| Algorithm | β₀ (95% CI)  | β₁ (95% CI) |\n|-|-|-|\n| OLS | (-1.12, -0.90)  | (1.88, 2.24) |\n| EFI  | (-1.12, -0.93)  | (1.87, 2.23) |\n\n**[Q3.] UQ is widely applied in computer vision and has been evaluated on various image classification tasks. The proposed method could also be tested on more challenging real-world tasks to better demonstrate its value. Can the method apply on classification tasks?**\n\nAlthough this paper focuses on UQ for PINNs, EFI framework can be extended to classification and logistic regression scenario. Consider a logistic regression:\n$$P(y=1|x, \\theta) = \\frac{1}{1 + \\exp\\{-x^T\\theta\\}}$$\nWe can adopt EFI framework by modifying the energy function as follows:\n$$\\sum_{i=1}^n \\rho((z_i-x_i^T \\theta)(2y_i-1)),$$\nwhere $\\rho(\\cdot)$ is the ReLU function, and \n$z_1,z_2,\\ldots,z_n \\stackrel{iid}{\\sim} Logistic(0,1)$ with the CDF given by $F(z)=1/(1+e^{-z})$. In this experiment, the samples $x_i$ are generated from two 2D Gaussian distributions with means \n$\\mu_1 = (2, 2)$ and $\\mu_0 = (-2, -2)$, and a covariance matrix \n$\\Sigma = 2.56 I_2$. Each class contains 500 samples. The true parameter is set to \n$\\theta = (1, 1)$. We are able to achieve 95% coverage rate on the parameter $\\theta$. We can also incorporate DNN structure to adapt nonlinear decision boundaries:\n$$\\sum_{i=1}^n \\rho((z_i-h(x_i;\\theta))(2y_i-1)),$$\nwhere $h(\\cdot;\\theta)$ is an arbitrary DNN parameterized by $\\theta$. Furthermore, we can extend EFI framework to multiclass classification problem. We define:\n$$\np_{ij} = \\frac{e^{-x_i^T \\theta_j}}{\\sum_{l=1}^{k} e^{-x_i^T \\theta_l}}, \n$$\nThen the fitting error term for multi-class logistic regression can be defined as follows: \n$$\\sum_{i=1}^n \\left[ \n\\rho(u_i- \\sum_{l=1}^{m_i} p_{ij}) + \n\\rho( \\sum_{l=1}^{m_i-1} p_{ij} -u_i) \n\\right],  $$\nwhere $u_i \\sim Unif(0,1)$ denotes a random number, $m_i\\in \\{1,2,\\ldots,k\\}$ denotes the true class of the training sample $x_i$, \n$\\theta_{j}$ denotes the parameters corresponding to class $j$, and \n $x_i^T \\theta_j$ is referred to as the log-odds of sample $x_i$ for class $j$."}, {"review_id": "Mm43UCxFK4", "reviewer": "Reviewer_Uzpp", "summary": "The paper proposes Extended Fiducial Inference (EFI)  to attach calibrated uncertainty estimates to physics-informed neural networks (PINNs).  EFI learns a slim “narrow-neck” hyper-network that maps observed data plus latent noise to the PINN’s weights; by Monte-Carlo sampling the latent noise one obtains predictive intervals whose width automatically adjusts to data quality. The authors evaluate their method on artificial and more real-world physics problems.", "questions": "Eq. (1)\n\nthe uncertainty of z comes from things that can't be measured (noise, partial observability). This can be called aleatoric uncertainty.\n\nHowever, the uncertainty over \\theta  will shrink as more data becomes available, it can be called epistemic.\n\nBut see https://iclr.cc/virtual/2025/poster/31334 for a more critical take on this.\n\nIt can be argued  that fusing the uncertainty from z into w will create epistemic and aleatoric uncertainty.\n\nExample:\nlet y = \\theta_1*x*z  + \\theta_2*x*(1-z) and z from Bernoulli({0,1})\n\\theta and z are latent. As we collect more data in the form of (x,y) pairs ideally the posterior over \n \\theta will shrink its posterior as more data is available (epistemic). Uncertainties over \\theta can best be understood by uncertainty over the slope of the bimodal function. \n\n\nEq. (2) will lead to \\theta containing both the \"slope uncertainty\" as well as the variation that originates from the bimodality. It is thus a mix of epistemic and aleatoric uncertainty.\n\nWhat kind of uncertainty is being captured once EFI pushes the observation noise through G?\nCan we still distinguish whether high uncertainty over \\theta originates from lack of data or from a high noise level? (i.e. are epistemic and aleatoric components recoverable?)", "limitations": "n/a", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts:\n\n- (+) The paper is written very well. Especially Section 1 & 2 are excellent.\n\n- (+) The paper is adressing an important area of research (uncertainty modeling) and the EFI method is an underexplored subfield. \n\n- (+) The experiments make sense given the research question and are evaluated in a clean way. (Some design choices like showing confidence bands, coverage can be questionable: for instance for uncertainty modeling a standard method is to calculate the avg. log-likelihood, ECE etc.). \n- (+/-)  Carried over from EFI: Compared to variational inferences, the uncertainty modeling approach has few free hyper-parameters and does not need explicit prior modeling. However, some typical drawback of MCMC carry over: (algorithm 1: burn-in periods, difficulty to check convergence, sampling process usually slower than energy-loss minimization.  perhaps wall-clock comparison of the algorithm compared to dropout, ensembling and the likes?)\n\n- (-)  It is unclear whether epistemic uncertainty is truly captured by the approach (or if its both epistemic and aleatoric which can't be disentangled, see questions below). There is a  lack of discussion on the epistemic and aleatoric uncertainty concept in Section 2. Also the approach appears to share similarity with the BNN+LV approach (see e.g. Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning). In Appendix A5 these concepts are discussed, but very limited y = f(x) + \\epsilon (normal, uniform noise) is quite limited noise assumption. \nWill the method still work if the noise is heteroscedastic? \nwhat about the most general case, where if y = f(x,z) and z is from some distribution? \nCan you refute the following statement: \"EFI conflates epistemic & aleatoric uncertainty\"?", "was_revised": true, "final_justification": "my initial review still stands. i believe the papers contribution license acceptance, but I'm still cautious regarding practicality and how principled epistemic and aleatoric uncertainty modeling is.", "rebuttal": "**[W1.] Carried over from EFI: Compared to variational inferences, the uncertainty modeling approach has few free hyper-parameters and does not need explicit prior modeling. However, some typical drawback of MCMC carry over: (algorithm 1: burn-in periods, difficulty to check convergence, sampling process usually slower than energy-loss minimization. perhaps wall-clock comparison of the algorithm compared to dropout, ensembling and the likes?)**\n\nIn Table 1, we report the wall time for the Poisson-1D experiment using different algorithms. As shown, EFI is slower than PINN (with dropout) but much faster than B-PINN with the HMC sampler. Importantly, we have demonstrated that EFI is the only existing method capable of correctly constructing confidence intervals in a statistically rigorous manner. Since we employ a narrow-neck hypernetwork structure, the total number of parameters depends linearly on the size of the neck layer. Therefore, EFI is scalable with respect to the size of the $\\theta$-network. For even larger networks, we can adopt transfer learning techniques by constructing EFI hypernetworks only for the last few layers of the $\\theta$-network, which would significantly reduce the computational cost.\n\nIt is worth noting that the EFI training process is based on the Adaptive SGMCMC algorithm, which alternates between one step of SGD and one step of SGLD. Since both SGD and SGLD are stochastic gradient-based algorithms, EFI has the same order of sample complexity as standard optimization-based approaches. However, because EFI aims to construct accurate confidence intervals, it generally requires more training time compared to purely optimization-focused methods.\n\n**Table:** Wall time for PINN, B-PINN and EFI-PINN  \n(Device: MacBook Pro M4 Pro CPU)\n\n| Algorithm       | Hypernetwork   | Epoch (×10³) | Wall time (s) | Time per epoch (ms) |\n|-----------------|----------------|--------------|---------------|---------------------|\n| PINN (dropout)  | -              | 200          | 133           | 0.665               |\n| B-PINN          | -              | 100          | 1330          | 13.300              |\n| EFI             | [16,16,16]     | 200          | 391           | 1.955               |\n| EFI             | [16,16,4]      | 200          | 385           | 1.925               |\n\n\n**[W2.] It is unclear whether epistemic uncertainty is truly captured by the approach (or if it's both epistemic and aleatoric which can't be disentangled, see questions below). There is a lack of discussion on the epistemic and aleatoric uncertainty concept in Section 2. Also the approach appears to share similarity with the BNN+LV approach (see e.g. Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning). In Appendix A5 these concepts are discussed, but very limited $y = f(x) + \\epsilon$ (normal, uniform noise) is quite limited noise assumption. Will the method still work if the noise is heteroscedastic? what about the most general case, where if $y = f(x,z)$ and $z$ is from some distribution? Can you refute the following statement: \"EFI conflates epistemic \\& aleatoric uncertainty\"?**\n\nAs explained below (see our replies to Questions), EFI integrates epistemic \\& aleatoric uncertainty in a mathematically rigorous way. The method still works if the noise is heteroscedastic. In this case, we can set $\\epsilon$ to a very small value, enabling the noise to be recovered. Actually, this has been tested in Liang et al. (2025), where the example in Section 5.4 demonstrates that EFI works with heteroscedatic noise (or outliers).\n\nYes, the method also works in the most general case, where $y=f(x,z)$. Refer to Liang et al. (2025) for the theoretical development on this case, where the random error does not appear in the model in an additive form. \n\n[1] Liang, F., Kim, S., and Sun, Y. Exended fiducial inference: Toward an automated process of statistical inference. Journal of the Royal Statistical Society Series B, 87:98–131, 2025.\n\n**[Questions]**\n\nThank you very much for the thoughtful comments. EFI aims to \nlearn an inverse mapping for the model parameters, i.e., $\\theta=G(Y_n,X_n,Z_n)$, through which the uncertainty of $\\theta$ can be derived from the uncertainty of $Z_n$ (in the form of variable transformation as shown in equation (8) of the paper).  \n\nTo make equation (8) (of the paper) more interpretable in the context of uncertainty quantification, we\ncan rewrite it in a less formal manner as follows: \n$$\n\\mu_n^*(\\theta|Y_n,X_n)= \\int \\pi(\\theta|Y_n,X_n,Z_n) \\pi(Z_n|Y_n,X_n) dZ_n,\n$$\nwhere $\\pi(Z_n|Y_n,X_n)$ provides a measure for the \naleatoric uncertainty, while $\\pi(\\theta|Y_n,X_n,Z_n)$ provides \na measure for the epistemic uncertainty (whose variation can decrease to 0 \nas $n \\to \\infty$) and depends on the aleatoric uncertainty. That is, the uncertainty measured by EFI has integrated both types of uncertainty. \nWe agree with Kirchhof et al. (2025) that distinguishing between these two types of uncertainty is not always feasible and one \ncan depend on the other. \n\nTo better explain this concept, let's consider a linear regression example: \n$$\ny=x\\beta+ Z,\n$$\nwhere $Z$ follows the standard Gaussian distribution.\n As shown in Liang et al. (2025), an  \ninverse function for $\\beta$ is given by \n$$\nG(Y_n,X_n,Z_n)=(X_n^TX_n)^{-1} X_n^T(Y_n-Z_n). \n$$\nIntegrating with respect to the fiducial density of $Z_n$,  the fiducial \ndensity of $\\beta$ is given by  \n$$\nN((X_n^T X_n)^{-1} X_n^T Y_n, (X_n'X_n)^{-1}). \n$$\nThis theoretical result has been numerically verified in Liang et al. (2025) through the implementation of EFI.\n\nNext, let's consider the binary response example you provided. \nGiven a set of $Z$-values, \ndirectly solving the data-generating equation leads to  \n$$\n\\theta|y_i, x, z_i=\\begin{cases} \\frac{y_i}{x} (:=\\theta_{1}), & \\mbox{if $z_i=1$} \\newline\n   \\frac{y_i}{x} (:=\\theta_{2}), & \\mbox{if $z_i=0$}, \\\\\n   \\end{cases}\n$$\nfor $i=1,2,\\ldots,n$. \n\nLet $(z_i^{(1)}: i=1,2,\\ldots,n)$, $\\ldots$, $(z_i^{(m)}: i=1,2,\\ldots,n)$\ndenote $m$ imputations from $\\pi(Z_n|Y_n,X_n)$ in EFI, which measures the aleatoric uncertainty. \nBy empirically integrating out $Z_n$ as what does in EFI, we obtain \n$$\n\\theta=\\begin{cases} \\theta_1, & \\mbox{with prob $\\hat{p}$} \\newline\n        \\theta_2, & \\mbox{with prob $1-\\hat{p}$}, \\\\ \n     \\end{cases} \n$$\n\nwhere $\\hat{p}$ is given by \n$$ \\hat{p}= \\frac{1}{m} \\sum_{k=1}^m \\frac{ \\\\# \\\\{i: z_i^{(k)}=1\\\\} }{n}, $$\nwhose variance decreases to 0 as $n\\to \\infty$. \nAgain, the uncertainty of $\\theta$ has integrated both \nthe aleatoric and epistemic uncertainty. EFI can be implemented for the example by replacing its latent variable sampling algorithm with a Metropolis-based algorithm (dealing with discrete values of Z)."}, {"review_id": "wkVao9JUXt", "reviewer": "Reviewer_4Xoo", "summary": "This paper introduces a novel approach to uncertainty quantification for Physics-Informed Neural Networks (PINNs), based on the recently proposed Extended Fiducial Inference (EFI) framework by Liang et al. (2025). A key advantage of EFI is that it circumvents the potentially misleading specification of a prior distribution, which is often a concern in Bayesian frameworks—particularly in approaches like MC Dropout where even the dropout rate can implicitly act as a prior. The proposed algorithm is empirically validated through several experiments.", "questions": "N/A", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "###Do we still require a prior distribution in EFI?###\n\nWhile I acknowledge that I am not an expert in fiducial inference, I find the concept of EFI quite confusing. Based on Appendix A1, it seems that a prior $\\pi(w_n)$ is still required. If so, how does EFI avoid the pitfalls associated with overly informative priors that could overshadow the observed data? A more transparent discussion of this point would be helpful.\n\n###Why is $\\delta_g = 0$?###\n\nIn the paragraph following Lemma 3.1, the authors claim that with imputation at each step for every observation, we have $\\delta_g = 0$. However, this is not obvious to me. This assumption is crucial since it ensures the Wasserstein-2 distance bound vanishes as both $T$ and $k$ tend to infinity. I request a more detailed and rigorous explanation for why $\\delta_g = 0$ under the described setup.\n\n##Eigenvalue assumption###\n\nThe paper assumes a uniform lower bound on the minimal eigenvalue of $E[m_i m_i^\\top]$. This is a strong assumption, and I would appreciate a simple toy example or practical justification showing when and why this condition would hold in real-world scenarios.\n\n###Interpretation of Theorem 3.2###\n\nTheorem 3.2 states an asymptotic consistency result on the estimation of the inverse function $G$ with finite samples. However, the practical implications of this result are unclear. Can the authors provide a more quantitative version—perhaps a sample or iteration complexity bound that shows how fast the method converges to within an error $\\delta$?\n\n============================\n\nHonestly, I find the paper difficult to review—not necessarily due to its quality, but because it builds on EFI, a relatively new and non-standard inference framework that is challenging to digest within the limited time available for conference reviews. Furthermore, I am not deeply familiar with the PINN literature, which made the model specification in Section 3.2 particularly hard to follow.\n\nGiven these factors, I strongly encourage the authors to consider submitting this paper to a journal venue, where the review process would allow for a more careful and thorough evaluation. For now, I will defer to the Area Chair for a final decision.", "was_revised": true, "final_justification": "After reading the response from the authors and the author's response to other reviewers, I feel my score 4 borderline accept is a fair assessment. Many claims in this paper are very much based on a previous paper on Extended Fiducial Inference (EFI) framework by Liang et al. (2025), for which I would encourage the authors to give a clearer explanation of the related necessary background knowledge on EFI. In this current format, as a reviewer, I have huge difficulty understanding and evaluating the real contribution and potential impact of this paper. This is my major negative perspective of this paper. The rest of the weakness stated in my reviews, like the validity and presentation of the assumption, can be addressed should the authors spend more time polishing the paper. I look forward to see another updated version of this paper.", "rebuttal": "**[W1.] Do we still require a prior distribution in EFI?**\n\nIn Appendix~A1, we reviewed the original EFI theory, which requires imposing a sparse prior on $w$ to guarantee posterior consistency for the hypernetwork. Instead of adopting a Bayesian approach on the hypernetwork, we designed a narrow-neck network architecture to eliminate the need for prior assumptions on $w$.\n\n**[W2.] Why is $\\delta_g = 0$?**\n\nThanks for pointing this out. As mentioned in Theorem 4.2 of Liang et al. (2025), since we used the full dataset to calculate the gradient, the stochastic gradient reduces to the true gradient. Hence, $\\delta_g = 0$.\n\n[1] Liang, F., Kim, S., and Sun, Y. Exended fiducial inference: Toward an automated process of statistical inference. Journal of the Royal Statistical Society Series B, 87:98–131, 2025.\n\n\n**[W3.] Eigenvalue assumption**\n\nIn real-world implementations, when the penalty coefficient $\\eta_\\theta$ in Equation~(10) is large, the estimated parameters $\\hat{\\theta}\\_i $ tend to be almost identical across all $i$. Since the $\\hat{\\theta}\\_i$'s are highly correlated, the same is true for the corresponding $m_i$'s. As a result, $\\mathbf{E}[m_i m_i^T]$ is nearly a rank-1 matrix, with one dominant eigenvalue and all others close to zero. On the other hand, if $\\eta\\_\\theta$ is not sufficiently large, more variation is allowed across the $\\hat{\\theta}\\_i$'s, and the lower bound for the minimum eigenvalue of $\\mathbf{E}[m_i m_i^T]$ will be slightly larger. In theory, as $\\eta_\\theta \\to \\infty$, the bottleneck size could be reduced to $1$. In practice, however, $\\eta_\\theta$ does not approach infinity, so we choose a slightly larger bottleneck size to promote more robust convergence of the algorithm while still satisfying the eigenvalue assumption.\n \n**[W4.] Interpretation of Theorem 3.2**\n\nThe asymptotic consistency of the inverse function essentially ensures a one-to-one correspondence between $ Z_n $ and $\\theta $; that is, different realizations of the random error $Z_n$ lead to different estimates of $ \\theta $, thereby ensuring that the uncertainty in $Z_n $ is properly propagated to $ \\theta $ and  the uncertainty in $\\theta$ is accurately quantified."}]

Author rebuttal:

Meta-review: The paper follows in the footsteps of many works aiming to tackle uncertainty quantification in machine learning. In contrast to many of the currently well known methods, the paper however explores an alternative based on extended fiducial inference (EFI). This uncertainty quantification method is claimed to surpass some of the caveats of traditional methods, including Monte-Carlo Dropout or Bayesian uncertainty estimates. The paper first introduces EFI, then provides theoretical arguments for its use, and finally empirically compares it on a simulated example and small real-world data. The main strength of the paper is definitely its novelty and uniqueness, demonstrating out of the box thinking and a well-written exposition of a potentially new direction. This simultaneously could be regarded as the main weaknesses, making the paper hard to follow for non-experts in EFI (which is likely to be the majority of present researchers in the ML field) and limiting the current empirical assessment and comparison. The paper also does not inspect the use of uncertainty estimates in a deeper way. 

Overall, this is a very hard to judge paper, as has also been noted by various reviewers. Like the reviewers, the AC is not an expert in fiducial inference, and the confidence in judgement of the paper remains similarly low to all the reviewers. On the positive side, the reviewers have all agreed that the perspective is likely novel and the paper appears to be technically correct, to the degree they can judge it. The AC agrees with this perspective and did not find obvious technical flaws. The main points of initial concern mostly seem to resolve around a) the limited discussion of other uncertainty quantification methods in the literature, b) a limited empirical investigation and comparison on any real world scenarios (beyond the simple one presented), c) a general lack of clarity whether the method indeed achieves what it promises in terms of overcoming limitations by other uncertainty quantification strategies. Especially the latter was a big point in the rebuttal, where the authors have provided further evidence and discussions to support their claims. Ultimately reviewers were decently satisfied, although their confidence remains low, finally all agreeing that the paper has a lot of potential value, that remains hard to judge. 

The AC shares the sentiment with the reviewers that there are many ways in which the paper could be improved. The AC recommends the authors to carefully revise the paper for its final version. In particular, given that the paper doesn't currently have an appendix, the AC strongly recommends to provide substantially more background in appendix sections, so that readers less familiar with such a new topic have an easier time to follow. In these appendix sections, the AC also strongly encourages the authors to add their additional experiments wrt hyper parameter choices, that were provided in the rebuttal. 

Taking all of the above into account, the AC finally recommends to accept the paper. Despite general low confidence, all reviewers and the AC agree that the paper may have a lot of value, even if it remains hard to judge by today's standards. This is generally the fate of any papers that pursue out-of-the box and novel directions, and it would feel wrong to penalize the paper for pursuing non-standard directions. Given the assumed technical correctness of the paper, the AC recommends to accept the paper and let the community decide the use and impact of the paper as time progress.
Decision: Accept (poster)