Calibrated Physics-Informed Uncertainty Quantification

Authors: Vignesh Gopakumar, Ander Gray, Daniel Giles, Lorenzo Zanisi, Stanislas Pamela, Matt Kusner, Marc Peter Deisenroth
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Surrogate Models, Uncertainty Quantification, Neural-PDE, Physics-Informed, Conformal Prediction

Abstract:
Neural PDEs have emerged as inexpensive surrogate models for numerical PDE solvers. While they offer efficient approximations, they often lack robust uncertainty quantification (UQ), limiting their practical utility. Existing UQ methods for these models typically have high computational demands and lack guarantees. We introduce a novel framework for calibrated physics-informed uncertainty quantification to address these limitations. Our approach leverages physics residual errors as a nonconformity score within a conformal prediction (CP) framework. This enables data-free, model-agnostic, and statistically guaranteed uncertainty estimates. Our framework utilises convolutional layers as finite difference stencils for gradient estimation, our framework provides inexpensive coverage bounds for the violation of conservation laws within model predictions. In our experiments, we utilise CP to obtain marginal coverage for each cell and joint coverage over the entire prediction domain of various PDEs.

Decision: reject
Reviewer scores: 6, 3, 3, 6

Reviews and discussion:
<thread>
creation date:
1730245107981

thread_type:
official_review

<thread_object>
creator: reviewer_kq4a

<content>
summary:
The paper introduces a method to quantify the prediction uncertainty of a data-driven neural PDE solver using conformal prediction. This technique leverages the assumption that the physical residual operator, which defines the PDE dynamics, is known. By using a discretized version of this operator, the method assesses whether the neural solver’s predictions are consistent with the expected dynamics.

strengths:
- The paper introduces a novel approach that integrates a physical prior with uncertainty quantification techniques to evaluate the performance of purely data-driven neural PDE solvers.
- The idea is presented with clear details and supported by experiments on diverse equations. The authors effectively illustrate how this helps assess and improve the reliability of the neural PDE predictions.

weaknesses:
- My primary concern is about the choice to use only the physical residual for PRE non-conformity. Since most neural PDE solvers for temporal dynamics aim to solve an Initial-Boundary Value Problem (IBVP), it's essential not only to consider the differential operator but also to account for how the solution aligns with the imposed initial and boundary conditions. The authors may need to address this aspect of the problem and explore how incorporating these constraints could enhance the conformity assessment.

questions:
I suggest that the authors provide a more intuitive explanation of how the conformal prediction techniques applied in the paper work. Offering clearer insights into these methods could help a broader audience better understand the concepts and their relevance to the study.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730357191238

thread_type:
official_review

<thread_object>
creator: reviewer_yec3

<content>
summary:
The paper uses physics residual errors as nonconformity scores within a conformal prediction framework for efficient and calibrated uncertainty quantification in neural PDEs.

strengths:
The paper is clearly written and easy to understand.

weaknesses:
1. The proposed method appears relatively simple and lacks novelty. Using the Physics Residual Error as a nonconformity score for conformal prediction seems to be a minor contribution.
2. The paper lacks a comparison with other uncertainty quantification (UQ) methods. I think various UQ methods could be applied based on the point-wise PDE residuals.
3. Additional analytical experiments are necessary to gain a deeper understanding of the proposed method. Could you conduct some tests in real-world scenarios?
4. The numerical PDE residual shows considerable numerical error due to discretization issues, particularly in 2D+time PDEs. How is the time derivative calculated over a 20-frame sequence? Please provide further clarification.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730479344139

thread_type:
official_review

<thread_object>
creator: reviewer_u71s

<content>
summary:
The paper proposed to use a conformal prediction (CP) framework to estimate uncertainty in neural PDEs. The main proposal of the paper is the utilisation of physics residual errors as nonconformity scores within a CP framework. The main goal of the paper is to provide a fast uncertainty quantification framework for neural PDEs. The proposed method is illustrated on several toys examples.

strengths:
If demonstrated more extensively through additional experiments and theoretical results, the paper potentially provides a computationally cheap uncertainty quantification framework for neural PDEs.

weaknesses:
- At the current moment, the paper provides only a minor extension of the CP framework to the context of neural PDEs.
- There are no theoretical results in the paper supporting claims about bounds and statistical guarantees. 
- The experiments are limited to several toy problems with only illustrative examples (and no simulation study with diverse/relevant evaluation metrics, such as l2 measure for the solution, coverage for the quality of CI, etc.). 
- There is no comparison to any other uncertainty quantification methods.

questions:
1)“The majority of literature in UQ for neural PDE solvers has been looking at Bayesian
methods, such as dropout, Bayesian neural networks and Monte Carlo methods (Geneva & Zabaras, 2020; Zou et al., 2024; Psaros et al., 2023), which lack guarantees or are computationally expensive.”  
This statement is rather high-level. The authors should specify what guarantees these methods do not provide which conformal prediction does provide. 
2) The paper does not distinguish between epistemic and aleatoric uncertainties and should be more explicit and precise about what type of uncertainty quantification it provides. 
3) On line 155 there is a claim: “We introduce a novel data-free nonconformity score for Conformal Prediction (CP) to obtain statistically valid and guaranteed error bounds for neural PDE solvers.”
Theoretical support should be provided to this claim. 
4) Both Figure 4 and Figure 5 illustrate very large confidence intervals for joint CP which would not correspond to 95% coverage. For marginal CP, it is hard to visually tell what coverage is. The same holds for the later figures. The paper should provide additional metrics to evaluate the results quantitatively. 
5) Figure 4 and 5 provide a single illustrative example and not a proper Monte Carlo study to illustrate the calibration and coverage of the method. Considering that the paper suggests the method is computationally cheap, there is no reason to not provide more extensive results supporting the paper's claims. 
6) There is no comparison to any other uncertainty quantification methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642091193

thread_type:
official_review

<thread_object>
creator: reviewer_mcyw

<content>
summary:
This paper presents a data-free conformal prediction approach for uncertainty quantification in neural PDE solvers, which does not require additional true PDE solutions to evaluate nonconformity scores. This data-free capability is achieved by assessing physics residual errors (PRE), or violations of conservation laws, for each prediction. PRE-CP provides both marginal (dimension-wise) and joint (domain-wide) coverage, enabling statistically sound conformal bounds for predictions. Additionally, the authors employ convolutional layers as finite difference stencils for faster and more accurate gradient estimation, as well as a rejection sampling method that uses CP-based criteria to reject predictions with a predetermined probability.

strengths:
•	Very interesting and important topic for SciML community and very well written

•	Strong motivation

•	Complete experiments and discussion with strong empirical results

•	Methods are clearly written and are easy to follow

weaknesses:
I have some question regarding how will PRE-CP behave when the neural model fails to predict PDE solution, or generating consistently high physical residual errors.

questions:
Krishnapriyan et al. [1] show that it is highly likely for PINNs to fail in predicting PDE solutions when the PDE coefficients are high, which can lead to consistently high residual errors across almost the entire PDE domain. I am curious whether PRE-CP would generate very wide conformal bounds that still provide relatively high coverage of these residual errors, potentially underestimating the poor performance of the neural solver.

[1] Krishnapriyan, Aditi, et al. "Characterizing possible failure modes in physics-informed neural networks." Advances in neural information processing systems 34 (2021): 26548-26560.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a data-free conformal prediction approach for uncertainty quantification in neural PDE solvers, which does not require additional true PDE solutions to evaluate nonconformity scores. The authors had tried to address the reviewers' comments and questions. Some reviewers still have some concerns and do not want to support the publication of the paper. The reviewer thinks that the method is limited in practice as we often lack knowledge of the exact form of the underlying PDEs. Moreover, the reviewer also mentioned that the method is somewhat straightforward and lacks substantial novelty in addressing the underlying challenges. Unfortunately, I cannot recommend the acceptance of this paper based on the reviewers' recommendations. I strongly encourage the authors to clarify all the points that reviewers mentioned to improve the paper for the future submissions.