Calibrated Uncertainty Quantification for Operator Learning via Conformal Prediction
Authors: Ziqi Ma
David Pitt
Kamyar Azizzadenesheli
Anima Anandkumar
Venue: Accepted by TMLR ()

Abstract: Operator learning has been increasingly adopted in scientific and engineering applications, many of which require calibrated uncertainty quantification. Since the output of operator learning is a continuous function, quantifying uncertainty simultaneously at all points in the domain is challenging. Current methods consider calibration at a single point or over one scalar function or make strong assumptions such as Gaussianity. We propose a risk-controlling quantile neural operator, a distribution-free, finite-sample functional calibration conformal prediction method. We provide a theoretical calibration guarantee on the coverage rate, defined as the expected percentage of points on the function domain whose true value lies within the predicted uncertainty ball. Empirical results on a 2D Darcy flow and a 3D car surface pressure prediction task validate our theoretical results, demonstrating calibrated coverage and efficient uncertainty bands outperforming baseline methods. In particular, on the 3D problem, our method is the only one that meets the target calibration percentage (percentage of test samples for which the uncertainty estimates are calibrated) of 98%. Code is available at https://github.com/neuraloperator/neuraloperator/blob/main/scripts/train_uqno_darcy.py.

Published in Transactions on Machine Learning Research (08/2024)
Calibrated Uncertainty Quantification for Operator Learning via Conformal Prediction
Ziqi Ma
zigima@caltech. edu
California Institute of Technology
David Pitt
dpitt&@oaltech.ede edu
California Institute of Technology
Kamyar Azizzaeneehelli
kamyoro@nvidia..cm
NVIDIA
Anima Anandkumar
anima@caltech.. edu
California Institute of Technology
Reviewed on OpenReview: https: opwrrevee.....eeeoem net/ forum? id= cOpegay121
Abstract
Operator learning has been increasingly adopted in scientific and engineering applications, many of which require calibrated uncertainty quantification. Since the output of operator learning is contimuous function, quantifying uncertainly simultaneously at all points in the domain is challenging. Current methods consider calibration at a single point or over one scalar function or make strong assumptions such as Gaussianity. We propose a risk-controlling quantile neural operator, distribution-fret finitc-sample functional calibration conformal prediction method. We provide theoretical calibration guarantee on the coverage rale, defined as the expected percentage of points the function domain whose true value lies within the predicted uncertainty ball. Empirical results 2D Darcy flow and a 3D car surface pressure prediction task validate our theoretical results, demonstrating calibrated coverage and efficient uncertainty bands outperforming baseline methods. Tn particular, on the 3D problem, our method is the only one that meets the target calibration percentage (percentage test samples for which the uncertainty estimates are calibrated) of 98%. Code is available at htttp///st...tcooooee.c..tt///// (UQNO module).
Introduction
Neural operators have been increasingly adopted to solve partial differential equations (PDE), demonstrating traditional numerical methods. Their applications span various scientific and engineering domains, including weather forecasting (Pathak et al. 2022), carbon capture (Wen et al. 2023 automotive design (Li et al.) 2023),, and nuclear fusion (Goppkumar et al. 2023) to name a few. One key challenge in their adoption in real-world sconarios lics in uncertainty quantification estimate how uncertain the model output is. Applications like plasma evolution prediction for nuclear fusion (Gopakumar et al.) 2023) and pressure field prediction for automotive design (Li et al. 2023) are safety-critical and thus require reliable quantification of model uncertainty. Applications such as extreme weather forecasting Kurth et al. (2023) Lam et al. (2023): ; Persson & Grazzini (2007), carbon capture and storage (Wen et al. 2023) affect high-impact decision-making, which also requires uncertainty quantification. Existing methods for uncertainty quantification face challenges in three key arcas, viz., lack of function-space coverage, distributton--fee calibration guarantee, and scalability.
significant speedup
capability
Function-space formulation: Neural operators provide function-space solutions that can be evaluated any point in the domain. The necessary formulation of uncertainty for neural operators requires to be

---

Published in Transactions on Machine Learning Research (08/2024)
Base Neural Operator Output (red dashed)
Simultaneous Point-wise Uncertainty (blue dotted)
Predicted uncertainty band
1. Train with Generalized Quantile Loss
2. Calibrate with Conformal Prediction
Prediction Ground Truth
Figure 1: Overall schematic of UQNO, a risk-controlling quantile neural operator. In operator learning, the learned neural operator outputs a function (red dots sampled at grid points). We train a residual operator with generalized quantile loss and then calibrate with conformal prediction, which yields simultaneous pointwise uncertainty estimates with a PAC guarantee calibration coverage the expected percentage of true value (black) that lies within our predicted uncertainty bands (green=upper bound and yellow=lower bound). In this example, since the output is 1D, we output pointwise uncertainty band. In higher dimensions, we output a pointwise heterogeneous uncertainty ball.
the function space, which provides simultaneous uncertainty estimation for all points. For example, in an automotive design setting that uses neural operators to predict surface pressure Li al. (2023), simultancous uncertainty estimates on the whole surface can inform designers of structured error around the front face ridge, shown in Figure 1 whereas a single-point. aggregate measure of uncertainty cannot convey such information.
Uncertainty quantification function spaces is more challenging than naively combining point predictions in the function space. Even if we obtain point-wise guarantees on calibration with probability 1 p, the standard union bound leads to loose bound the probability of simultaneous calibrations. Prior work focuses single-point uncertainty estimation (Guo et al. 2023), and theoretical developments investigate uncertainty quantification in the function space, yet focus on transformed formulations such as functional projection or scalar function properties such as pseudo-density or loss (Lei et al. 2015 Benitez et al. 2023
Distribution-free calibration guarantee: Classical deep learning uncertainty quantification methods such as MCDropout (Gal & Chahramani] 2016) or ensemble learning (Maddox et al. 2019) do not provide
calibration guarantees. These methods output mean and variance estimates under Gaussian assumptions, and the results are generally hcuristic. Recent works on uncertainty quantification for operator learning are either heuristic (Guo et al., 2023) Akhare et al., 2023 Nehme et al. 2023), or rely on Gaussian assumptions and approximations that may not hold in real-world settings (Magnani et al., 2022). Heuristic uncertainty estimation is insufficient for safety-critical or high-impact applications. We need method with rigorous calibration guarantee that works "in the wild', where distributional assumptions might be broken. Since real-world applications have finite data, we also want our calibration guarantee to be finite-sample rather than asymptotic.
Scalability: Frameworks such as Bayesian inference are principled, yet challenging to scale up. For example, Meng et al. 2022) considers problems with per-function samples on the order of hundreds, yet many practical applications are magnitudes larger in scale, e.g., 7.2 million mesh points in fluid dynamics Li et al. 2023 creating computational challenges for such methods.
We present uncertainty-quantin neural operator (UQNO) framework that simultaneously addresses these challenges by leveraging the conformal prediction framework. UQNO is distribution-frre and finite-sample functionally calibrated conformal prediction method built on the framework of risk-controlling quantile operator learning that leverages the classical conformal prediction principle (Lei et al., 2015 Angelopoulos & Bates 2021). high-level schematic of our method is shown in Figure We develop generalized quantile loss extended the function space to train neural operator that takes a function as input and outputs heuristic uncertainty band that be queried at any point. Then, provide the conformal prediction
2

---

Published in Transactions on Machine Learning Research (08/2024)
framework to calibrate the risk-controlling prediction set for operator learning. This framework enables function prediction along with point-wise calibrated coverage uncertainty.
We demonstrate that our method satisfies the desired calibration guarantee while providing efficient uncertainty bands, outperforming baselines in a 2D Darcy flow problem, and a 3D automotive surface pressure field prediction. In the 2D Darcy flow problem, our method provides uncertainty bands that are 1.52x tighter than MCDropout and 76.1x tighter than Laplace approximation while satisfying the target calibration percentage of 98%. In the 3D car surface pressure field prediction problem, our method is the only method that satisfies the target calibration percentage of 98%. Our main contributions as follows:
We provide a function-space uncertainty formulation for operator learning by leveraging the conformal prediction framework.
We provide calibrated uncertainty estimates simultancously for all points on the function domain.
We provide PAC bound on the coverage percentage of our uncertainly estimates.
We empirically demonstrate that UQNO outperforms existing uncertainty methods in terms of calibration percentage and band tightness for both data-rich and data-scarce regimes, with per-function point samples up to 177k.
2
Problem Formulation
2.1
Uncertainty Quantification for Operator Learning
Operator learning Li et al. (2020) can be formulated as,
Ć = min E (a. (G((),,))
(1)
where a A = A(D,R")) is the imput function, u E == U(D, is the output function, D C Rd is compact and A, U are separable Banach spaces. We aim to learn the operator & which minimizes expected loss I: U U E over the distribution of input-output functions. The point estimate of operator 9 A - does not give information of uncertainty beyond its empirical loss available data. With uncertainty quantification, want to output an uncertainty set instead of a point estimate. We formulate the uncertainty quantification problem as finding
C:A-U', ='= U'=U'(D,((R")))
where B(Rd)) is the Borel set on Rdd, i... C evaluated at any function A gives correspondence (Aliprantis & Border 2006 (D, D(Rd", C(a)) which maps any point E D to prediction set on Rdd the co-domain of
u.
2.2 Risk TokkCConroollng Prediction Set for Operator Learning
The uncertainty mapping C defined above is required to be calibrated following the established definition of risk-controlling prediction set Angelopoulos el. al. 2022); ; Bates el al.] (2021). The Risk-controllingg prediction
set for operator learning is defined as the following PAC bound: A (a, t-rii--ootrrollinn prediction set satisfies
P (B{((((((())))) E Cx(a)()))
33)
where the predicted uncertainty set (x) is connected. The risk-controlling prediction set specifies that the probability (over (u, a)) that when sampling points its domain, the expected percentage of "violating points" (i.e. points whose true function value falls outside of the provided uncertainty ball) exceeds level a does not exceed 8.
2.3 Parameterizing Uncertainty Set
We paramcterize the uncertainty set as pointwise ball on the co-domain of output function u. Assuming we have non-calibrated uncertainty estimating operator E with ((a)(z) e R estimates the pointwise radius of "uncertainty ball' for each new input function a, specify the following parameterization of
3

---

Published in Transactions on Machine Learning Research (08/2024)
C. We choose this formulation below because although obtaining calibrated uncertainty estimates is hard, we have access to heuristic uncertainty estimates methods that we extend to operator learning our case, we develop a "residual neural operator' for its flexibility and discretization convergence. Defining the calibrated uncertainty set as a scaled version of the heuristics is a way to transform the function-space uncertainty quantification problem inlo a problem on scalers, thus making amenable to existing principles from conformal prediction.
(x(a)(z) = to lp- 1p & (E(a)(z)
(4)
For given function and given point 2., the set defined above is the set of points in the co-domain of output function whose distance (measured by 2-norm) from the base operator prediction is no greater than (z), i.c. the true value lies within ball centered at the basc operator prodiction with radius (z) scaled by X. In our implementation, E is parameterized by a neural operator as described in Section
3 Methods
To obtain a valid risk-controlling prediction set C, we leverage two main ingredients: 1) a generalized quantile loss formulation for operator learning to obtain E in equation | that provides a good heuristic for model uncortainty; 2) the conformal prediction framework to select À so that the prediction set is well-calibrated.
3.1 Pre- Calibration Quantile Neural Operator
We use statc-of-the-art neural operator architectures to parameterize E in in cquation Fourier Noural Operator (FNO)
2021) for the 2D Darey problem and its variant Geometric-Informed Neural 2023) for the 3D pressure field prediction problem. We generalize quantile loss (Koenker Operator (Li et al.
(Li et al.
2021 to the operator learning setting. Similar to the canonical quantile loss formulation, 2005; Chung et al.
this loss penalizes out-of-band distance weighted by - and in band distance weighted by a, and thus encourages the model to output a value close to the 1 - a percentile of the error magnitude seen during training. E is trained with the loss defined below:
(9()))) u) = a S. (())))))((()())bb F(a)())(()((m))>> 9 - (m)))
+(1 +(1-a) -a-a a)), /, [1ll(())
((a)()))2 -[|1(())-(())]]]] -ooo
((a)() - [l((x) -(()(=)))))
(5)
where & is the base neural operator as defined in Equation equation 1 provide well-calibrated quantiles (Chung et al. 2021 and we only use this prediction as a pre-calibration estimate.
It is known that quantile loss does not
3.2 Calibration via Split Conformal Prediction
We leverage conformal prediction as an overall framework (Vovk et al., 2005), which provides a finite- sample, distribution-free confidence set for a scalar-value prediction problem, utilizing calibration set that is exchangeable with test data. Split conformal prediction is used to avoid re-calibration for each new test data point Vork et al. (2020). Note that we do not require training data to come from the same distribution.
Under this framework, given training set Dtrain {(a), w)),, calibration set Deal {af uf)li 1, ..... and nonconformity score function s (A, U) + I, under the assumption that the calibration set and test set are exchangeable, the possible outcomes which the 1 - a quantile of obtained on the calibration quantile on the test set, we have the following: Given training
set provides a valid estimate of the set Dtrain = {(af, 1 calibration set Deat = {aa uf)li = 1,..,,), we can define any nonconformity score function s (A, U) + and calculate the score for all samples in the calibration set, i.e. 8; = s(a;, ie., let.
[nn + 1)(1 - a:] 'th quantile of s;
C(a) (uE U: 5(); S 5
(6)
4

---

Published in Transactions on Machine Learning Research (08/2024)
provides the guarantee that Pfu C(a)) 2 1 new test sample. We define the nonconformity score function 8 (A, 1J) > R as follows:
sg(a, u) -0--att
(7)
In jj is the j'th smallest value in the set ( [ (aa)(.))l2 1....m. m is the number of
where >> points on the domain for each function sample. Combined with equation this gives a (a, 5) risk-controlling confidence set. Theoretical derivation is presented in Section
B(a)(z.)
3.3 Risk- Disk-Controlling Quantile Neural Operator
Combining the two ingredients above, we obtain a risk-controlling quantile neural operator that can simulta- neously predict uncertainty estimates for each point, given a test function. The overall algorithm is described in Algorithm 1
Algorithm 1: Risk-Controne Controlling Quantile Neural Operator
1: Input: Base model 9, training set (separate from the training set of G) Derain {aat u()) calibration set Deal {(af, } of size n. Each (af, uf) is measured with discretization m, i.e. of uf evaluated at -I....mm, E R.
2: Train: Train quantile neural operator E on Draain' with generalized quantile loss defined in equation Calibrate:
for (uf,u() E Deal do
s(at,u9) f 01 1 - a a +1 as defined in equation
where m min,mm.-.. mm
2m
5: end for
1-1- ((+1)(6-e-2mc) quantile of s(af, uf) on Deal
Predict: given a test input function and any E Rdd outpul uncerlainty ball B(G(a)())
which is (a, 6) rist-controlling prediction equation
This method has three advantages: (1) Heteroscedasticity:
The base uncertainty estimator is parametrized neural operator Li et al. 20021 2023 which can predict higher uncertainty for input samples that
are dissimilar to training input. (2) Simultaneous pointwise prediction: Simultaneous prediction of
uncertainty estimates for all points on the domain allows for structural understanding of error, as visualized in Figures and This is not possible in prior methods such as Guo et al. (2023) and Benitez et al. (2023). (3) Controllablility: We note that both the domain coverage threshold a and function-level coverage (equation 3 are user-specified. This is demonstrated empirically in Figures and We note that the calibration guarantee of our method is an upper bound over the true calibration, i.c., the actual coverage is guaranteed to be less than the target coverage. The sparser our sample points (on the domain), the more conservative band becomes.
Our formulation naturally exlends to scenarios where data is in various ieteetiiaaaiooniireiiiii grid geome- tries. Different discretizations is regarded as different approximations of continuous functions that satisfy exchangeability. The thcorctical derivation is presented in Section 5.2
4 Experiments
We demonstrate neural operator calibration method on two tasks: high-resolution, data-rich 2D Darcy flow problem and data-scarce pressure prediction problem on 3D car surfaces. We compare with existing methods that can be used or adapted for these tasks, including,
MCDropout Gal & Ghahramani (2016): which predicts uncertainty by aggregating results from multiple (we use 10) models trained with random dropout, which we generalize to the operator learning setting.
Deep Ensemble Lakshminarayanan al. (2017): which predicts uncortainty by aggregating results from an ensemble (we use 10) of models with Gaussian assumptions, which we generalize to the operator learning setting.
5

---

Published in Transactions on Machine Learning Research (08/2024)
Table 1: Calibration percentage and bandwidth comparison of UQNO with other methods (V indicate passing target of 1- aa and X indicate below target). We see that UQNO consistently provides the tightest uncertainty band that salisfies calibration conditions. For both lasks, we show higl-domain-thresbun scenario (a 0.02 for Darey and 0.04 for car, values larger for car due to its lower resolution due to the correction term t in Equation 7 and low-domain-threshold sccnario (a = 0.1 for Darcy and a = 0.12 for car). In the Darcy problem, where we have sufficient data, most methods satisfy our calibration target of 98%, but our method gives the most efficient band. In the car problem with scarce data, our method is the only method that provides calibrated results. Training time is approximate GPU hours on a single RTX4090.
2D Darcy Flow
3D Car Pressure
High domain threshold
Low domain threshold
High domain threshold
Low domain threshold
Bandwidth Calibrated
Bandwidth Calibrated
Training time
Bandwidth Calibrated
Bandwidth Calibrated
Training
time
MC
0.00108
V 99.5%
0.00108
100%
10 hours
0.12333
X 18.9%
0.12332
X 66.7%
30 hours
Dropout
Deep
0.00075
X 32.2%
0.00075
90.2%
10 hours
0.42972
X 74.8%
0.42972
96.4%
30 hours
En-
semble
Laplace|
0.03453
V 100%
0.03453
100%
hour
1.94708
X 94.6%
1.94708
94.6%
3 hours
Ap-
prox.
Neural
N/A
X 0%
N/A
15.1%
hour
N/A
X 0%
N/A
0%
3 hours
Poste-
rior
UQNO
0.00062 v /98.6%
0.00045
99.8%
2 hours
0.55077 V 98.2%
0.40018
100%
6 hours
Laplace approximation Magnani et al. 2022): which takes point predictions as the Maximum a posteriori (MAP) estimator and does local Laplace approximation by leveraging the Hcssian of the last layer weights. Note that bandwidth is not defined for this method since its predicted uncertainty subspace unbounded.
Neural posterior principal components Nehme et al. 2023 :: which trains the model to not only predict a point estimate but also the first k principal components residual, thus outputting an "uncertainty subspace' based on these components.
We focus on two aggregate metrics: (1) Calibration percentage: the percentage of functions in the test set that satisfy the target threshold. For each test sample, "satisfying the target threshold' means over 1 - of uniformly-sampled points the function domain lie within predicted uncertainty sets, described in Equation Note that calibration percentage is an aggregate metric defined on the whole test set. For a single instance, we also define "coverage" as the percentage of uniformly sampled points on the function domain that lic within predicted uncertainty sets for a specific function. This metric is used in single-instance visualizations, as shown in Figures 2 5 (2) Bandwidth: the average predicted uncertainty ball radius averaged across all sampled points on the domain. This metric shows how efficient method is triviiiilll, an infinitely large uncertainty prediction will always have perfect calibration percentage, yet. is uninformative since the balls are not efficient.
We demonstrate that our method provides good calibration and tight bandwidth, compared to baselines. The Darcy flow task illustrates data-rich setting with 5000 data points, whereas the car pressure prediction task represents data-scarce setting with a 500-sample training set for a 3D problem. Table I provides statistics of bandwidth and calibration percentage percentage for all methods on both tasks. We see our method consistently outputs the most efficient uncertainty bands that satisfy the calibration targets. In the data-rich setting of Darcy flow, we achieve up 1.53x efficiency improvement band tightness. More notably, in the data-scarce problem of 3D car pressure prediction, all other basclines fail to provide calibrated uncertainty estimates in both high- and low-domain-threshan settings.

---

Published in Transactions on Machine Learning Research (08/2024)
VONO
Ensembre
Laplace Aporoctration
Eroor
Coverage 0.991
Coverage D 964
Enveraee 8.8
Coverage 1.000
50000
Figure 2: Uncertainty quantification comparison across methods on 2D Darcy flow problem. The leftmost heatmap plots true error. The top panels show the prodicted pointwise uncertainty, and the bottom panels show the coverage (i.e. true error less than predicted error) for each point on the domain yeelowyellwwwwwoooo---- points are covered by our predicted uncertainty bands, and purple points uncovered. The coverage percentage for each method is shown above the bottom panels. Our method of UQNO predicts uncertainty that corresponds well with true error while providing 99.1% domain coverage. MCDropout does not capture the uncertain region well. Laplace approximation greatly overestimates uncertainty- on the scale of 50x larger than true crror, and thus appears all yellow in the heatmap.
4.1 Darcy Flow
The 2D Darcy Flow problem second-order, linear, elliptic PDE of the form below:
V (a(z)
f(z) (0, 1)² 0 1)2
u(x)
(8)
This is a data-rich scenario with 5000 total training data and 421 x 421 resolution, for which obtain the ground truth from prior work Li et al. 20211. We fix the same Fourier Neural Operator architecture for all methods. For UQNO, we split the training set in half for training the base and the quantile model. Figure provides a visualization of domain-levee uncertainty predictions and coverage for one function sample for UQNO (our method), MCDropout, and Laplace approximation. UQNO captures the error structure well and provides good domain-level coverage.
In addition to coverage, UQNO also provides a light band, shown in Figures 3 and 7 Among the methods that satisfy calibration (green shaded), our method (purple dot) consistently provides the lowest bandwidth. We also plot the average bandwidth against calibration percentage for different methods in Figure 3 UQNO provides the tightest band among methods while providing a calibration percentage 98.8%. We note that the neural posterior principal components method does not provide finite band since it outputs subspace, and thus, bandwidth does not apply. Nevertheless, we take the percentage of error variance falling in the predicted subspace its calibration percentage to compare with the other methods. We see the variance ratio falling into the dominant linear subspace is low (15%), suggesting the nonlinearity of the problem. We further demonstrate the flexibility of UQNO by showing calibration results at different target calibration percentages (1 5 in Equation g and domain coverage thresholds (1 a a in Equation 3) There is trado-off between the tightness of the uncertainty band and calibration percentage- srrentage-intuutivelyy an infinitely large uncortainty set has a perfect calibration percentage yet is uninformative. By varying the domain threshold (a in Equation 3) and target percentage (6 in Equation 3/, we calibrate UQNO to provide different guarantees. Figure demonstrates the flexibility of our method by trading off band tightness with calibration percentage. Intuitively, wider bands correspond to higher calibration percentage.
We also show the efficacy the calibration percentage guarantee in Figure 6 The empirical results agree with our theoretical guarantce that the true calibration percentage should be no less than the expected calibration percentage of 6 as in Equation 3 for various domain threshold a values. We note that by definition, our method is conservative bound due the finite resolution of data samples (the term in Equation 7
7

---

Published in Transactions on Machine Learning Research (08/2024)
Average bandwidth (lower is better)
UQNO (Ours)
MCDropout
Laplace Approximation
Deep Ensemble
10
PosteriorPC
Target calibration percentage
10-1
0.0
0.2
0.6
0.8 1.0
Calibration percentage
Figure 3: Bandwidth vs. calibration pgrcentage comparison across methods on Darcy flow problem. We see a clear advantage of UQNO (purple dot), providing 1.52x tighter band than MCDropout and 76.1x tighter band than Laplace approximation. This plot is generated with 0.1, target calibration percentage 98% for UQNO, and N 3 principal components for posterior principal component method.
0,00670
alpha=2%
alpha=4%
0.00065
alpha=6%
alpha=8%
0.00060
alpha=10%
Bandwidth
0.00055
0.00050
0.00045
Calibration percentage
Figure 4: Bandwidth vs. calibration percentage tradc-off of UQNO on 2D Darcy flow. Each curve shows the bandwidth vs. calibration percentage trade-off for fixed domain threshold (ar in Equation 3) demonstrating the flexibility to achieve a higher calibration percentage with wider bands. Overall bandwidth increases as the domain threshold becomes more stringent (smaller a).
4.2 3D Car Surface Pressure Prediction
This task aims to learn the solution operator for computational fluid dynamics (CFD) simulations of the Navier-Stokes equation for car shapes from Umetani & Bickel (2018) modified from the Shape-Net dataset Chang et al. 20155) car category. The car surface is represented 3D mesh of 3586 points, and we obtain ground-truth time-averaged pressure fields by solving the Reynolds-averagel Navier-Stokes Stokes (RANS) equation with finite element solver Zienkiewicz et al. (2014) with Reynolds number 5x106 and inlet velocity 72km/h as in Li et al. (2023). This is a data-scarce setting with only 500 lotal training samples on a 3D mesh of 3586 points. We have access to the signed distance function (SDF) and point-cloud representations as input and aim to predict the pressure at every mesh point on the car surface. Figure compares UQNO, MCDropout, and Laplace approximation on a sample car shape and shows UQNO to capture the error structure and provides good coverage of 99.7%, albeit being conservative (over-estinates uncertainty).

---

Published in Transactions on Machine Learning Research (08/2024)
UQNO (Ours)
MCDropout
Deep Ensemble
Laplace Approximation
True Error
12
Coverage=0.9967
Coverage-0.9313
Coverage=0.
Coverage=0.9989
64
Figure 5: Uncertainty quantification comparison across methods on 3D car pressure prediction problem. The leftanost healmap plots true error. The top panels show the predicted pointwise uncertainty, and the bottom panels show the coverage (i.e. true error less than predicted error) for each point on the domain-yellow points are covered by our predicted uncertainty bands, and purple points are uncovered. The coverage percentage for each method is shown above the bottom panels. Our method, although conservative (over-estimates uncertainty); able to capture the error pattern on both the top and bottom of the front. face of the car. Due to the conservative nature of our calibration procedure, our method satisfies the coverage threshold of > 98% (actual 99.7%). MCDropout, although on the same scale as true error, misses the top error region and fails to meet the coverage threshold. Laplace approximation greatly over-estimates the error and does not capture the error structure well.
1.00
Actual calibration percentage
0.98
0.94
0.92
alpha=2%
alpha=4%
0.90
alpha=6%
alpha=8%
0.88
alpha=10%
0.88
0.90
0.92
0.94
0.96
0.98
1.00
Target calibration percentage
Figure 6: Actual vs. larget calibration percentage of UQNO. Different a values are plotted with different colors. We see the guarantee is always satisfied.
9

---

Published in Transactions on Machine Learning Research (08/2024)
We see that UQNO provides interpretable error structure that shows high error along the edges of the front of the car, whereas MCDropout mainly captures the bottom region, and Laplace approximation provides a very coarse етrоr estimate around the front center and over-estimates the error magnitude, Figure 7 compares the bandwidth and calibration percentage across different methods, showing UQNO (calibration percentage 100%) to be the only method that satisfies the calibration percentage larget of 98%. We see that MCDropout, has low calibration percentage at 66.7%, and Laplace approximation gives 4.85x wider band while only providing 94.6% calibration pgrcentage. The low calibration percentage of the Neural Posterior PC method (0%) shows the problem to be highly nonlinear.
Average bandwidth (lower is better)
UQNO (Ours)
MCDropout
Laplace Approximation
Deep Ensemble
10a
PosteriorPC
Target calibration percentage
0.0
0.4
0.8
Calibration percentage
Figure 7: Bandwidth calibration percentage comparison across methods 3D pressure prediction problem. Note the y-axis is plotted on logarithmic scale. We see that UQNO (calibration percentage 100%) is the only method that meets the target of 98%. MCDropout gives a low calibration percentage of 66.7%, and Taplace approximation gives a 4.85x wider band while only providing 94.6% calibration percentage.
5 Theoretical Guarantee
Recall from Section 2.2 our goal is to quantify uncertainty for the operator learning problem 1
ć = min (o, (G
where A == vdd, ) is the input function, e U(D, ) is the output function, D RR compact and A, U separable Banach spaces. We formulate uncertainty quantification finding a correspondence C:
C:A-U', === -(D((D,(RR..))
where dR is the Borel set Rdd C evaluated any function A gives correspondence (Aliprantis & Border 2006 (D, D(RB(...,, C(a)) which maps any point x E D to prediction set Rd,, the co-domain of u.
To obtain uncertainty quantification with conformal guarantee, want to construct (a, roikioo--onn confidence set satisfies:
Pu (u((()(() e Cx(a)())) < - a) S 5
where (a) is the desired operator that outputs the radius of an "uncertainty ball' at any given point r on the function domain.
Cx(a)(=) {p Rd 1pp (())(1)(=)1le2 s AE(a)(z))
with the minimum possible X.
10

---

Published in Transactions on Machine Learning Research (08/2024)
5.1 Fixed Discretization
We start with the simple case of fixed discretization where m points are sampled from the domain for each function in calibration set. Following the definition of nonconformity score introduced in Equation
st(a, u) [1--+1]
where is the j'th smallest value in the set { [ ((a))=lle 1, 1,... 10m.1. 1>
1 2m
We have,
B(a](z.)
x= I [u(a;)) ((a)( < 2) < <m(1 -
(t u)
((a)(x))
m
(u(a,) e (x(a((n))) < <1-att
(10)
m
Via conformal prediction (Equation 6 let A =1 [(+ 2m (of size n), so we know for test (a, u): we obtain,
th quantile of s(a, u) calibration set
P(sc(a, \1 < 2mn2
(11)
Denote the event that that the true value of a function evaluated at point x lies inside the predicted uncertainty ball evaluated at that point, ie. u(a)(x)) E (x(a)()) as A(r).
[[E,[A(>)] - = ((E P(EAAA)) <
aΛ Σ
H(((a))
<1-a+t
2=1
+ P[E]((()) <1
<1 a
H((x;)) 21
(12)
m
Note that by calibration we have,
[[Ex( < a 1 Σ
<1 a [((x))) <1 < e- 2mm
(13)
==1
and by Hoeffding bound we have,
1 m
[[Ex[A()) a
21 2mn
(14)
m
Combining above we get [[Ex[ <1 <1 5.
5.2 Mixed Discretization
For the fixed discretization case, instead of sampling points from the domain for each function in calibration set, assume my points are sampled for each input-oulpul function pair (a.,u:). We assume test sample (a',²') has discretization m'. The underlying assumption is that even though our query discretization changes, the functions are still exchangeable. Thus, our main analysis still holds, and we just need to account for
wz).
different discretization errors caused by the varying sampling resolution. The definition of nonconformity score is discretizatiion-deeedeen, iee.
se u:) [1-o+t]
where o; is the j'th smallest value in the set { ((o l...m,).. By similar reasoning as inequality E(a)(z))
10 in Section 5.1
mi
s, Uz)
Eu(x,) Cx(o((a))) <l-at4
m
11

---

Published in Transactions on Machine Learning Research (08/2024)
Since the t term is used in the definition of X, it also affects the overall relaxation in the quantile value across all functions (of varying discretization). Thus, need uniform across functions, satisfying t > In
where min (mi, M,. m'}. Let Å == (6 'th quantile of s(a,, u,) on calibration set (of size nt),
P(st(a, u) 1 e 2ma2 6 2m't²
By similar invocation of Hoellding bound, denote (x)')xx) (2(a))x)) A(x),
1
P[Ez[A(x)] <1 a) < <1 as
(A(x,)) <1 <1-a+1
m
m
<1 -
+ [M()]
Σ ((A(z)) 2 a+
m
Note that by calibration we have,
[[E,[A(=)] <1-61-1- a an
1
Σ <1-
<1
m'
i=1
m
S P
<l- <1 22m S 22m't
Σ
and by Hoeffding bound we have,
P[E,[A(T)]< <1-a 1
Σ
11
21 a + 2m't²
Combining above we get P[Ex[A(x)] < <1- a]
6 Related Work
The conformal prediction framework was introduced in the 2000s 2005) to provide distribution free and finite-sample uncertainty estimates for scalar values. It has subsequently been extended to settings such as regression (Romano et al. 2019), time-series forecasting (Ajroldi et al., imaging (Angelopoulos et al.) 2022), and functional data analysis (Lei et al. 2015 Diquigiovanni
Conformal Prediction:
(Vovk et al.
2023
et al.
2022 2021 In this work, provide conformal prediction framework for operator learning, and our main distinction from prior work is heteroscedasticity on function spaces. Methods based modulation pscudo-density (Lei et al. 2015) may limit lcarning statistical error patterns across samples and do not adapt uncertainty estimates based on each test sample. We provide heteroscedastic estimates by combining the conformal prediction framework with a base quantile neural operator.
Many existing methods for uncertainty rely on Gaussian assumptions or approximations that result in heuristic, rather than rigorous uncertainty estimates. For example, Magnani et al. (2022) allows for sampling from a Gaussian process approximate the posterior operator via Laplace approximation, which have shown to overestimate errors. [Akhare et al. (2023) similarly relies Gaussian assumption and over-estimates uncertainly in empirical studies. (Zou et. al. 2023) also uses Gaussian
Approximate Gaussian Methods:
assumption to learn pointwise noise variance as a heuristic, which does not give coverage guarantee. By leveraging the conformal prediction framework, we obtain principled uncertainty estimates.
Data Uncertainty: Bayesian methods [Zou et al.) 2023 Meng et al.) 2022 provide the advantage of dccomposing uncertainty sources, yet face challenges scaling up to high-dimension problems. Nchme et al. (2023) studies data uncertainty by learning oov-diennional latent representations of error provides interpretability, but similar faces challenges when error does lie in low-dimensional manifold
12

---

Published in Transactions on Machine Learning Research (08/2024)
problem dimension grows. Our method focuses model uncertainty and is complementary to these works. How to incorporate uncertainty decomposition into our method remains open area for future work.
Uncertainty in Operator Learning: As operator learning starts to gain popularity, various works start to explore how to formulate uncertainty in this setting. Guo et al. (2023) explores variance estimation but only provides standard deviation under Gaussian assumption and constrained to single-point uncertainty estimates, [enitez et al. (2023) approaches uncertainty estimation from a statistical learning lens, and focuses loss. We provide bound simultancously for all points the domain that is able to capture error structure.
7 Conclusion
We show method of risk-controlling neural operator, which leverages conformal prediction and quantile loss to provide principled uncertainty estimates simultaneously for all points on the domain. We show empirically 2D Darcy flow problem and a 3D car pressure prediction problem that our method consistently provides the tightest band while satisfying target calibration percentage among various uncertainty quantification methods. Notably, in the 3D problem, our method is the only method that satisfies target calibration percentage of 98%. We also show that our method helps reveal interpretable error structures via visualization. In addition to empirical results, we also provide proof that shows the calibration percentage guarantee of our method.
While method is the first introduction of conformal prediction to operator learning, note that, method is prescriptive and comes with principled calibration guaranlee, we still rely on a good heuristic uncertainty estimator (E' in equation 4 to obtain tight uncertainty balls. In the worst case, if the heuristic estimator E is completely uninformative, our method will yield very wide uncertainty bands. In this work, we locused on expectation over the domain, which is an average, as a measure of risk. Ofien, other notions of risks, including the worst-case formulation, may be of interest that we leave for future work. There are still many open problems in extending principled uncertainty quantification to scientific ML problems involving PDEs. For example, uncertainty decomposition, better representations of error structure, and efficient sampling are all important directions for future work.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Author Contributions
Z. Ma came up with the method, developed the theory, motivated the problem, and performed the experiments. Pitt merged the project code with the neural operator library and reproduced the results. K. Avizadeenshelli and A. Anandkumar advised on the overall direction of the project.
Acknowledgments
Z. Ma is supported by the Kortschak Scholarship. A.Anandkumar is supported by the Bren Named Chair, Schmidt AI 2050 Senior fellow, and ONR (MURI grant 000014-18-1261/1 The authors thank Jean Kossaifi for several helpful discussions regarding code reproducibility. The authors thanks Julius Berner, Zongyi Li, and Nikola Kovachki for helpful discussions.
References
Niccolò Ajrokdi, Jacopo Diquigiovanni, Mattco Fontana, and Simone Vantini. Conformal prediction bands for two-dimensional functional time series. Compulational Statislàcs & Dala Analysis, pp. 107821, 2023.
Deepak Akhare, Tenglei Luo, and Jian-Xun Wang. Diflhybrid-uq: Uncertainly quantification for differentiable hybrid neural modeling, 2023.
Charalambos D. Aliprantis and Kim C. Border. Infinite Dimensional Analysis: a Hitchhiker's Guide. Springer, Berlin; London, 2006. ISBN 9783540326960 3540326960.
doi: 001007/3-540-295877
Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution- free uncertainty quantification. arXiv preprinl arXiv:2107.07511, 2021.
Anastasios Angelopoulos, Amit Pal Kohli, Stephen Bates, Michael Jordan, Jilendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Image-to-image regression with distribution-fee uncertainty quantification and applications in imaging. In International Conference on Machine Learning, pp. 717 730.
PMLR, 2022.
Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribation-free, risk-controlling prediction sets. J. ACM, 68(6), sep 2021. ISSN 0004-5411. doi: (0.1145/3478533 URL
https://doi.oog 01/10.01////01111/4775//5333 1145/3478535
J. Antonio Lara Benitez, Takashi Furuya, Florian Faucher, Anastasis Kratsios, Xavier Tricoche, and Maarten V. de Hoop. Out-of-distributiinnll risk bounds for neural operators with applications to the helmholtz equation, 2023.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shaponet: An information-rich model repository, 2015.
Youngseog Chung, Willie Neiswanger, Ian Char, and Jeff Schmeider. Beyond pinball loss: Quantile methods for calibrated uncertainty quantification. Advances in Newral Information Processing Systems, 34:10971-1089 2021.
Jacopo Diquigiovanni, Matteo Fontana, and Simone Vantini. The importance of being a band: Finitc-sample exact distribution-free prediction sets for functional data. arXie preprint arXiv:2102.0674 000.6666, 2021.
Jacopo Diquigiovanni, Matteo Fontana, and Simone Vantini. Conformal prediction bands for multivariate functional data. Journal of Multivariate Analysis, 189:104879, 2022.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050 1059. PMLR, 2016.
Vignesh Gopakumar, Stanislas Pamela, Lorenzo Zanisi, Zongyi Li, Ander Gray, Daniel Brennand, Nitesh Bhatia, Gregory Stathopoulos, Matt Kusner, Mare Peter Deisenroth, Anima Anandkumar, JOREK Team, and MAST Team. Plasma surrogate modelling using fourier neural operators, 2023.
14

---

Published in Transactions on Machine Learning Research (08/2024)
Ling Guo, Hao Wu, Wenwen Zhou, Yan Wang, and Tao Zhou. Ib-uq: Information bottleneck based uncertainty quantification for neural function regression and neural operator learning, 2023.
Roger Koenker. Quantile regression, volume 38. Cambridge university press, 2005.
Thorsten Kurth, Shashank Subramanian, Peter Harrington, Jaideep Pathak, Morteza Mardani, David Hall, Andrea Miele, Karthik Kashinath, and Anima Anandkumar. Fourcastnet: Accelerating global high-resolution weather forecasting using adaptive fourier neural operators. In Proceedings of the Platform for Advanced Scientific Computing Conference, PASC '23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701900. doi: 1145/3592979.35933
URL https:://di..rrg::
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Bhundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Remi Lam, Alvaro Sanzaez, Malthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, Alexander Merose, Stephan Hoyer, George Holland, Oriol Vinyals, Jacklynn Stott, Alexander Pritzel, Shakir Mohamod, and Peter Battaglia. Learning skillful medium-range global weather forecasting. Science, (62 1421, 2023. doi: 10.1126/science. adi2336. URL https: //wp:.::/www... science. org/
ad12336
Jing Lei, Alessandro Rinaldo, and Larry Wasserman. A conformal prediction approach to explore functional data. Annals of Mathematics and Artificial Intelligence, 74:29-43, 2015.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprinl trXiv:2003.034 2020.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021. URL htttpss//ppparrrreee.nntftoo c8P9NQVtmnO
Zongyi Li, Nikola Borislavov Kovachki, Chris Choy, Boyi Li, Jean Kossaifi, Shourya Prakash Otta, Moham- mad Amin Nabian, Maximilian Stadler, Christian Hundt, Kamyar Azizzadenesheli, and Anima Anandkumar. Goomettyy-iformeed neural operator for large-scale 3d pdes, 2023.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in neural information processing systems, 32, 2019.
Emilia Magnani, Nicholas Krämer, Runa Eschenhagen, Lorenzo Rosasco, and Philipp Hennig. Approximate bayesian neural operators: Uncertainty quantification for parametric pdes. arxie preprint XrXiv:2208. 01566, 2022.
Xuhui Meng, Liu Yang, Zhiping Mao, José del Águila Ferrandis, and George Em Karniadakis. Learning functional priors and posteriors from data and physics. Journal of Computational Physics, 457:111073, May 2022. ISSN 0021-9991. doi: 2...01//..ep..222.1111.. 111073. URL http://dxxxdoo.o//1///// 2022. 111073
Elias Nehme, Omer Yair, and Tomer Michaeli. Uncertainty quantification via neural posterior principal components. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL
https://openrevier nett/forumm?dd=nn2jjnxxyzz
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Podram Hassanzadch, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators, 2022.
15

---

Published in Transactions on Machine Learning Research (08/2024)
A Persson and Federico Grazzini. User guide to ecmwf forecast products. Meteorological Bulletin, 3:153 pp., 01 2007.
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019.
Nobuyuki Umetani and Bernd Bickel. Learning three-dimensional flow for interactive aerodynamic design.
ACM Trans. Graph., 37(4), jul 2018, ISSN 0730-0301. doi: 10.1145/3117517.22003255 URL https aadoi.or/// wtt.://g./....../........ 1wt.1/11//9991..... 3201325
Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer- Verlag, Berlin, Heidelberg, 2005. ISBN 0387001522.
Vladimir Vovk, Ivan Petej, Ilia Nouretdinov, Valery Manokhin, and Alexander Gammerman. Computationally efficient versions of conformal predictive distributions. Nearocompuling, 397:292-33 308, 2020.
Gege Wen, Zongyi Li, Qirui Long, Kamyar Azizzadenesheli, Anima Anandkumar, and Sally M Benson. Real-time high-resolution co geological storage prediction using nested fourier neural operators. Energy 8 Environmmntal Science, 16(4):1732-1741, 2023.
O.C. Zienkiewicz, R.L. Taylor, and P. Nithiarasu. The finite element method for fluid dynamics. In The Finite Element Method for Fluid Dynamics (Seventh Edition), pp. xxxv xxxvi. Hanterrworth-Heinemaae Oxford, seventh edition edition, 2014. ISBN 978-1-45617-633-4 doi: https://doo.orr//10./ 10. 111.:///-7-----11---1111 00024-8. URL bettps:/wwwe htw.:A/.....tggge////aaaaaaaaaa.
Zongren Zou, Xuhui Meng, and George Em Karniadakis. Uncertainty quantification for noisy inputs- physics-inffe informed neural networks and neural operators, 2023.
16


Reviews:
[{"review_id": "1s5ZSo9pCN", "reviewer": "Reviewer_U9P7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. **Interesting Research Problem**: The paper addresses a novel and important problem by applying Conformal Prediction to calibrate operator learning, which is a fresh approach in the field.\n2. **Good Performance**: The proposed method demonstrates strong empirical performance, outperforming existing methods in the experiments conducted.\n3. **Clear Visualization**: Figure 1 is particularly well-designed and easy to understand, effectively illustrating the core concept of the proposed method.\n4. **Theoretical Guarantees**: The paper provides robust theoretical guarantees, enhancing the reliability and rigor of the proposed approach.\n\n### Weaknesses\n1. **Lack of Comparison Methods**: While the paper compares with MCDropout, there are many recent methods in the field of uncertainty quantification. A broader comparison with more advanced methods would strengthen the evaluation.\n2. **Limited Application Domains**: The method is tested on only two domains. Expanding the experiments to include more diverse domains would better demonstrate the generalizability of the proposed approach.\n3. **Heuristic Dependency**: The method relies on a good heuristic uncertainty estimator. If this estimator is uninformative, the resulting uncertainty bands could become excessively wide, reducing the practical utility of the method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the Risk-Controlling Quantile Neural Operator (UQNO) framework, which leverages conformal prediction to provide calibrated uncertainty estimates for operator learning. This method offers uncertainty quantification over the entire function space, distinguishing it from traditional point-wise approaches. It provides robust, distribution-free, finite-sample calibration guarantees. Empirical validation on 2D Darcy flow and 3D car surface pressure prediction tasks demonstrates that UQNO outperforms existing methods, showing superior performance in both data-rich and data-scarce scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Compare with More Advanced Methods: Incorporate comparisons with a broader range of recent methods in the field of uncertainty quantification to provide a more comprehensive evaluation.\n2. More Experimental Domains: Expand the experiments to include additional application domains to better test the generalizability of the proposed methods.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1C7q9hHZau", "reviewer": "Reviewer_7Nsy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n**Clarity**: overall the paper is clear and easy to follow. The problem is well-motivated. In the intro the authors discussed what is missing in existing methods: function-space coverage, distribution-free calibration guarantee, and scalability. Related works are discussed. \n\n**Novelty**: the proposed method can be considered novel, the theoretical and empirical results are also interesting \n\n**Significance**: Table 1 and Figure 2 show qualitative and quantitative evidence that the proposed method is better than other baselines. Theoretical results also add to significance. \n\nWeaknesses: \n\n**Quality**: overall good quality, but with some typos and missing references\n\n**Significance**: for the alternative methods you compare to, what are the hyperparameter configurations you used? Are these methods giving their best performance? (is the comparison against strong baselines? Or weak baselines)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a function-space uncertainty formulation for operator learning. This method has better coverage of the function domain and is more scalable. Theoretical results are provided to show the effectiveness of the method together with empirical evaluation showing the method works better than other uncertainty methods such as MCDropout and Laplace approximation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Significance:\n\nCan you explain how the alternative approaches are setup to perform well in the experiment setting you use? \n\nDemonstrating scalability: \n\n- To my understanding the bandwidth results are showing the proposed method can perform well compared to others. Can you provide some other metrics on scalability to make it even more clear? For example, is there some larger tasks where it might take a very long time for other methods to tackle but the proposed method can finish in a small amount of time? \n\n\nMinor issues: \n\n- A missing equation reference at Algorithm 1 line 4, and middle of page 5, end of page 7 and middle of page 10. \n- Figure 1: while I think the figure is very nice and interesting, it can be a little difficult to perceive the relative positions of points in 3D. Can you also provide a 2D illustions? (sth like a vertical slice of the landscape). The other thing is it can be hard for colorblind readers to distinguish between e.g. red and green colors, would be nice to enhance the visualized points by using different types of points (dot, cross, triangle for example). \n- Page 12 \"Equation equation 4\"", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pOWyLi7bcA", "reviewer": "Reviewer_L1xe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n 1.  The ideas make sense as presented, and improvements over baselines.  The idea of introducing operator learning allows to harness modern approaches.  \n\n 2.  The method does well on the two nicely chosen data sets.  \n\nOverall, these probably make the paper suitable for TMLR.  \n\nWeaknesses:\n 1.  It only considers 2 data sets.  A more thorough paper would have used several more.  \n\n 2.  The writing was not that accessible for someone from outside the area due to how it was disjointed.  I think the Theoretical Guarantees in Section 5 may have been better integrated earlier, since some concepts like Split Conformal Prediction were not defined clearly in Section 3 (although Section 5 and Section 4 ultimately can mostly explain what is going on).  \n\n 3.  Similarly, this reader would have appreciated more detail on how the operator E was learned.  It cites some papers, but more should be said in the paper.  Was in a NN or a kernel machine or a random forest?  \n\n 4.  Figure 6 and 7 overlap.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper develops a method for predicting the uncertainty of predictions in a PAC style setting, where the goal is to identify a threshold for error alpha, where an 1-delta fraction of new data have error within that threshold.  \nIt achieves this in two steps:\n  1.  It uses operating learning (learning a function E) under a quantile regression loss to estimate this threshold for future query point.  \n  2.  It calibrates this function E, with another parameter lambda, so final estimate of error at any point p is lambda * E(p).  It does this through data splitting.  Evaluating E on a train set, and then using test set to choose lambda to fit the desired (alpha, delta) parameters.  \n\nOn experiments it does well against some useful baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Need to address Weakness 4.  \n\nI encourage authors to address other weaknesses, as I think will strengthen the paper.  But no strictly required.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all suggest accept, and I agree with them. The paper provides a novel method for uncertainty estimation in complex settings, and the authors provide both theoretical and empirical evidence to justify their claims. Most of the reviewer comments were addressed, except for the request for more datasets, but I do not feel this is reason to not accept it.
Decision: Accept (poster)